Using native modules in Electron

Recently, we built NeetoRecord, a loom alternative. The desktop application was built using Electron. In a series of blogs, we capture how we built the desktop application and the challenges we ran into. This blog is part 8 of the blog series. You can also read about part 1, part 2, part 3, part 4, part 5, part 6, part 7 and part 9 .

Native modules allow developers to access low-level APIs like hardware interaction, native GUI components, or other system-specific features. Because Electron applications run across different platforms (Windows, macOS, Linux), native modules must be compiled for each target platform. This can introduce challenges in cross-platform development.

To simplify this process, Electron provides tools like electron-rebuild to automate the recompilation of native modules against Electron's custom Node.js environment, ensuring compatibility and stability in the Electron application.

How to use electron-rebuild

electron-rebuild can automatically determine the version of Electron and handle the manual steps of downloading headers and rebuilding native modules for our app.

To do a manual rebuild, run the below command.

./node_modules/.bin/electron-rebuild

// If we are on windows
.\node_modules\.bin\electron-rebuild.cmd

This process should be run after each native package is installed. We can add this command to the postinstall script to automate it.

"scripts": {
    "postinstall": "./node_modules/.bin/electron-rebuild"
    //...others
}

Since Electron uses Chromium browser windows as the user interface, we need to exclude any native modules from being bundled in the renderer process, which runs inside the browser window. To achieve this, we need to separate frontend modules from native modules and install them in separate node_modules folders.

Two package.json structure

To tackle this problem, Electron developers started using two package.json structure, where the first one, which sits at the root of the project, includes the dependencies that are needed for the user interface and all the devDependencies that are needed to develop, build, and package the application.

// root package.json
{
  "name": "my-app",
  "version": "1.0.0",
  "description": "A sample application",
  "license": "Apache-2.0",
  "main": "./src/main/main.mjs",
  "dependencies": {
    "react": "^18.2.0",
    "react-router-dom": "5.3.3"
  },
  "devDependencies": {
    "electron": "^31.2.1",
    "electron-builder": "^25.0.1"
  }
}

And a second package.json file, located at ./app/package.json, includes all the native dependencies that should only run in a Node.js environment. The electron-rebuild postinstall script we discussed should be added to the ./app/package.json.

// ./app/package.json
{
  "name": "my-app",
  "version": "1.0.0",
  "description": "A sample application",
  "license": "Apache-2.0",
  "main": "./dist/main/main.js",
  "scripts": {
    "postinstall": "./node_modules/.bin/electron-rebuild"
  },
  "dependencies": {
    "sqlite3": "^5.1.7",
    "sharp": "^0.33.5"
  }
}

We can add a postinstall script in the root to automatically install the dependencies listed in ./app/package.json when installing the root package.json.

// root package.json
{
  "name": "my-app",
  "version": "1.0.0",
  "description": "A sample application",
  "license": "Apache-2.0",
  "main": "./src/main/main.mjs",
  "dependencies": {
    "react": "^18.2.0",
    "react-router-dom": "5.3.3",
  }
  "devDependencies": {
    "electron": "^31.2.1",
    "electron-builder": "^25.0.1",
  },
  "scripts": {
    "postinstall": "yarn --cwd ./app install"
  }
}

Now, if we run the yarn install from the root, after installing root dependencies, it will install native dependencies in the app/ folder and then rebuild the native packages. All in one command.

When building, we should output the frontend bundles to the app/dist folder. This way, when packaging, both our native packages and other dependencies will be contained within the app folder. Read this blog to learn more about how to build and publish an Electron application.

electron-app
├── assets
├── app
│   └── node_modules
│   └── dist
│   └── package.json
├── src
├── node_modules
├── package.json

This approach works well while packaging the app, but during development, how can we access the native modules? To solve this, we need to create a symlink from app/node_modules to src/main/node_modules. We can create a script to handle this.

// ./scripts/link-modules.mjs
import fs from "fs";

fs.symlinkSync("./app/node_modules", "./src/main/node_modules", "junction");

We can run this script in postinstall as well:

// ./app/package.json
{
  "name": "my-app",
  "version": "1.0.0",
  "description": "A sample application",
  "license": "Apache-2.0",
  "main": "./dist/main/main.js",
  "scripts": {
    "link-modules": "node ../scripts/link-modules.mjs",
    "postinstall": "./node_modules/.bin/electron-rebuild && yarn link-modules"
  },
  "dependencies": {
    "sqlite3": "^5.1.7",
    "sharp": "^0.33.5"
  }
}

Problem with two package.json structure

In most JavaScript projects, metadata such as version, name, and description is typically stored in the root package.json. However, in this case, when packaging the app, we'll be including ./app/package.json instead of the root package.json. This means all metadata should be in ./app/package.json rather than in the root file.

This approach poses a challenge, particularly with tasks like automatic version bumping. When updating the version or other metadata, changes need to be made in two places, which can lead to inconsistencies.

To simplify this process and avoid maintaining two separate package.json files, we can dynamically create ./app/package.json, allowing us to manage everything in one place. Since we cannot add native dependencies directly to the root dependencies, we can introduce a new field called nativeDependencies. When dynamically generating ./app/package.json, we can copy the nativeDependencies into the dependencies field of ./app/package.json.

This can be accomplished by creating a script to automate the process:

// ./script/create-native-package-json.js
import fse from "fs-extra";

const packageJson = JSON.parse(fse.readFileSync("./package.json", "utf8"));

const APP_DIR = "./app/";

const releaseJson = {
  name: packageJson.name,
  version: packageJson.version,
  description: packageJson.description,
  license: packageJson.license,
  author: packageJson.author,
  main: "./dist/main/main.js",
  scripts: {
    postinstall: "/node_modules/.bin/electron-rebuild && yarn link-modules",
    "link-modules": "node ../scripts/link-modules.mjs",
  },
  dependencies: packageJson.nativeDependencies,
};

fse.mkdirSync(APP_DIR, { recursive: true });

fse.writeFileSync(
  APP_DIR + "package.json",
  JSON.stringify(releaseJson, null, 2)
);

In the script, we first fetch the root package.json, create a JSON file, and copy all the metadata. We then update the main field to point to the compiled version of main.js, copy nativeDependencies into the dependencies, and add the postinstall script for electron-rebuild and node_modules linking we discussed earlier.

We can run this script in the root postinstall, so that if we make any changes to package.json, it will be updated in ./app/package.json during yarn install.

// root package.json
"scripts": {
    "nativeInstall": "yarn --cwd ./app install",
    "postinstall": "node ./script/create-native-package-json.js && yarn nativeInstall",
    //...others
}