How we fixed the Cypress "Out of memory" error for Chromium browsers

At BigBinary, we use Cypress as our primary end-to-end testing framework because of its simplicity and compatibility. We have 400+ tests across multiple products most of which are long-running tests handling complex workflows. We use the most commonly used version of Chromium as the test browser to make sure the tests capture how the majority use our products. While the development has always been smooth sailing, the same cannot be said about the test runs. As the number of tests and the duration for each of them increased, our tests would randomly crash with the following error:

1We detected that the Chromium Renderer process just crashed.
2
3This is the equivalent to seeing the 'sad face' when Chrome dies.
4
5This can happen for a number of different reasons:
6
7- You wrote an endless loop and you must fix your own code
8- You are running Docker (there is an easy fix for this: see link below)
9- You are running lots of tests on a memory intense application.
10    - Try enabling experimentalMemoryManagement in your config file.
11    - Try lowering numTestsKeptInMemory in your config file.
12- You are running in a memory starved VM environment.
13    - Try enabling experimentalMemoryManagement in your config file.
14    - Try lowering numTestsKeptInMemory in your config file.
15- There are problems with your GPU / GPU drivers
16- There are browser bugs in Chromium
17
18You can learn more including how to fix Docker here:
19
20https://on.cypress.io/renderer-process-crashed
Copy

The occurrence of the crashes was rare initially. But as our test suites expanded the crash frequency increased as well. The crashes were so frequent at a point in time that none of our tests would run to completion. Neither the solutions mentioned in the official documentation nor the suggestions in the community discussions (like enabling experimentalMemoryManagement) were effective. This led us to investigate this problem.

About our CI setup

We used to run Cypress on CircleCI on a medium Docker resource class. This resource class allocates 4GB of memory to the process. Later on, we moved to our home-grown CI solution, NeetoCI for running our Cypress tests which gave us much more control over the test environment.

The investigation setup

Since the errors were caused because Cypress ran out of memory, we started by looking into the resource utilization on the VM environment. We noticed that none of the crashed runs used more than 50% of the allotted memory. The memory starvation while using only a portion of the allocated resources, meant that Cypress was not utilizing the full memory.

We couldn't reproduce this issue reliably so we attempted to simulate the error by creating a high memory usage scenario. For the simulation, we created a dummy test that takes the following steps.

1. Visit a page.
2. Get an element.
3. Save the element in the memory as a new alias.
4. Repeat steps 1-3 infinitely until the browser crashes.
5. During each iteration, log the iteration number to know how many iterations were completed successfully before the crash.

For logging the iteration number, we used the Cypress task - log illustrated in the official documentation. The iteration number provided us with an additional metric to compare the performance of the solutions we tried. The code for implementing the investigation setup can be seen below.

1const saveButtonAsAlias = iteration => {
2  cy.get(".button").as(`button-${iteration}`);
3  saveButtonAsAlias(iteration + 1);
4  cy.task("log", iteration);
5};
6
7it("dummy test", () => {
8  cy.visit("/");
9  saveButtonAsAlias(1);
10});
Copy

The above code will save the same button component as different aliases in the memory thus simulating a high memory usage test environment. On executing this test we saw that the memory usage peaked at about 1GB - 1.5GB in a 4GB docker environment before the browser crashed.

Solutions

1. Using an alternate browser

Even though Google Chrome is the most popular browser in the market, it's far from being the most memory efficient. So we tested out with other chromium-based browsers available for Cypress and concluded that Microsoft Edge ran the tests in a much more memory-efficient manner. While running the dummy test, we observed the memory usage by each of the browsers and compared the results.

Google Chrome ran the tests faster and crashed first when memory was starved. Microsoft Edge ran the tests at a similar pace initially but when the memory was almost used up completely, the tests slowed down and the browser started rigorous garbage collection. The memory usage was increasing at a gradual rate and more iterations were completed successfully, as compared to Chrome, before the browser crashed. The table below shows the runtime comparison between Google Chrome and Microsoft Edge (higher runtime is better).

While switching the browser improved the completion rate of the runs, it still didn't solve the issue completely. This led us to look for further enhancements.

2. Increasing the max-old-space-size

The most unusual behaviour we noticed in resource utilization was that Cypress did not use the entire allocated memory before crashing. To understand why Cypress behaves like this we need to have a basic understanding of its architecture which can be seen below.

Cypress works as two different processes. The NodeJS application and the browser on which the tests run. When executing cypress run and cypress open commands, we start the NodeJS application. This NodeJS application goes through our tests and configuration and loads them into our preferred browser where they are executed.

The split architecture of Cypress means that the memory allocation for the NodeJS process and the Chromium browser are different. This is why the total memory usage by the NodeJS process doesn't give us proper insights into why the Chromium process crashed and was starved of memory. To analyze the browser memory usage we used the browser Performance APIs.

We found that the Cypress tests were allocated only about 500MB of memory despite the test environment having 4GB of memory. So the solution was to increase the heap memory allocated to the chromium renderer. The max-old-space-size command-line flag is used to set the V8 engine's maximum old memory limit. When the memory usage approaches this limit, garbage collection begins in an effort to free up memory. So by manually increasing the max-old-space-size for the chromium renderer, we can increase the heap memory allocated to it.

If it were a node application, the process of increasing the max-old-space-size would be as simple as executing the Cypress command like-wise:

NODE_OPTIONS=--max-old-space-size=3500 yarn cypress run

But because of the split architecture, executing the above command only increases the max-old-space-size for the NodeJS application and not the actual Cypress tests running in the Chromium browser. To increase the max-old-space-size for the Chromium renderer we need to make use of the Browser launch APIs provided by Cypress.

1// cypress.config.js
2
3const { defineConfig } = require("cypress");
4
5module.exports = defineConfig({
6  // setupNodeEvents can be defined in either
7  // the e2e or component configuration
8  e2e: {
9    setupNodeEvents(on, config) {
10      on("before:browser:launch", (browser = {}, launchOptions) => {
11        launchOptions.args.push("--js-flags=--max-old-space-size=3500");
12
13        return launchOptions;
14      });
15    },
16  },
17});
Copy

In the configuration above, we can see that we have passed in the --max-old-space-size command line flag within the --js-flags Chromium flag. This is because Chromium expects NodeJS options using the --js-flags command line switch. The above configuration increases the maximum usable heap size of the Cypress tests to 3500MB.

Depending on the available memory on the test environment, we can increase or decrease the max-old-space-size value. The benchmarking results we received after making this configuration change showed a significant improvement in the performance. The table below documents the runtime comparison between the default max-old-space-size and max-old-space-size set to 3500 MB (higher runtime is better).

The benchmark above shows the improvement in performance after increasing the max-old-space-size in the Google Chrome browser. By switching the browser to Microsoft Edge we got even better results. The table below shows the runtime comparison between the default max-old-space-size and max-old-space-size set to 3500 MB in each of these browsers (higher runtime is better).

Additional tips to reduce memory usage in Cypress

1. Chromium browsers sandbox the pages which increases the memory usage. Since we're running the Cypress tests on trusted sites, we can enable the --no-sandbox flag to reduce memory consumption.

2. When running Cypress tests in headless mode, we can disable the WebGL graphics on the rendered pages to avoid additional memory usage by passing the --disable-gl-drawing-for-tests flag.

3. When running tests on low-resource machines, using hardware acceleration can impact performance. To avoid this we can pass the --disable-gpu flag.

1// cypress.config.js
2
3const { defineConfig } = require("cypress");
4
5module.exports = defineConfig({
6  // setupNodeEvents can be defined in either
7  // the e2e or component configuration
8  e2e: {
9    setupNodeEvents(on, config) {
10      on("before:browser:launch", (browser, launchOptions) => {
11        if (["chrome", "edge"].includes(browser.name)) {
12          if (browser.isHeadless) {
13            launchOptions.args.push("--no-sandbox");
14            launchOptions.args.push("--disable-gl-drawing-for-tests");
15            launchOptions.args.push("--disable-gpu");
16          }
17          launchOptions.args.push("--js-flags=--max-old-space-size=3500");
18        }
19        return launchOptions;
20      });
21    },
22  },
23});
Copy

Conclusion

Since Cypress tests are executed inside the browser all the constraints of a browser environment apply to them including the memory constraints. The default configurations in the browsers are targeted to run on the most number of systems. When facing memory starvation issues during complex and long-running tests, we should configure Cypress according to the resources available in the environment in which our tests are running to achieve peak performance. Increasing the available memory for the browser by manually setting an appropriate max-old-space-size value and choosing a memory-efficient browser will make sure that Cypress will be able to run smoothly in most of the scenarios.

References

• Chromium renderer crash GitHub issue
• Memory management glossary
• Increasing node memory size
• Cypress docker images
• Chromium command-line switches
• Docker resource constraints