Until early 2024, Cypress used to be the most downloaded end-to-end (e2e) testing framework in JavaScript. Since then, it has seen a steep decline in popularity and Playwright has overtaken it as the most downloaded end-to-end testing framework.

We at BigBinary also switched from Cypress to Playwright in late 2023. In this article, we will see some critical reasons for this change in trends and our personal views on why we think Playwright is the superior JavaScript testing framework.

Why we chose Cypress initially

At BigBinary, we are building a number of products at Neeto. When the number of products in our product suite grew and the complexity of each one increased, we needed an automated end-to-end solution to ensure our applications were stable since manual testing was no longer viable.

When the discussion about choosing the e2e testing framework began in mid-2020, a few names emerged, including top players like Selenium and Cypress and new players like Playwright. We chose Cypress owing to its popularity and simplicity.

We were satisfied with its overall performance and easy learning curve. We chose Cypress as our primary e2e testing framework and wrote extensive e2e tests for the entire application suite. While things were smooth sailing initially, we soon encountered many issues with Cypress.

Why we decided to switch to Playwright

In late August 2023, Cypress released version 13, a major upgrade to the framework that brought along many new features. As Cypress users, we were overjoyed. But the excitement quickly turned to frustration when we realized that, along with the latest features, Cypress had introduced a few changes that were not so open-source in nature.

It's a known fact that Cypress Cloud is a very expensive platform. A few third-party providers like Currents.dev and Testomat provided similar services at much more affordable costs. However, Cypress version 13 blocked all third-party reporters. The main reason offered by the Cypress team was that Cypress Cloud was their primary source of income and that they had to block the third-party tools to survive in the market. They later revised this explanation with other arguments on how these third-party reporters misused the Cypress name for personal gains due to public backlash.

We had switched to Currents.dev ourselves a few months prior to the event and were affected by this change. At that point, we had two options: switch to Cypress Cloud and incur the additional cost or stay with Currents.dev but get locked into an older version of Cypress permanently.

Both of these choices were unacceptable to us. This was the final nudge we needed to switch to a new framework. We were already dissatisfied with many issues with Cypress, so we took advantage of the opportunity to research the best e2e testing framework available and switch to that. We compared all the popular frameworks available and observed how they solved our pain points with Cypress. That is when we fell in love with Playwright.

In our comparison, Playwright was the fastest framework in terms of raw performance and had the highest adoption rate compared to all the other frameworks. It is an open-source framework maintained by Microsoft. The architecture would enable us to automate more scenarios that were deemed unautomatable using Cypress. We were thrilled to learn that Playwright would fix most of the issues we faced with Cypress.

Features locked behind a paywall and control over third-party software

While Cypress supports parallelism and orchestration, it is blocked behind a paywall with a subscription to Cypress Cloud. This means these features, which can easily be implemented with the base Cypress package, are only accessible through an external package.

While Cypress provides APIs for reporters and orchestration, it deliberately blocks popular third-party tools and services. That's not a good open source practice. The combination of both makes Cypress an incomplete tool without subscribing to the expensive Cypress Cloud plans, even though the tool is considered free and open-source.

At the same time, Playwright is an entirely open framework in which anyone can create and publish third-party reporters. It comes in-built with features such as parallelization, sharding and orchestration without needing third-party tools and services. The Playwright team goes a step further by showcasing the popular third-party reporters on their official documentation.

Performance

Cypress is the slowest of the e2e testing frameworks available in JS. Here is the list of the most popular frameworks in decreasing order of performance.

Let's compare the performance when the same scenario is implemented in Cypress and Playwright. The scenario is to visit the Neeto homepage and verify the page title.

1// Cypress
2
3cy.visit("https://neeto.com");
4cy.title().should("eq", "Neeto: Get things done");
Copy

1// Playwright
2
3await page.goto("https://neeto.com");
4expect(await page.title()).toBe("Neeto: Get things done");
Copy

The results speak for themselves. While Cypress took 16.09 seconds to finish the execution, Playwright took only 1.82 seconds. This is an improvement of 88.68%! Here, the execution time combines the time taken for setup and the time to complete the test. This is the actual time that matters because this is the time an engineer has to wait until they see the final test result.

This shows how much of a performance gain switching to Playwright gave us. If we look at a more practical example, our authentication flows through Cypress, and Playwright gives a much better idea of the time saved. The authentication flow, which consistently took around 2 minutes in Cypress, is completed in under 20 seconds using Playwright.

Playwright's out-of-the-box support for parallelism and sharding can have multiplicative effect in time savings. If we provide a process-based parallelism of 4 and shard the tests in 4 machines, then 16 tests are run concurrently, which reduces the execution time dramatically.

Implementing these additional configurations reduced the total test duration for one of our products from 2 hours and 27 minutes to just 16 minutes. This 89.12% of time saving, directly translating to CI cost savings.

Memory issues

Cypress follows a split architecture. This means that Cypress executes the tests with a NodeJS process, which orchestrates the tests in the browser where the tests are executed. This also means that the browser execution environment limits the memory available for tests. Due to this, we have faced crashes in between tests multiple times. At one point, the crashes became so frequent that we had to invest a lot of time and energy into finding a solution because no test executions were running to completion. We have written a detailed blog on this topic, which can be found here.

Playwright fixes these issues because it handles the test execution in a NodeJS service and communicates with the browsers using CDP sessions. This means that the memory management can be done on the NodeJS application while the browser only has to worry about handling the actual web application we're testing.

Architecture prone to flakiness

Many of Cypress's features are closely tied to its architecture. For example, one of the popular features in Cypress is its retry mechanism and chaining. However, these features do not always go hand-in-hand.

Let's consider this snippet of Cypress code.

1cy.get(".inactive-field").click().type("Oliver Smith");
Copy

While this code looks great syntactically, it will cause the test to be flaky. This is because, in Cypress, only queries are retried, not commands. In the example above, consider that the class name of the field is updated to active-field when we click on it. This means that the cy.get query locates the field and the click command work fine, but the chain fails at the type command. This is because an element with the class name .inactive-field no longer exists in the DOM tree.

With the Cypress retry mechanisms, one would think that the whole chain would be retried from fetching the element. However, the chain was completed successfully until the click action. So, only the type action will be retried and will cause the whole chain to fail. To avoid this issue, we must rewrite the tests after splitting the chain.

1cy.get(".inactive-field").click();
2cy.get(".inactive-field").type("Oliver Smith");
Copy

While this works without issues, the syntactical sugar that Cypress provides by chaining the commands is no longer usable. Now, let's observe the Playwright code for the same.

1await page.locator(".inactive-field").click();
2await page.locator(".inactive-field").type("Oliver Smith");
Copy

It looks pretty similar. This is because Playwright is designed to reduce flakiness as much as possible. To achieve that goal, it prevents the user from implementing anti-patterns that can lead to flaky results.

Misleading simplicity

Cypress is well known for its simplicity and natural syntax, which even the most non-technical person can learn. The code samples in the official documentation (which is still one of the best documentation for any framework) make it seem like a walk in the park. But you soon realize that the examples are for very straightforward application scenarios that we seldom encounter when working on large projects. When you start automating complex scenarios, things soon get complicated.

While performing simple tasks such as clicking on a button or asserting a text is extremely simple, doing something more moderately complex, such as storing the text contents of a button in a variable, becomes highly complex. This is because Cypress architecture works by enqueuing the asynchronous commands. This means that there are no return values for the commands, and the only way to retrieve values from Cypress commands is through a combination of closures and aliases. Let's consider a scenario where we have to verify that the sum of the randomly generated numbers on the screen is the same as the value shown on the page.

Let's see the difference in code when automating this scenario in Cypress and Playwright.

1// Cypress
2
3// Considering all elements have proper data-cy labels
4
5cy.get('[data-cy="generate-new-numbers-button"]').click();
6cy.get('[data-cy="first-number"]').as("firstNumber");
7cy.get('[data-cy="second-number"]').as("secondNumber");
8cy.get('[data-cy="sum"]').as("sum");
9
10cy.get("@firstNumber").invoke("text").then(parseInt).as("num1");
11cy.get("@secondNumber").invoke("text").then(parseInt).as("num2");
12cy.get("@sum").invoke("text").then(parseInt).as("displayedSum");
13
14// Use the aliases to perform the assertion
15
16cy.get("@num1").then(num1 => {
17  cy.get("@num2").then(num2 => {
18    cy.get("@displayedSum").then(displayedSum => {
19      const expectedSum = num1 + num2;
20      expect(displayedSum).to.equal(expectedSum);
21    });
22  });
23});
Copy

We can see how complicated the code becomes when the scenario is just slightly complex. Meanwhile, the Playwright code will look like this.

1// Playwright
2
3// Considering all elements have proper data-cy labels and the default test-id-attribute is data-cy
4
5await page.getByTestId("generate-new-numbers-button").click();
6const firstNumber = await page.getByTestId("first-number").innerText();
7const secondNumber = await page.getByTestId("second-number").innerText();
8const sum = await page.getByTestId("sum").innerText();
9expect(parseInt(firstNumber) + parseInt(secondNumber)).toBe(parseInt(sum));
Copy

We can see from the code above, how easily we can implement the same logic in Playwright.

Cost of maintenance for Cypress vs. Playwright tests

Cypress is an easy-to-learn framework. This simplicity is due to the abstraction of the most commonly used functionalities into Cypress commands. However, this is a double-edged sword. The abstraction of logic into commands means that customization is complicated in Cypress.

One of Cypress's significant drawbacks is its reliance on HTML tags and attributes to locate an element. While this makes sense from a programming standpoint, the end user is concerned about the roles of the page element (button, heading, etc.) and not how they have been implemented. For the same reason, the text, appearance and functionality of the application are bound to remain consistent throughout the various iterations, while the attributes themselves are prone to changes.

This ultimately means the developers must keep fixing/rewriting the Cypress tests for minor UI updates. Cypress is also the slowest of all the e2e testing frameworks in JavaScript, resulting in longer CI runtimes and costs. These combined make the cost of maintaining Cypress tests exceptionally high.

Besides this, Cypress has many features locked behind its Cypress Cloud platform, which is very expensive, considering the fact that all it does is to collect the test results. This is an additional cost to bear over the already expensive costs to maintain the Cypress tests. Given these factors, the cost of preserving Cypress tests can quickly outweigh the benefits of its simplicity and ease of use.

Playwright solves all of these issues. It has many built-in reporters and an excellent API for creating custom reporters, so many third-party reporters are available. We can even build our custom reporter to save even more costs.

Browser support and support for mobile viewport

Since Cypress tests run directly on the browsers, only a few are supported. Until recently, it did not even support WebKit browsers, even though Safari has a considerable market share. Even at the time of writing this article, Cypress's WebKit support is still in beta. Even if it supports the required browsers, there is still the constraint that only one browser can be used during an execution.

Playwright fixes all these issues with minimal effort from our end. It has complete support for WebKit browsers and conveniently provides a set of presets for the browsers, user agents and viewport of the most popular devices in the market, including mobile devices. Furthermore, Playwright allows us to execute the same test in different browsers concurrently with the help of projects. These configurations give us the confidence that a passing Playwright test means the features will work fine for all users.

Support for multiple tabs and browsers

While most of the features of a web application can be tested within a single tab, there are a few cases where multiple tabs or browsers become necessary. One such scenario we encountered while writing tests for NeetoChat, a real-time chat application. To test NeetoChat we need to open two screens - one for the sender and the other for the receiver.

Cypress lacks the support for multiple tabs, so the only way to test these scenarios was to do this long and complicated process:

1. Login as the sender
2. Send a message
3. Logout
4. Login as the receiver
5. Verify the message
6. Send a reply
7. Logout
8. Login as the sender
9. Verify the response.

We can see the tedious steps that we need to perform for a relatively simple scenario. This becomes even more tedious if we configure sessions in Cypress because we need to invalidate them each time we log out so that we can log in as a different user.

On the other hand, Playwright provides support for multiple tabs and multiple browsers. This means we can log in as the sender from one tab and the receiver from another, making the scenario more straightforward and effective. Additionally, we could identify whether the messages were being delivered in real time because there is no delay in the user switching between the message posting and verification processes. Playwright also supports browser contexts, which isolate the events between two browser instances, aiding in test isolation during parallel test execution.

Lack of necessary tools

Cypress depends on plugins for many necessary tools. These are features we have come to expect from any modern testing framework. Let's examine a few such tools and how Playwright handles them natively.

We must consider that Playwright includes all these tools out of the box while still being more performant than Cypress. As we add such plugins in Cypress, the package size also increases.

Random errors during tests due to Cypress's iFrame Execution Model

As discussed already, Cypress tests are executed inside a browser. They work by running Cypress as the main page and running the application which is being tested as an iframe within the page. This can lead to a lot of unexpected errors during the test execution.

One of the most commonly encountered errors is related to security issues with cookies. When the tested application uses cookies, it might throw random errors during the Cypress execution depending on the configuration. This is because the SameSite configuration of the cookies might block it from being shared with the parent Cypress application. This can lead to them not be sent during API requests causing authentication failures and also cause difficulties with session configuration.

Additional benefits of using Playwright

More resistance to test failures due to minor text changes

At BigBinary, we follow a behavioral testing pattern where we verify the features, and not minor details like texts and styles. We made this decision to ensure that the tests don't fail due to minor changes in the application. While this is our go-to testing style, there are still some cases where we cannot avoid testing the texts on the page (for example, the error message shown while testing negative test cases). This required our tests to be frequently updated while working with Cypress whenever a minor text change was made in the application.

When we switched to Playwright, we got excited about how much we could customize it according to our needs. This customization is available because, under the hood, it's still a Node.js application. We already use i18next to serve the texts on our application. We figured that the tests should use the same translation file. The translation keys remain consistent even when the texts are updated.

This minor change brought a huge difference in our test stability. The average number of tests we had to update each week decreased from 22 to 2. That is a lot of time saved, which we could effectively use to expand our test coverage instead of wasting it fixing the existing suite.

Ability to build our own in-house reporter

When working with Cypress, we had to switch between many reporting tools, including Cypress Cloud, Currents.dev and many other third-party tools. While they all had benefits and drawbacks, we couldn't find one that addressed all our needs. This is where the excellent reporter APIs offered by Playwright allowed us to write our own Playwright reporter - NeetoPlaydash.

We currently use NeetoPlaydash for all our reporting needs and can customize it according to our requirements. Most importantly, we reduced the monthly reporting tool costs by 77% ($405 to $90 per month). We also didn't have to worry about exhausting the monthly test limits of third party reporters allowing us to run our tests more frequently thus improving the stability of our applications.

More coverage on tests relating to third-party integrations

In Neeto products, we have support for third-party integrations. For example, in NeetoCal we can have integrations for Google Calendar, Zoom, Microsoft teams and a lot more third-party applications. Most of these applications have bot detection algorithms implemented in place to ensure that their platforms are not misused by bad actors. This also meant that we had to consider the integration features to be unautomatable when tested using Cypress.

Playwright does things differently. Since it's a Node.js application, it supports all the packages available for the platform. Because of its wide support, the community has developed a lot of tools for Playwright. We took advantage of these tools and plugins and were able to bypass the bot-detection algorithms which prevented us from testing the third-party integrations. This allowed us to test the these integrations in our products effectively and ensure that the application ran smoothly with the help of automation tests.

Conclusion

We strongly believe that migrating to Playwright is one of the best decisions we have ever made. We did not know what we were missing out on until we decided to take the leap and migrate. We got better performance, less flakiness and more coverage from our test suites. The cost and time saved helped us to effectively divert resources to things that actually matter and let the tests do testing instead of using additional resources to maintain the tests themselves.