How to Improve Integration Tests with AI in Next.js with Playwright & Applitools

Disclaimer: This post is sponsored by Applitools. Learn more about sponsored content.

What is Applitools Eyes?

Eyes is a product from Applitools that performs visual comparisons between different snapshots of a website or mobile app.

“Performs visual comparisons” somewhat sells it short though, as the key differentiator with Eyes is that it uses AI to perform that comparison. This means less false positives when comparing results due to dynamic content or screenshots being 1px off.

There’s also a variety of configurations built into this like whether to look for content or layout changes, both from an SDK perspective and UI, giving developers (and teams) a lot of flexibility for customization and control.

How does visual testing with Applitools work?

From a basic perspective, a screenshot is taken at two points in time. Maybe that’s a baseline and on a pull request branch.

Those two screenshots are compared to see if there’s a difference between the two (remember, using AI) and if there is, it’s highlighted, throwing an exception, like any other standard test.

With Applitools, Eyes is integrated via its test runner through pretty much every popular testing tool including Playwright which we’ll use today, but others like Cypress, Puppeteer, Selenium, and the list goes on.

But with that in mind, you can automate a browser to really do whatever you want, such as working its way through a checkout funnel of an online store, taking a snapshot every step of the way, meaning you’re going to get a full-on integration test (a real integration test with visual UI) with a simple set of checkpoints.

Less tests and less assertions you have to write to become more confident in your work.

What are we going to build?

We’re going to spin up a simple app using a Next.js Starter I built that gives us a fake news website called The Hundred (Succession anyone?).

Once ready to go, we’ll first install Playwright, which we’ll use for automating our tests, as well as a first example of opening up a page.

We’ll then get set up with Applitools, set up some snapshots, and even a user flow with Playwright, while capturing checkpoints with Applitools.

Tip: Prefer Cypress? Check out Visual Regression Testing on a Next.js App with Cypress and Applitools

Step 0: Creating a new Next.js app

Digging in, we’ll use a Next.js Starter template as a way to follow along.

In your terminal run:

npx create-next-app -e https://github.com/colbyfayock/demo-news-starter my-news-website

And once it’s complete, you’ll now have a new directory located at my-news-website with your Next.js app.

Next, navigate to your new directory with:

cd my-news-website

And start your local development server with:

npm run dev

When ready, you’ll now be able to visit http://localhost:3000 and see your new app!

Step 1: Installing & Getting Started with Playwright

Starting from the top, we first need to install and set up Playwright.

In your terminal run:

npm init playwright@latest

This will start up a tool that will guide us through our Playwright installation.

After confirming we want to install and use the package, it will ask a few questions including:

• Typescript or JavaScript? This is a personal preference, but I’m going to move forward with JavaScript (But Applitools does support Typescript!)
• Where to put your tests? The tests directory is the default, we can leave it at that and hit enter
• Add a GitHub Actions workflow? GitHub Actions are a fantastic way to automate tests. We won’t cover it here, but it’s usually a good idea!
• Install Playwright browsers? You might not necessarily need to, but let’s hit Yes (the default) to be safe

At this point Playwright will install everything we need.

The only issue is they’re not configured to work with our app, we need to do a little extra work.

By default, Playwright doesn’t know anything about our app. It can load any URL we throw at it, but our app isn’t available unless we start a development server. But we can fix this in our config to automatically start the app!

Open up playwright.config.js and scroll to the bottom where we’ll see a webServer property commented out.

First, uncomment those lines and then update the command property to npm run dev and the url to http://localhost:3000.

Note: Playwright uses http://127.0.0.1:3000 instead of http://localhost:3000 for the URL by default, which will technically work for our case, but feel free to update for consistency and less confusion.

The resulting object should look like:

webServer: {
  command: 'npm run dev',
  url: 'https://localhost:3000',
  reuseExistingServer: !process.env.CI,
},

To run our tests, we can run npx playwright test or we can add it as a test script in our package.json.

Inside package.json under scripts add:

"test": "npx playwright test"

Next let’s create a new basic test.

Create a new file called app.spec.js inside of tests.

Inside tests/app.spec.js add:

const { test, expect } = require('@playwright/test');

test('it should have a title', async ({ page }) => {
  await page.goto('http://localhost:3000');
  await expect(page).toHaveTitle('The Hundred - Your antidote to information overload.');
});

Here we’re:

• Importing the Playwright dependencies
• Creating a new test case with a description
• Navigating to our web app
• Checking if the page title matches what we expect

And then we can run this test!

In your terminal run:

npm run test

Which will now handle starting your local development server, running our tests, and showing the results, which should be passing!

Follow along with the commit!

Step 2: Installing & Configuring Applitools Eyes with Playwright

Now let’s actually get started with Applitools.

First, let’s install our Applitools Eyes dependency:

npm i -D @applitools/eyes-playwright

Next we’ll need to configure our API key.

If you don’t have an account, head over to applitools.com where you can sign up for a free account.

Once inside your Applitools dashboard, select the user dropdown on the top right and select My API Key.

Once selected, a modal will pop up where you can copy your key.

Note: Reminder, this is a secret key that you should not share or commit publicly in your code!

Next, let’s add this to our project.

Create a file in the root of the project called .env.local.

Inside .env.local add:

APPLITOOLS_API_KEY="<Your API Key>"

Now by default, Playwright won’t be able to find our environment variable as that capability isn’t built in by default.

To make this work, we’ll install dotenv which we can easily use to pull in our environment variable into Playwright and with it, Applitools.

First install dotenv with:

npm i -D dotenv

Then at the top of our playwright.config.js file, we’ll actually see that there’s a line already available for us to use dotenv, so let’s uncomment that line, but we also want to change the path to the correct .env.local file:

require('dotenv').config({
  path: './.env.local'
});

And at this point, we should be all configured, and the only thing left to do is run Applitools in our test!

Follow along with the commit!

Step 3: Running a Visual Regression Test with Applitools Eyes

To wire in Applitools into our existing test, let’s first import our dependencies.

Inside tests/app.spec.js add:

import { Eyes, Target } from '@applitools/eyes-playwright';

Then after our title test, let’s add a new test:

test('it should load the homepage', async ({ page }) => {
  await page.goto('http://localhost:3000');
  const eyes = new Eyes();
  await eyes.open(page, 'The Hundred', 'App');
  await eyes.check(Target.window().fully())
  await eyes.close();
});

Here we’re:

• Creating a new instance of Eyes, which is the API that handles running our visual tests
• We open our eyes with the open method (think like a camera, or an… eye) so that we can start capturing information, along with the name of the app and a test name
• We grab a new checkpoint (screenshot) with the check method, specifying that our target is the Window and that we want to capture the full page (.fully())
• And finally we close our eyes, meaning we’re finished capturing checkpoints

And now, if we run our tests again:

npm run test

We should now see our tests passing!

But that’s not telling the whole story.

Head over to your Applitools dashboard (or if you’re in there hit the refresh button) and you should see several new green, passing tests in there!

Now why 3? The default configuration Playwright comes with configures Chrome, Firefox, and Safari to all run together. That means we’re even getting cross-browser testing!

But now let’s see where this really shines.

Let’s head into src/pages/index.js and make a change to the page, such as removing the Unlock component.

Once you do that, run the tests again:

npm run test

This time we should see failing tests where each of them include a link to inspect.

Note: Another Playwright default is to include an HTML reporter that pops up automatically. You can turn this off by removing reporter: 'html', from playwright.config.js

Follow one of those links or simply refresh the Applitools dashboard and we should see another 3 new tests only this time, we get a yellow warning showing that something has changed.

Select one of those tests and click on the checkpoint card, where we’ll see a UI that shows the before and after.

By default, it will just show the parts of the page that changed, but, this is where it gets cool, select the Show diffs caused by element displacement button (or hit the P key).

Applitools will automatically determine what caused the change and highlight only that, not just show every little pixel that’s changed.

This is huge! While it might be pretty obvious in our case that we removed a big chunk of content, for complex layouts and designs, those changes can easily get lost.

This gives us an easy way to inspect the changes!

Now before we move on, we’ll want to either accept these changes or reject them, so either:

• Accept: click thumbs up (👍) and accept the code changes
• Reject: click thumbs down (👎) and reject the code changes.

Finally click the Save button (floppy disk 💾).

And we’re ready to step it up!

Follow along with the commit!

Step 4: Adding Eyes Checkpoints to test user flows

Capturing a page alone is powerful. There are a lot of components (both React components and pieces of the site or app) that a screenshot alone can cover.

But interacting with the application is the most important part of ensuring a person can actually accomplish what they need.

With Playwright, we can easily bounce around our application, clicking on whatever and typing what we need, and all we need to do is simply take more snapshots along the way!

So to start off, let’s create a new test that works its way through a common user journey—signing up.

We’ll start off by navigating to an article, entering in a fake email address to our form, and submitting it, dropping us into a confirmation page.

Note: The form doesn’t actually do anything, the button just redirects to the next page, so no concern with saved data! Source.

Inside tests/app.spec.js add:

test('it should navigate to an article then signup', async ({ page }) => {
  await page.goto('http://localhost:3000');

  await expect(page).toHaveTitle(`The Hundred - Your antidote to information overload.`);

  const $articleLink = page.getByRole('heading', { name: 'Latest' }).locator('..').locator('ul li:first-child p a')
  const title = await $articleLink.evaluate(node => node.innerText)

  await $articleLink.click();

  await expect(page).toHaveTitle(`${title} - The Hundred`);

  await page.getByRole('textbox').click();
  await page.getByRole('textbox').fill('test@test.com');
  await page.getByRole('button', { name: 'Sign Up' }).click();

  await expect(page).toHaveTitle(`Confirm - The Hundred`);
});

Here we’re:

• Navigating to the homepage
• Making sure the page loads
• Grabbing the first article under the Latest column in the sidebar
• Reading the title to use it later
• Clicking the article link to navigate to the page
• Making sure we’re successfully navigated with that title we saved
• Filling out the form with a fake email
• Clicking sign up
• Making sure we’re successfully navigated to the confirmation page

Basically, we’re automating going to an article page and trying to sign up with Playwright.

That wasn’t too bad right? But what’s missing here?

• The page title tests are fragile, what if we change the format?
• This tests that it functionally works, but what if the UI is broken and the HTML / click handlers still work?
• Or worse yet, what if you can’t even see the form because of some kind of messed up HTML / CSS?
• How do we know the rest of the page works?

While that last answer is “add more tests”, what if we can do more with our tests?

As we saw in the last step, that’s where Eyes comes in and captures more than just whether the HTML is recognized, it captures the entire context of the page

So let’s replace some of those assertions there.

Starting off, let’s create a new instance of Eyes, open them up, and grab our first snapshot instead of the first expect statement:

const eyes = new Eyes();
await eyes.open(page, 'The Hundred', 'Signup');
await eyes.check('Home', Target.window().fully())

This is mostly similar to what we did before, but if you notice this time, we’re setting the test name to “Signup” and setting “Home” as the first argument of our first check, moving our Target to the second.

This allows us to name our checkpoint which is useful when we’re trying to capture multiple checkpoints in a single test or user flow.

Continuing on, replace the next expect that captures the article page with:

await eyes.check('Article', Target.window().fully())

And finally the confirmation page with:

await eyes.check('Confirmation', Target.window().fully());
await eyes.close();

Where we’re also running close to tell Eyes we’re done.

And if we run our new tests, we should now be able to see in Applitools our new baseline for our Signup test.

Like before, if we make any changes we’re going to immediately get feedback from Applitools showing exactly what changed.

But what if it’s not something that we changed in code like the content?

Given we’re working with a news site (or blog), it’s not unreasonable to think that we’ll have new content all the time, won’t that break our tests?

Follow along with the commit!

Step 5: Testing dynamic content with ignore regions and match levels

Content published on a news website, or any type of content-driven site, is bound to change and likely change frequently.

This could throw a huge wrench in our ability to run tests.

While AI can be quite powerful, how does it know that your giant header image changing is an intended side effect of a newly published article?

Luckily we have a few tools for how we can handle these types of scenarios.

Starting off, to test this out, how about we randomize the post content being pulled into the application so that it shows something different in the slots each time (or to the best our randomization can).

To do this, we’re going to first rename our posts import to:

import postsData from '@/data/posts';

We’ll then add a new data fetching function to use those posts, randomize the order, and pass it to our page as a prop.

Add the following below the Home component (or above, doesn’t matter):

export async function getServerSideProps() {
  return {
    props: {
      posts: postsData.sort(() => 0.5 - Math.random())
    }
  }
}

And make the posts prop available with:

export default function Home({ posts }) {

Tip: Why use getServerSideProps instead of randomizing the posts after import? It helps avoid hydration errors (different versions on server and client) and allows us to reliably load a different randomized sort each page load.

If we try to reload the page in our browser, we should see the page change each time.

And if we try to run our tests, of course they throw exceptions.

To resolve this, we can start off by ignoring regions, where with ignoring, we have a few options for how we can handle this.

Looking first inside of the Applitools dashboard, when inspecting one of our checkpoints, we have an option right at the top under Annotations where we can select Ignore and draw the areas we want to ignore. Simple as that.

This is really helpful if you want to stick in to the UI. It’s certainly easier to just draw this on the page.

But alternatively, we can add some configuration to tell Applitools the exact elements to ignore using code selectors.

The trick here, is we need a safe selector that will always give us the same results.

While we can use a selector like:

ul li:first-child p a

Like we did in our Playwright tests, we can instead add IDs or Classes to our elements to make this much simpler. It will also hold up better for any refactoring.

Inside src/components/PostListItem/PostListItem.js, find the <Image tag and add a className prop:

<Image className="PostListItem-Image"

Note: you can name the class whatever you want!

Then back inside our test file at tests/app.spec.js, let’s update the “Home” checkpoint of our signup flow test to:

await eyes.check('Home', Target.window().fully().ignoreRegion('.PostListItem-Image'))

If we run the tests again, we’ll still see failing tests, but if we notice this time inside of the dashboard, we’re not getting any differences highlighted on the images inside of our Latest sidebar.

We see blue boxes over our images just like if we were to draw them. That means it’s working!

Now before you go and add classes to every element, there’s an even easier way to handle this with Applitools, which is match levels.

First inside of the UI, we can preview this on our existing test.

Select the 3 dots icon under View, hover over Preview match level and select Layout.

Once the test finishes loading, like magic, all of our issues are gone.

And it’s not because we’re ignoring these changes, it’s because Applitools is able to determine the difference between a content change and a layout change.

It determined that all of those changes were indeed from content, resulting in no actual Layout issues.

But like before, we can configure this in the code to our liking.

We actually have two options here, where we can set our match level individually for each test, such as:

await eyes.check('Home', Target.window().fully().layout())

Or we can configure it directly on our Eyes instance by first importing the MatchLevel module:

import { Eyes, Target, MatchLevel } from '@applitools/eyes-playwright';

Then every time we create a new Eyes instance, right after it add:

eyes.setMatchLevel(MatchLevel.Layout);

And next time we run our tests, they should “magically” disappear.

Note: If you start to see 30s timeouts with this change, try reducing the number of browsers the tests run on or increase the timeout. Using the Classic test runner on the free tier has limited parallel connections, where upgrading to the Ultrafast Grid would allow more connections AND it would run the tests with the Applitools cloud.

While match level is likely the best solution for our content changes, being able to have both the option to ignore and set the match level gives us different tools to solve different problems.m

Learn more about the different match levels over on the Applitools Docs.

Follow along with the commit!

What else can we do?

Automate tests with GitHub Actions

In an ideal world, these tests like any other test would run on CI/CD.

GitHub Actions or other CI/CD tools are a perfect way to automate this, making sure the tests are always ran in a consistent environment, not slowing you down in your work.

Learn how to link Actions over on the Applitools blog.

Better team workflows with branching and GitHub

Applitools has a branching mechanism that works hand-in-hand with GitHub, even including merges and conflict resolution.

By installing the Applitools GitHub Integration, you’ll be able to take advantage of more features to integrate into your team’s workflow.