Test suite failing when running all tests and tests passing when testing individually

Question

I am running a test suite with jest and puppeteer with 4 tests in it. When I run all of them 2 passes and 2 fails. When I run each one individually, they all pass. I tried reloading the page before each test, but then, all of the 4 tests fails. Did someone experience this? How was it solved?

import ChangePasswordPage from "../pages/ChangePasswordPage";
import 'expect';
import {login} from "../utils/loginUtils";
import Menu from "../pages/Menu";

let changePassword;

describe ('Change Password Flow', ()=> {
        beforeAll(async () => {
        let page = await login();
        let menu = new Menu(page);
        await menu.setUp();
        changePassword = new ChangePasswordPage(page);
        await changePassword.setUp();
    });

        test('new and confirm password dont match', async () => {
        await changePassword.fillChangePasswordForm('Test1234', 'Test4567', 'Test7890');
        await changePassword.submitChangePasswordForm();
        const changePasswordError= await changePassword.getNewAndConfirmPasswordDontMatch();
        await expect(changePasswordError).not.toBeNull();
    });

    test("new password don't meet requirement", async () => {
        await changePassword.fillChangePasswordForm('Test1234', '1234567890', '1234567890');
        await changePassword.submitChangePasswordForm();
        const page = changePassword.page
        await page.waitForTimeout(2000);
        const newPasswordRequirementError = await changePassword.getNewPasswordDontMeetRequirementsError();
        await expect(newPasswordRequirementError).not.toBeNull();
    });

    test('wrong current password', async () => {
        await changePassword.fillChangePasswordForm('currentPassword1', 'Test4567', 'Test4567');
        await changePassword.submitChangePasswordForm();
        const currentPasswordDontMatch = await changePassword.getWrongPasswordModalError();
        await expect(currentPasswordDontMatch).not.toBeNull();

    });

    test ('successful changed password', async () => {
        await changePassword.fillChangePasswordForm('Test1234', 'Test4567', 'Test4567');
        await changePassword.submitChangePasswordForm();
        const passwordChanged = await changePassword.getSuccessfulChangePassword();
        await expect(passwordChanged).not.toBeNull();

    });
});

Answer 1

It's hard to tell from your code example since it isn't complete, but the problem is almost certainly that all of the test blocks share the same underlying page or browser instance (or other stateful object), yet may be run concurrently or in an unintended order by Jest, causing nondeterministic interleaving and state mutations of the same shared data.

Even when run serially in a specified order and appearing to work, it's still problematic to share state between tests for the same reasons you want state to be local and (ideally) immutable. It's easier to isolate behavior (and problems) and reason about the code when side effects are contained to a single function, module or test case.

The correct way to write tests is to use beforeEach and afterEach to set up each test case with its own state, then completely destroy/clean up the data afterwards. This makes each test idempotent, easy to reason about and not dependent on a particular execution order.

For example, here's the incorrect pattern:

describe("counter", () => {
  let counter;

  beforeAll(() => {
    counter = {
      count: 0,
      up() { this.count++; },
      get() { return this.count; },
    };
  });
  
  it("should increment to 1", () => {
    counter.up();
    expect(counter.get()).toEqual(1);
  });
  it("should increment to 2", () => {
    counter.up();
    expect(counter.get()).toEqual(2);
  });
});

The correct pattern is:

describe("counter", () => {
  let counter;

  beforeEach(() => {
    counter = {
      count: 0,
      up() { this.count++; },
      get() { return this.count; },
    };
  });
  afterEach(() => {
    // contrived here, but in Puppeteer `page.close()`/`browser.close()`,
    // or do whatever you need to tear down your state to make
    // sure it won't leak over to the next test case
    counter = null;
  });
  
  it("should increment to 1", () => {
    counter.up();
    expect(counter.get()).toEqual(1);
  });
  it("should increment to 2", () => {
    counter.up();
    counter.up();
    expect(counter.get()).toEqual(2);
  });
});

As you can see, each test needs to rebuild all of the state from scratch, which can have performance impacts and likely requires more layers of setup code than shown here. Nonetheless, you'll need to make sure this setup code doesn't interfere with shared state, which includes persistent database operations.

If the setup and teardown are prohibitively expensive, it's common to share a single browser instance between tests, or even a common page instance, still taking care to make sure the page state is reset, perhaps with a goto, sufficient to guarantee no data leakage.

If there's any doubt, start by closing out everything to isolate the bug, then slowly reintroduce shared (but effectively immutable) state as needed to increase speed.

---

As a Puppeteer-specific aside, await page.waitForTimeout(2000); is almost always a poor substitute for page.waitForFunction or page.waitForSelector. Waiting for timeouts causes a race condition at worst and unnecessarily slow tests at best. I suggest finding a selector or predicate to await on.

As a Jest-specific aside, expect doesn't return a promise so the await on every expect call is pointless.