How often have you had a test fail in your test suite because of some distant setup in the test suite? Maybe you’ve got seed data you weren’t expecting, or you’ve got wrapping contexts that you “just have to remember are there.” Or maybe you have flaky tests stemming from leftover data between tests. These situations could be a case of an Obscure Test.

A remedy I often recommend for this testing smell is simple: “every test should start with a clean slate”.

Another way to interpret this is: the test harness contains all the setup necessary for the test to pass. You may have also heard this called “Test Isolation” or “Hermetic Testing.” If we’re being deeply pedantic, I do actually think you should get as close to zero-state as possible prior to a test running (though there are cases where that may not be sensible). Your system may have “sensible defaults” that you can depend on (say, a few database seeds, or similar). Generally, however, the local test context should import/invoke whatever setup is necessary to each test to pass.

Why?!

Prior setup means that tests that don’t need it, get it anyway. This slows down boot up time for the harness/suite. It also pollutes the environment, violating the Principle of Least Surprise.

By minimizing the amount of unseen setup, you increase debuggability and readability. Additionally, tests that only rely on their own data are easier to parallelize, increasing the speed at which you can run your suite.

Remember, your tests are a system, too - they just happen to depend on your production system (much like your users, no?)

Examples:

Database seeds

Favor a test invoking seeds, over global setup. This ensures tests that don’t need the setup don’t get it, and keeps global state clean.

# less ideal

# Sets up state in a global/test-harness-level place
RSpec.configure do
  config.before(:all) do
    UnitedStates.import_the_50_nifty!
  end
end

RSpec.describe "United States" do
  it "has 50 states" do
    expect(UnitedStates.count).to eq 50
  end
end

# more ideal

# All necessary test data is visible in the test file
RSpec.describe "United States" do
  config.before do
    UnitedStates.import_the_50_nifty!
  end

  it "has 50 states" do
    expect(UnitedStates.count).to eq 50
  end
end

Env variable state

Favor explicitly calling out the values you want

Imagine a function like the following:

const envAdd = (num) => parseInt(process.env.FOO, 10) + num;

One way you might write a test for this could depend on a default being set prior to the test running and just assert on the value. This is less ideal, though because it’s sensitive to the runner’s machine state.


describe(envAdd, () => {
  it("uses the value of env.FOO", () => {
    expect(envAdd(2)).toEqual(4);
  });
})

// more ideal - makes the dependency more visible, ensures cleanup

describe(envAdd, () => {
  let originalEnv;

  beforeEach(() => {
    originalEnv = process.env.FOO;
    process.env.FOO = "2";
  });

  afterEach(() => {
    process.env.FOO = originalEnv;
  });

  it("adds the numeric value of env.FOO to the passed value", () => {
    expect(envAdd(2)).toEqual(4);
  });
})

// possibly even more ideal - makes the expected value inside the it block

describe(envAdd, () => {
  it("adds the numeric value of env.FOO to the passed value", () => {
    withOverriddenEnv({FOO: "2"}, () => {
      expect(envAdd(2)).toEqual(4);
    })
  });

  function withOverriddenEnv(overrides = {}, operation = () => {}) {
    try {
      const originalEnv = process.env;
      process.env = {...process.env, ...overrides};
      operation();
    } finally {
      process.env = originalEnv;
    }
  };
})

// most ideal - envAdd accepts configuration for source, removes global dependency
// heck, you might even chose to just pass in FOO
// you know your application and usage patterns better than I do

const envAdd = (num, env = process.env) => parseInt(env.FOO, 10) + num;

describe(envAdd, () => {
  it("adds the numeric value of env.FOO to the passed value", () => {
    expect(envAdd(2, { FOO: "2" })).toEqual(4);
  });
})

Reasons you might break this rule

  1. Performance

    If your test suite depends on, for example, database seeds that basically every test uses, you may wish to globally seed them. You should still have a way to run tests that don’t depend on the database without these seeds.

    If setting up some test state is costly though, you may want to consider your architectural decisions. Is there a way to factor your code so you can test the logical parts without the expensive parts?

  2. Sensible defaults

    Enough tests in your test suite or your business model expect something to be set one way, and one area wants a different setting? It may be worth the cost of adjusting the default in that one area to provide a reasonable ground-truth setup for most of the application.

    One way this might show up is with truly immutable seed data. Suppose your application has some “dictionary” tables (what some domains call concept or vocabularies), that much or all of your application uses in its database tests. These might be worth pre-“seeding” at the beginning of your test suite for increased per-test speed.

  3. End-to-end (e2e) test suites

    This is mostly a subset of the performance case, but it’s worth calling out on its own. Your e2e test suites often need to balance performance, correctness, and expressiveness. While it may make it more difficult to debug, the increased speed from not setting up deeply complex context on every test may make it worthwhile.

  4. Legacy codebases

    Finally, sometimes you just have a codebase that’s already broken this pattern. It may be worth evaluating if a cost-benefit suggests you should undo that or not. Is there a section of your codebase you could carve off that’s easier to work with by keeping a clean slate?

Conclusion

Next time a test surprises you because of some unseen data or Mystery Guest, consider starting with a clean slate. Just because they’re tests doesn’t mean they can’t benefit from some architecture themselves. As we strive for in our production software, we should strive to make our tests easier to understand. By investing in “clean slate” tests, you make your suite easier to read, debug, and (maybe) faster. It’s an investment that can pay dividends in productivity and confidence.