Product case study · November 20, 2026

Ask Claude Code why a test passes alone but fails in a suite

Fix an order-dependent pytest failure with Claude Code. Verify environment cleanup and both test orders without changing working application code.

Reading time
6 min
Checked
Nov 20, 2026
Paper test trays and a blue toggle tile with a charcoal return arrow illustrating restoration between test cases.
Each test should leave the next one a clean starting point.
Bottom line

When a test passes alone, inspect what the previous test leaves behind before changing the application.

The default-setting test passes on its own. Run it after the override test and it fails. Before asking Claude Code to rewrite the settings reader, look at what the first test left behind.

This lab has a working application and broken tests. That distinction is the exercise: an agent can make a suite green by changing the wrong file. You’ll give it a narrower job, then check that the repair survives a different execution order.

You need Bash on Linux, macOS or WSL, Python 3 with venv, and an installed, signed-in Claude Code. The Python reproduction and reference repair were tested locally with Python 3.14.4 and pytest 9.2.0; the agent prompt is an authored exercise, not a recorded Claude run. Official sources were checked November 20, 2026.

Build the failure in a practice folder

Create an isolated Python environment and install pytest there:

env_lab=$(mktemp -d /tmp/claude-env.XXXXXX)
cd "$env_lab"
python3 -m venv .venv
.venv/bin/python -m pip install pytest

Save this as settings.py:

import os


def report_format():
    return os.environ.get("SUPERBASH_LAB_FORMAT", "text")

The contract is small: read the environment at call time, return its value when present, otherwise return "text". Python documents os.environ as the process environment mapping. Updating it changes what later calls in that process can read.

Save these deliberately flawed tests as test_settings.py:

import os
from settings import report_format


def test_override():
    os.environ["SUPERBASH_LAB_FORMAT"] = "json"
    assert report_format() == "json"


def test_default():
    assert report_format() == "text"

Run the default case alone, then explicitly run the override first:

env -u SUPERBASH_LAB_FORMAT .venv/bin/python -m pytest -q \
  test_settings.py::test_default

env -u SUPERBASH_LAB_FORMAT .venv/bin/python -m pytest -q \
  test_settings.py::test_override test_settings.py::test_default

The local baseline was one pass for the first command and one pass, one failure for the second. The failing assertion received "json" where it expected "text".

env -u removes this one variable from the child command’s starting environment. It does not prevent the first test from setting it again. Both tests in the second command run within the same pytest process, so the assignment survives into the default case.

This is why rerunning only the failed test can hide the problem. You have removed the test that causes it.

Give Claude the right repair boundary

Before opening the agent, save a separate test_cleanup.py:

import os


def test_environment_restored():
    assert "SUPERBASH_LAB_FORMAT" not in os.environ

This probe is specifically for commands launched with env -u. Run it after the other tests to catch a patch that merely hides the leaked value from the default assertion.

Start claude in the practice folder and send:

Read settings.py, test_settings.py and test_cleanup.py.
The default test passes alone but fails after test_override.
Reproduce both commands using .venv/bin/python -m pytest and
an initially absent SUPERBASH_LAB_FORMAT (env -u).
Explain which test changes shared process state before editing.

Repair only test_settings.py using pytest's monkeypatch fixture.
Keep both test names and both assertions unchanged.
Keep settings.py and test_cleanup.py unchanged.
Each test must establish the environment it needs and restore
what existed before it. Do not reorder tests, skip tests, clear
the whole environment, or change application behavior.

Verify override then default then the cleanup probe, and default
then override then the cleanup probe, each in a fresh process.
Also run just the two settings tests with the variable initially
set to csv. Report commands, exit codes and actual results.

Providing the reproduction command and concrete symptoms follows Anthropic’s debugging workflow guidance. The file boundary and acceptance conditions above are this lab’s instructions, not a guarantee that the agent will follow them. Review its diff.

I would reject a patch that hardcodes "text" in the application. The override is valid behavior. The test setup is what needs repair.

Compare the repair with a reference

A complete reference test_settings.py is:

from settings import report_format


def test_override(monkeypatch):
    monkeypatch.setenv("SUPERBASH_LAB_FORMAT", "json")
    assert report_format() == "json"


def test_default(monkeypatch):
    monkeypatch.delenv("SUPERBASH_LAB_FORMAT", raising=False)
    assert report_format() == "text"

The pytest monkeypatch guide documents setenv, delenv and automatic undo when the requesting test or fixture finishes. raising=False lets the default test remove a variable that may already be absent.

There are two separate fixes here. The override test stops leaving its setting behind. The default test explicitly creates the missing-variable condition, so it does not depend on your terminal’s configuration. Removing the variable manually at the end of a test is weaker: an earlier assertion failure can prevent that cleanup from running.

Verify the patch yourself

Run these commands after reviewing the changed file:

env -u SUPERBASH_LAB_FORMAT .venv/bin/python -m pytest -q \
  test_settings.py::test_override test_settings.py::test_default \
  test_cleanup.py::test_environment_restored

env -u SUPERBASH_LAB_FORMAT .venv/bin/python -m pytest -q \
  test_settings.py::test_default test_settings.py::test_override \
  test_cleanup.py::test_environment_restored

env SUPERBASH_LAB_FORMAT=csv .venv/bin/python -m pytest -q \
  test_settings.py

The reference repair produced three passes, three passes, and two passes, respectively. The last command deliberately excludes the cleanup probe, whose assertion requires an initially absent variable.

To check restoration of an existing value as well, run both tests inside a Python process that checks its environment after pytest returns:

env SUPERBASH_LAB_FORMAT=csv .venv/bin/python - <<'PY'
import os
import pytest

status = pytest.main(["-q", "test_settings.py"])
assert status == 0, status
assert os.environ["SUPERBASH_LAB_FORMAT"] == "csv"
print("original value restored")
PY

Locally, this printed original value restored after two passing tests. That is stronger evidence than merely seeing the default assertion pass: the original setting survives the whole run.

If your real suite still fails

If the default test fails by itself, check its starting environment first. If the repair passes these commands but your real project still fails, look for module-level cached settings, shared fixtures, or another test that writes the same variable directly. This fixture reads the environment inside the function; it does not cover applications that capture configuration during import.

Keep the before-and-after commands with the diff. A useful agent handoff explains which test polluted the process, why the application stayed unchanged, and how cleanup was checked. “Tests pass” leaves out the part that made this bug interesting.

Sources

Checked November 20, 2026. The examples and recorded local results are Superbash’s authored fixture, not vendor benchmark claims.

Put this to work

How a process environment change can leak between pytest tests.

Try

Use pytest's monkeypatch fixture in a disposable Claude Code practice folder.

Prove it worked

Keep the assertions and application fixed, then run both test orders and a cleanup probe.

Keep in view

  • Reproduce a test that passes alone and fails after another test.
  • Give Claude Code a repair limited to test setup and teardown.
  • Verify both execution orders and an inherited environment value.
Learn the workflow: Set up a Claude Code project session