- Front page
- Playbooks
- Flaky test hunt
Playbooks · Code
Flaky test hunt
When a test fails at random, the agent reruns it, finds the pattern and proposes a fix on a branch.
Paste this into your agent
When a test fails in CI and then passes on a rerun, record it. Run the suspect test in isolation enough times to confirm it is flaky, then look for the cause, such as timing, shared state or test order. Propose a fix on a new branch, with a pull request laying out the evidence. Do not disable, skip or delete tests, and do not merge or push to the main branch.What the agent does
Spot tests that fail and then pass
Rerun the suspect test in isolation
Trace the cause of the flakiness
Open a pull request with the fix and the evidence
Rules to add
- Changes go on a branch and wait for review
- No disabling, skipping or deleting tests
- No new packages without your approval
What you end up with: Flaky tests found, explained and fixed in a pull request you can review.