Fix the Flaky Test at Its Cause - Vets Who Code

New Cohort Starts: Apr 05, 2027
A flaky test is a test that passes and fails on the same code. Every team has a few, and every team has the same three bad fixes: rerun the pipeline, add retry: 3, or skip the test "for now." Each one trains the team to ignore red builds. Sooner or later a real failure gets rerun too.
This is the fourth post in our series on how Vets Who Code teaches code review, debugging and testing.
On the testing foundations domain page, the topic Diagnose flaky tests is described as "removes nondeterminism from a test suite." Its evidence criterion:
Fixes an intermittent failure at its cause
The words that matter are at its cause. A retry makes the build green and leaves the cause in place. The criterion asks for the opposite: find what varies between runs, and remove the variation.
Here's a function that labels tasks in a to-do list, and the test that came with it:
CI is green on every run. One developer in Chicago sees it fail before lunch and pass after dinner:
The first instinct is to run it in a loop:
On the same machine at the same time of day, it gives the same answer 50 times. That's useful. It means the variable isn't the run. Nondeterminism in tests comes from a short list of places. Go down the list in order:
This test calls new Date() twice and touches a date string. Start at the top.
If time is the variable, control time. Pin the clock with fake timers and run the test at two hours of the same day:
Then pin the time zone too:
Now it's deterministic. It fails every time under the conditions that matter, which means you can fix it and prove the fix.
There are two clocks in this code, and they disagree.
In the morning, the test builds "2026-03-10", the code reads it as March 9 local and calls it overdue. In the evening, the two UTC conversions cancel out by accident and the test passes. CI runs in UTC, where there's nothing to cancel, so it always passes.
So the flakiness wasn't the real problem. The code has a time zone bug: for anyone west of Greenwich, a task due today reads as overdue, all day long. The test only caught it in the morning because it built its input with the same UTC mistake, which cancelled the bug out every evening. A retry would have hidden a production bug behind a green build.
Parse a date-only string as a local date, so it's on the same clock as now:
Then make the test say what it means. Use a literal date instead of deriving one from the clock, pin the clock, and keep both hours:
It passes in Chicago and in UTC, morning and evening. Run the suite under a non-UTC TZ in CI as well, so the next time zone bug fails where everyone can see it.
If the loop in Step 1 had failed only sometimes, suspect order. Vitest can shuffle test order and report the seed, so you can replay the exact order that failed:
An order-dependent failure almost always traces to state that outlives a test: a module-level cache, a mock that's never restored, a database row nobody cleans up. The fix is the same shape as above. Find the shared thing and give each test its own.
Model output is nondeterministic on purpose, so tests for AI features need a different tool. Troops learn that on the evals domain page, where Write deterministic checks is judged by whether a troop "catches malformed output without a model." Building with AI doesn't excuse a flaky suite. It raises the bar for knowing which parts of the system should be deterministic.
A PR that fixes a flaky test should explain the cause and include the reproduction: the TZ, the seed, the pinned time. "Added a retry" is not a fix. "The code parsed date-only strings as UTC; this pins the clock and fixes the parse" is.
Next in the series: Tests That Survive a Refactor.
Vets Who Code is free for every troop, and it stays that way because people like you keep it running. If this post was useful, Donate or sponsor us on GitHub.
Vets Who Code is a free, remote software engineering accelerator for U.S. veterans, service members and spouses. If you know one who's ready to become a software engineer, send them this post and tell them to Apply. We'll take it from there.
31860 Sandy Plains Road, Ste 204 PMB 206
© 2026 Vets Who Code — All Rights Reserved
Vets Who Code is a registered 501(c)(3) nonprofit — EIN 86-2122804. Donations are tax-deductible to the fullest extent allowable under the law.