When AutoFix proposes a change to your code, we do not accept it because a model says it is right, and we do not accept it because the tests went green once. We accept it when your test suite passes three times in a row, each time from a clean checkout, with a fresh database.
That sounds like caution for its own sake. It is not. Each extra run catches a specific, common failure, and the third one catches the failure most teams never see coming.
Tests that pass because of what came before
Most test suites leave something behind. A row in a database, a file in a temporary folder, a cached token, a migration that only ran because the previous run created a table. The next run quietly depends on it. Everything is green, and nobody notices that the suite could not pass on a machine that has never run it before.
The same thing happens with order. Test B passes because test A happened to run first and set something up. Shuffle the order, or run B on its own, and it fails.
A fix that is checked against a suite like that has not really been checked. It has been checked against the leftovers of the last run.
What "thrice, from clean" means
- A fresh checkout of the repository with the patch applied. Nothing carried over from earlier work.
- A fresh database, built from your migrations, not copied from the last run.
- The full suite, not just the tests near the change. A fix in one place can break something two folders away.
- Then the same again, from nothing. If the second run differs from the first, the suite depends on state, and the fix is not accepted until that is understood.
- Then a third time, from nothing again. Two runs catch a suite that depends on leftovers. A third catches the one that is simply unreliable - the race that resolves the right way two times out of three, the timeout that usually has enough headroom, the test that depends on the clock. A fix merged on the strength of a flaky green is a fix you will be debugging later, under worse conditions.
What it costs and what it buys
It costs two more runs of your suite. On most projects that is minutes, and we run them while you are doing something else.
What it buys is a fix you can merge without wondering. It also tends to surface problems that were already in the suite before we arrived. When that happens we tell you, because a suite that passes two runs out of three is a finding in its own right - it means your pipeline has been telling you things it cannot actually prove.
For larger engagements with a fixed-price fix sprint, the same rule decides when the work is done. The balance falls due when your suite passes three times in a row from a clean checkout in your own pipeline. Not on our machine, and not on our word. The terms say so in section 3.