flakelog

Every suite I have ever inherited had a handful of tests that failed occasionally and got re-run, and everybody knew which ones informally without anybody knowing the actual numbers. I wanted the numbers, because 'this test is flaky' is an opinion until you can say it failed nine times in the last two hundred runs on the same assertion. I wrote the first version in a weekend for my own suite, then kept it because I reached for it on the next project too. It is deliberately dumb — it does not run tests, it just remembers.

It solves the problem of arguing about flakiness from memory. It ingests test result files after each run, keeps a rolling history, and produces a ranked list of tests by instability — how often they flip state without a code change touching them. The output is intentionally blunt: a list of tests, a flip rate, and the assertion that most commonly fails. It is for teams who have a suite people have started ignoring, which is most teams eventually.

All of it. The ingestion for the two result formats I care about, the storage, the instability scoring and the terminal report. The scoring is the part with any judgement in it — a test that fails every time is not flaky, it is broken, and a test that fails once in five hundred runs is probably real, so the ranking has to sit between those and it took some tuning against suites I knew well. I also wrote the pipeline snippet that most people use to feed it, which is four lines and is the reason anyone adopts it.

It stores everything in a single SQLite file, which people ask about because it means the history lives wherever you put that file rather than in a service. That is intentional: a flakiness tool that requires a server is a project, and a project needs a sponsor. Keeping it to a file means a single engineer can start using it on a Tuesday without asking anyone. I also chose not to quarantine or re-run anything automatically. Every tool in this space eventually offers to hide the flaky test for you, and hiding it is how the suite got untrustworthy in the first place.

Attributing a flip to flakiness rather than to a real change is the whole problem, and my first version got it badly wrong by ignoring the commit. If a test starts failing because someone changed the code it covers, that is not flakiness, but from the result file alone it looks identical. What works now is that each run is recorded with its commit, and a test only counts as unstable when it changes state across runs of the same commit — which is a much smaller and much more honest signal, and required me to ask people to record the commit, which took some persuading.

I use it on every project I join, and it has twice found that a test everyone blamed was stable and the real culprit was a shared fixture two tests earlier. On the retail point-of-sale suite it is how I held the flake rate at one and a half percent, because I could see the drift within a day rather than at the end of a release. About two hundred stars and a handful of teams I know of; it is not popular and does not need to be.

Nobody on the code, but two colleagues on separate teams ran it against suites much larger than mine and their reports are why the scoring changed. One of them found that my report was unusable past about fifty tests and made me add the cutoff.