Flakiness and the quarantine

How a flake is detected

Both sides of a story are captured moments apart in the same browser, and compared by an exact hash of their raw pixels. Equal means unchanged, and nothing further is computed.

shot(before), shot(after)      # sha256 over the raw pixels
if hashes equal      -> unchanged
else:
  sleep(delay)                 # 250ms by default, per-story overridable
  re-shoot both, re-hash
  if now equal       -> unchanged, recorded as flaky
  else               -> changed; compute the metrics

The second round is what separates “this changed” from “this had not finished settling”. If the two sides agree the second time, the story is recorded as flaky and treated as unchanged for this run: it does not appear as a difference, it does not go to the review stage, and it does not affect the check.

Only a story that still differs after the re-shoot is a real difference, and only then is anything expensive computed for it.

What makes a story flaky

Most of the usual suspects are already handled. Animations and transitions are disabled, the caret is hidden, prefers-reduced-motion is pinned to reduce, fonts and images are awaited, the network is quiet because there is no network, and Date and Math.random are pinned identically on both sides. What is left is usually one of:

• an animation driven by a timer rather than by CSS, so nothing declarative stops it;
• content that arrives after the story reports itself rendered — a lazily-imported chart, a deferred image, a fetch mocked with a delay;
• a play() function racing the capture;
• randomness the story generates itself rather than drawing from the seeded source.

The first fix is usually a per-story waitFor selector the story sets once it is genuinely ready.

export const Ticker = {
  parameters: { merrykat: { waitFor: '[data-settled="true"]' } },
};

If that doesn’t stabilize the story you can override the delay.

export const Slow = {
  parameters: { merrykat: { delay: 500 } },
};

The quarantine

A single unlucky capture says nothing, so nothing happens on the strength of one. The flake rate is tracked per (story, viewport) across runs, and a story enters quarantine once it has flaked at least twice and in at least 15% of the runs it took part in. Both conditions are needed: the count stops one bad afternoon from slowing a story down permanently, and the rate stops a story compared hundreds of times from being punished for early stumbles.

A quarantined story is rendered differently, not skipped:

• its settle time is raised to at least one second, whatever the configured or per-story delay was;
• it gets an extra attempt — two retries rather than one — before a difference is believed;
• it is named in the pull request comment, and listed on the repository’s quarantine page in the app.

A story that flakes every run is a bug in the story. The quarantine buys you a correct comparison in the meantime, not a permanent home.

Getting back out

Nothing needs to be reset by hand. The counters decay: once a story has enough history, each new observation halves both of its tallies before adding itself to them. A story that has genuinely stabilised falls back under the rate threshold within a few dozen runs however deep its bad history goes, while a story that is still flaky is still caught within a couple of runs.

What it does not do

Quarantine changes how a story is captured. It never changes how a difference is judged, never suppresses a real regression, and never excludes a story from the run: a quarantined story that changed is reported like any other.