Flaky Tests
debt(d7/e5/b7/t7)
Closest to 'only careful code review or runtime testing' (d7). The detection_hints note that flaky tests manifest as tests passing locally but failing in CI, or failing ~20% of the time. Tools like phpunit, jest, github-actions, and buildkite can log failures, but a test must fail multiple times across multiple runs before the pattern is recognized as flakiness rather than a real regression. No single run catches it; it requires observing repeated runs and correlating results over time.
Closest to 'touches multiple files / significant refactor in one component' (e5). The quick_fix says to quarantine flaky tests immediately into a separate CI job, but that is only the triage step. Actually fixing the root causes — wrapping database tests in transactions, mocking clocks, eliminating test-order dependencies, removing real external API calls — typically requires changes across multiple test files and potentially test infrastructure (factories, fixtures, helpers), making it a meaningful multi-file effort rather than a trivial one-line patch.
Closest to 'strong gravitational pull' (d7). Flaky tests impose a persistent productivity tax on the entire team: every CI run is suspect, developers re-run pipelines instead of investigating, and real regressions get dismissed as flakiness. The why_it_matters field explicitly states this dynamic — teams stop trusting CI entirely. This shapes how every developer on the team responds to failures, making it a broad, gravitational burden across all CI-touching workflows.
Closest to 'serious trap' (t7). The misconception field directly states the canonical wrong belief: that a 10% failure rate is acceptable. Developers intuitively think '90% pass rate is fine,' but the combinatorial math means a 100-test suite with one 10%-flaky test will produce a CI failure roughly every 10 runs, steadily eroding trust and masking real regressions. This contradicts developer intuition about acceptable reliability thresholds and is a well-documented, widely held misconception.
Also Known As
TL;DR
Explanation
A flaky test is one that produces different results on repeated runs without code changes. Common causes: tests sharing database state, time-dependent assertions, race conditions in async code, external service calls, random data without fixed seeds, and test order dependencies. Flaky tests erode team trust — when a build fails, developers assume flakiness rather than investigating the actual failure. The fix is isolation: each test must set up and tear down its own state completely.
Diagram
flowchart TD
TEST[Test runs] --> RESULT{Result}
RESULT -->|sometimes pass| FLAKY[Flaky Test]
RESULT -->|sometimes fail| FLAKY
subgraph Root Causes
TIME[Time-dependent<br/>new Date hardcoded]
ORDER[Order-dependent<br/>shared state between tests]
ASYNC[Async timing<br/>no proper await]
EXTERNAL[External dependency<br/>real API call]
RANDOM[Random data<br/>no fixed seed]
end
FLAKY --> ROOT[Find root cause]
ROOT --> FIX[Fix: mock time, isolate state<br/>proper waits, stub externals]
style FLAKY fill:#f85149,color:#fff
style FIX fill:#238636,color:#fff
Common Misconception
Why It Matters
Common Mistakes
- Not wrapping database tests in transactions that roll back — test data from one test affects the next.
- Using time() or new DateTime() in tests without mocking the clock — passes at 11:59pm, fails at midnight.
- Test order dependencies — test B relies on side effects from test A; run in isolation to find it.
- Calling real external APIs in unit or integration tests — network flakiness causes spurious failures.
Code Examples
// Time-dependent assertion — fails near midnight:
public function testCreatedToday(): void {
$user = User::create(['name' => 'Alice']);
$this->assertEquals(
date('Y-m-d'), // Current date at test run
$user->created_at->toDateString() // May differ if test crosses midnight
);
}// Fixed time — deterministic:
public function testCreatedToday(): void {
Carbon::setTestNow('2026-01-15 12:00:00'); // Mock the clock
$user = User::create(['name' => 'Alice']);
$this->assertEquals('2026-01-15', $user->created_at->toDateString());
Carbon::setTestNow(null); // Reset
}