Leader Election in Distributed Systems
debt(d9/e7/b7/t7)
Closest to 'silent in production until users hit it' (d9). No detection tools are listed in detection_hints. Duplicate job execution from missing or broken leader election — like double-charging customers or duplicate emails — surfaces only when users report problems or when logs are audited post-incident. No static analysis, linter, or runtime warning catches the absence of proper leader election.
Closest to 'cross-cutting refactor across the codebase' (e7). The quick_fix describes needing a database-backed lock with heartbeat renewal for PHP, or Kubernetes Lease objects — both require infrastructure changes (shared external store, TTL/heartbeat logic) plus code changes across every worker entry point. Moving from a naive file lock or missing lock to a proper Redlock or etcd-based solution touches deployment config, worker code, and potentially monitoring, making this a cross-cutting concern.
Closest to 'strong gravitational pull' (e7). The choice applies to cli and queue-worker contexts and shapes how every scheduled task and background worker must be written and deployed. Once a system runs on multiple servers or Kubernetes pods, every new cron job or worker must account for leader election. The pattern defines how all distributed coordination is handled, imposing a persistent tax on every future worker implementation.
Closest to 'serious trap (contradicts how a similar concept works elsewhere)' (t7). The misconception field is explicit: developers assume Redis SETNX is sufficient for production leader election, but it fails silently on leader crash (lock expires without a new election being safe). The common mistakes reinforce multiple layered traps — file locks that seem like they'd work don't across pods, TTLs that seem safe aren't without heartbeats. These contradict intuitions from single-server development where file locks and simple mutexes behave as expected.
Also Known As
TL;DR
Explanation
Many distributed tasks require exactly one node to act at a time: running a cron job, being the primary database writer, holding a distributed lock, or managing partition assignment. Leader election ensures exactly one node holds the leader role at any time. Common implementations: Raft consensus (used by etcd, Kubernetes) elects a leader as part of the consensus protocol itself. ZooKeeper ephemeral nodes — the node that creates an ephemeral znode becomes leader; if it crashes, the znode is deleted and re-election begins. Redis-based locks (Redlock or a single SETNX with TTL) are simpler but have edge cases. Kubernetes Lease objects provide leader election for controller pods. The key concerns are: ensuring only one leader at a time, detecting leader failure promptly, and avoiding split-brain (two nodes both believing they are leader).
Common Misconception
Why It Matters
Common Mistakes
- Using a file-based lock — file locks do not work across multiple servers or Kubernetes pods; use a shared external store (database, Redis, etcd).
- Not handling lock expiry — if the leader crashes without releasing the lock, the lock must expire automatically; always set a TTL and have a background process clean up expired locks.
- Setting too long a TTL — a 24-hour TTL means a crashed leader blocks the job for 24 hours; use a short TTL with a heartbeat renewal.
- Assuming Redis SETNX is atomic across a cluster — in Redis Cluster, SETNX on a key only locks that key's slot; use Redlock for true multi-node safety.
Code Examples
<?php
// ❌ No locking — all 3 pod replicas run the same cron job
// kubernetes CronJob with 3 replicas:
// Each pod runs sendWeeklyEmails() independently
// Result: every user receives 3 emails
class WeeklyEmailJob
{
public function run(): void
{
// No distributed lock — all pods execute this
foreach (User::all() as $user) {
Mail::to($user)->send(new WeeklyDigest());
}
}
}<?php
// ✅ Database-backed distributed lock with TTL
class WeeklyEmailJob
{
public function run(): void
{
$lockKey = 'weekly_email_job';
$lockTTL = 3600; // 1 hour max runtime
$acquired = DB::table('distributed_locks')->insertOrIgnore([
'key' => $lockKey,
'acquired_by'=> gethostname(),
'acquired_at'=> now(),
'expires_at' => now()->addSeconds($lockTTL),
]);
if (!$acquired) {
Log::info('Cron lock not acquired — another instance is running');
return;
}
try {
foreach (User::all() as $user) {
Mail::to($user)->send(new WeeklyDigest());
}
} finally {
DB::table('distributed_locks')
->where('key', $lockKey)
->delete();
}
}
}