Canary Deployment & Observability

TL;DR

Explanation

Canary: deploy new version to 1-5% of traffic. Monitor: error rate, latency p99, business metrics (conversion, payment success). Automated rollback if canary metrics degrade vs baseline. Implementation: load balancer weights (nginx upstream, AWS ALB weighted target groups), Kubernetes (Argo Rollouts, Flagger), feature flags. Key metrics to compare: error rate, latency (all percentiles), business KPIs. Duration: 10-30 minutes for fast feedback, hours for low-traffic metrics. Observability requirement: metrics must be tagged with version to split canary vs stable. Autocanary: Flagger automates comparison and rollback.

Common Misconception

Why It Matters

Common Mistakes

• No version labels on metrics — can't compare canary vs stable.
• Canary period too short — not enough traffic for statistical significance.
• Manual rollback only — automate based on error rate threshold.

Code Examples

# Deploy to 5% but no metric comparison:
upstream backend {
    server stable weight=95;
    server canary weight=5;
}
# No monitoring — how do you know if canary is healthy?

# Flagger automated canary:
analysis:
  interval: 1m
  threshold: 5
  metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
    - name: request-duration
      thresholdRange:
        max: 500
# Auto-rollback if success rate < 99% or p99 > 500ms

References

• ↗ https://flagger.app/docs/usage/how-it-works

Tags

Related Terms