Flagger Canary Deployments: Complete Implementation Guide

Integrating with Linkerd Instead of Istio

Flagger works seamlessly with Linkerd. The configuration differs primarily in the mesh provider setting and metric queries:

# Install Flagger for Linkerd
helm upgrade --install flagger flagger/flagger \
  --namespace flagger-system \
  --create-namespace \
  --set meshProvider=linkerd \
  --set metricsServer=http://prometheus-kube-prometheus-prometheus.monitoring:9090 \
  --wait

# Linkerd-specific metric queries in the Canary resource
metrics:
- name: request-success-rate
  thresholdRange:
    min: 99
  interval: 30s
  query: |
    sum(
      rate(
        response_total{
          namespace="canary-demo",
          deployment="myapp",
          classification!="success"
        }[1m]
      )
    ) / sum(
      rate(
        response_total{
          namespace="canary-demo",
          deployment="myapp"
        }[1m]
      )
    ) * 100
- name: request-duration
  thresholdRange:
    max: 500
  interval: 30s
  query: |
    histogram_quantile(0.99,
      sum(
        rate(
          response_latency_ms_bucket{
            namespace="canary-demo",
            deployment="myapp"
          }[1m]
        )
      ) by (le)
    )

NGINX Ingress Controller Integration

For environments without a service mesh, Flagger can use the NGINX ingress controller for traffic splitting:

# Install Flagger with NGINX support
helm upgrade --install flagger flagger/flagger \
  --namespace flagger-system \
  --create-namespace \
  --set meshProvider=nginx \
  --set ingressClass=nginx \
  --set metricsServer=http://prometheus-kube-prometheus-prometheus.monitoring:9090 \
  --wait

# Canary configuration for NGINX
spec:
  service:
    port: 80
    hosts:
    - myapp.example.com
  analysis:
    interval: 30s
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 30s
      query: |
        100 - sum(
          rate(
            nginx_ingress_controller_requests{
              namespace="canary-demo",
              service="myapp",
              status!~"2[0-9][0-9]"
            }[1m]
          )
        ) / sum(
          rate(
            nginx_ingress_controller_requests{
              namespace="canary-demo",
              service="myapp"
            }[1m]
          )
        ) * 100

Best Practices for Flagger Canary Deployments

1. Define Meaningful Metric Thresholds

Your metric thresholds should reflect real SLOs (Service Level Objectives). If your service targets 99.9% availability, set the success rate minimum to 99.9. If your P99 latency SLO is 200ms, set the request duration maximum to 200. Meaningless thresholds lead to false positives or, worse, promoting broken releases.

# Example of well-tuned thresholds aligned with SLOs
metrics:
- name: request-success-rate
  thresholdRange:
    min: 99.9  # Matches your SLO of 99.9% availability
  interval: 30s
- name: request-duration
  thresholdRange:
    max: 200   # Matches your P99 latency SLO of 200ms
  interval: 30s

2. Use StepWeights That Match Your Traffic Pattern

Aggressive step weights (like jumping directly to 50%) work for low-risk changes. For high-risk database migrations or API changes, use gradual steps:

# Conservative progression for high-risk changes
stepWeights: [1, 5, 10, 30, 70, 100]

# Moderate progression for normal releases
stepWeights: [5, 20, 50, 100]

# Quick progression for low-risk configuration changes
stepWeights: [10, 50, 100]

3. Combine Multiple Metrics for Robust Analysis

A single metric can miss problems. Combine success rate, latency, and resource utilization for a comprehensive health picture:

metrics:
- name: request-success-rate
  thresholdRange:
    min: 99
  interval: 30s
- name: request-duration-p99
  thresholdRange:
    max: 500
  interval: 30s
- name: cpu-usage
  thresholdRange:
    max: 80
  interval: 30s
  query: |
    avg(
      rate(
        container_cpu_usage_seconds_total{
          namespace="canary-demo",
          pod=~"myapp-canary-.*"
        }[1m]
      )
    ) * 100
- name: memory-usage
  thresholdRange:
    max: 1024
  interval: 30s
  query: |
    avg(
      container_memory_working_set_bytes{
        namespace="canary-demo",
        pod=~"myapp-canary-.*"
      }
    ) / 1024 / 1024

4. Implement Pre-Rollout Webhooks for Integration Testing

Run integration tests against the canary pod before any production traffic reaches it. This catches configuration errors, missing dependencies, and startup failures early:

webhooks:
- name: smoke-test
  type: pre-rollout
  url: http://smoke-tester.canary-demo:8080/test
  timeout: 3m
  metadata:
    testPlan: canary-smoke
    endpoint: http://myapp-canary.canary-demo:80
- name: migration-check
  type: pre-rollout
  url: http://db-migration-checker.canary-demo:8080/verify
  timeout: 5m

5. Set Appropriate Analysis Windows

The analysis interval and threshold together determine how long a canary must prove itself. For a service with 30s interval and threshold of 5, the minimum successful analysis period is 150 seconds (5 * 30s). Choose values that balance speed with safety:

# Fast canary for low-risk services (2.5 minutes minimum)
analysis:
  interval: 30s
  threshold: 5

# Extended analysis for critical services (10 minutes minimum)
analysis:
  interval: 60s
  threshold: 10

6. Namespace and Label Organization

Keep your Canary resources, MetricTemplates, and AlertProviders logically organized by namespace. Use consistent labels to filter Flagger's monitoring:

# Label your namespaces consistently
kubectl label namespace canary-demo \
  progressive-delivery=enabled \
  team=backend \
  environment=production

# Use selectorLabels in Flagger's Helm values
--set selectorLabels="app.kubernetes.io/name,app.kubernetes.io/instance"

7. GitOps Integration with Flux

Flagger is part of the Flux ecosystem. Store your Canary definitions in Git alongside your application manifests. Flux automatically reconciles changes, and Flagger picks up image updates from the Git repository:

# Example Flux HelmRelease for Flagger
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: flagger
  namespace: flagger-system
spec:
  interval: 5m
  chart:
    spec:
      chart: flagger
      version: 1.33.x
      sourceRef:
        kind: HelmRepository
        name: flagger
        namespace: flagger-system
  values:
    meshProvider: istio
    metricsServer: http://prometheus-kube-prometheus-prometheus.monitoring:9090

8. Monitor Flagger Itself

Flagger exports Prometheus metrics about its own operations. Set up alerts for Flagger failures to know when the canary system itself has issues:

# Sample Prometheus alert rule for Flagger
groups:
- name: flagger-alerts
  rules:
  - alert: FlaggerRolloutFailed
    expr: |
      increase(flagger_canary_rollout_failed_total[5m]) > 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Flagger canary rollout failed"
      description: "Canary {{ $labels.name }} in namespace {{ $labels.namespace }} has failed"
  - alert: FlaggerControllerDown
    expr: |
      up{job="flagger"} == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Flagger controller is not running"

Troubleshooting Common Issues

Canary Stuck in Progress

If a canary remains in "Progressing" phase indefinitely, check:

# Check the Canary resource events
kubectl describe canary myapp -n canary-demo

# Verify Prometheus connectivity
kubectl logs -n flagger-system -l app.kubernetes.io/name=flagger | grep -i prometheus

# Check if the VirtualService was created
kubectl get virtualservices.networking.istio.io -n canary-demo

# Ensure the canary deployment exists and pods are running
kubectl get pods -n canary-demo -l app=myapp

Metric Queries Returning No Data

Test your Prometheus queries directly in the Prometheus UI before adding them to a Canary resource. Ensure namespace and service name filters match exactly:

# Port-forward Prometheus to test queries
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090

# In another terminal, test your query
curl -s 'http://localhost:9090/api/v1/query?query=sum(rate(istio_requests_total{destination_service_namespace="canary-demo"}[1m]))' | jq

Traffic Not Shifting

Verify that your service mesh is correctly configured and the gateway exists:

# For Istio, check the gateway
kubectl get gateway -n istio-system

# Verify sidecar injection
kubectl get pods -n canary-demo -o json | jq '.items[].spec.containers[].name' | grep istio-proxy

# Check DestinationRule weights
kubectl get destinationrules.networking.istio.io -n canary-demo -o yaml | grep -A 10 "weight"

Conclusion

Flagger transforms Kubernetes deployments from a binary pass/fail operation into a graduated, metric-driven progressive delivery pipeline. By integrating with your service mesh and monitoring stack, Flagger automates the most critical phase of software delivery—validating new versions against real traffic—while providing safety nets through automatic rollback. The combination of customizable metric thresholds, flexible traffic step weights, and webhook-driven testing gates gives teams confidence to deploy frequently without sacrificing reliability. Whether you use Istio, Linkerd, NGINX, or another provider, Flagger's consistent Canary resource model means you define your deployment strategy once and apply it across your entire service fleet. Start with conservative thresholds and step weights, build robust pre-rollout testing webhooks, and integrate Flagger into your GitOps workflow for a complete progressive delivery system that keeps your users safe while accelerating your release velocity.