NGINX Service Mesh: Complete Setup and Configuration Guide

What is NGINX Service Mesh?

NGINX Service Mesh is a lightweight, high-performance service mesh that provides infrastructure-level networking for microservices. Built on top of the proven NGINX reverse proxy and load balancer technology, it offers a sidecar-based data plane and a Kubernetes-native control plane. Unlike heavier mesh solutions such as Istio, NGINX Service Mesh prioritizes simplicity, minimal resource consumption, and ease of deployment while still delivering the core capabilities required for modern cloud-native applications.

At its heart, NGINX Service Mesh injects a tiny NGINX-based sidecar proxy (called nginx-mesh-sidecar) alongside each service pod. This sidecar transparently intercepts inbound and outbound traffic, enabling advanced routing, load balancing, mutual TLS encryption, and rich observability without requiring application code changes. The control plane, deployed as a set of Kubernetes controllers, manages configuration distribution, certificate rotation, and telemetry aggregation across the entire mesh.

Key Features at a Glance

Transparent sidecar injection — no manual pod modifications required
Mutual TLS (mTLS) — automatic encryption of all service-to-service traffic
Traffic splitting — canary deployments and A/B testing with fine-grained percentage control
Circuit breaking — resilience patterns to prevent cascading failures
Rate limiting — protect services from excessive traffic bursts
Observability — integrated metrics, tracing, and logging via Prometheus, Grafana, and OpenTelemetry
Low resource footprint — typically 10–15MB memory per sidecar, far less than Envoy-based alternatives
Kubernetes-native CRDs — configuration via familiar YAML manifests

Why NGINX Service Mesh Matters

In the landscape of service mesh technologies, NGINX Service Mesh occupies a compelling middle ground. It's more capable than simple ingress controllers or DNS-based service discovery, yet dramatically simpler to operate than heavyweight platforms like Istio. This matters for teams that need the reliability and security benefits of a mesh without the operational overhead of managing complex Envoy configurations, multi-component control planes, or deep observability pipelines.

The NGINX data plane brings a battle-tested codebase with decades of production hardening. Its event-driven, non-blocking architecture delivers exceptional throughput with minimal CPU and memory consumption. For organizations already using NGINX at the edge, adopting NGINX Service Mesh internally provides a consistent proxy fabric from ingress to service-to-service communication, simplifying troubleshooting and reducing cognitive load on operators.

Furthermore, NGINX Service Mesh is ideal for resource-constrained environments — edge deployments, IoT clusters, or small Kubernetes nodes where every megabyte of memory matters. Its sidecar footprint of approximately 10MB makes it feasible to run a full mesh even on single-board computers or lightweight container platforms like K3s.

Prerequisites and Installation

Before installing NGINX Service Mesh, ensure your environment meets these requirements:

Kubernetes cluster version 1.19+ (any CNCF-conformant distribution)
kubectl configured with cluster admin access
Helm 3 (optional but recommended for production deployments)
Container runtime supporting privileged mode (for sidecar injection via iptables)
At least 200MB free RAM and 1 CPU core available for the control plane components

Step 1: Download the CLI Tool

The nginx-meshctl CLI is the primary tool for installing and managing the mesh. Download the latest binary for your platform:

# Linux AMD64
curl -sL https://github.com/nginxinc/nginx-service-mesh/releases/download/v2.1.0/nginx-meshctl_linux_amd64.tar.gz | tar xz
sudo mv nginx-meshctl /usr/local/bin/
nginx-meshctl version

# macOS
curl -sL https://github.com/nginxinc/nginx-service-mesh/releases/download/v2.1.0/nginx-meshctl_darwin_amd64.tar.gz | tar xz
sudo mv nginx-meshctl /usr/local/bin/
nginx-meshctl version

Step 2: Install the Mesh in Your Cluster

NGINX Service Mesh deploys into a dedicated namespace (default nginx-mesh) and sets up the control plane plus sidecar injection infrastructure. Run the installation command:

nginx-meshctl install \
  --namespace nginx-mesh \
  --registry nginx-mesh \
  --image-pull-policy IfNotPresent \
  --mtls-mode strict \
  --telemetry-export-interval 30s

This command performs several actions automatically:

Creates the nginx-mesh namespace with required labels
Deploys the mesh controller, certificate authority, and telemetry aggregator
Sets up a MutatingWebhookConfiguration for automatic sidecar injection
Creates default traffic policies and mTLS configurations

Verify the installation:

kubectl get pods -n nginx-mesh
# Expected output:
# NAME                                  READY   STATUS    RESTARTS   AGE
# nginx-mesh-controller-xxxxx           1/1     Running   0          30s
# nginx-mesh-cert-ca-xxxxx             1/1     Running   0          30s
# nginx-mesh-metrics-xxxxx             1/1     Running   0          30s

kubectl get mutatingwebhookconfiguration nginx-mesh-sidecar-injector
# Should show the webhook configuration for automatic injection

Step 3: Enable Namespace Injection

Sidecar injection is namespace-scoped. Label the namespaces where you want automatic sidecar injection:

# Label your application namespace for injection
kubectl label namespace my-app nginx-mesh.inject=true

# Verify the label
kubectl get namespace my-app --show-labels
# Output: my-app   Active   1m   nginx-mesh.inject=true,kubernetes.io/metadata.name=my-app

Alternatively, you can selectively inject individual deployments by adding an annotation to the pod template:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service
  namespace: my-app
spec:
  template:
    metadata:
      annotations:
        nginx-mesh.inject: "true"
    spec:
      containers:
      - name: app
        image: my-app:1.0

Deploying Services into the Mesh

Once a namespace is labeled, any new pods created in that namespace will automatically receive an NGINX sidecar proxy. Let's deploy a sample microservices application consisting of a frontend, an API gateway, and two backend services.

Sample Deployment: Frontend Service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: my-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: nginx:alpine
        ports:
        - containerPort: 80
          name: http
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-svc
  namespace: my-app
spec:
  selector:
    app: frontend
  ports:
  - port: 80
    targetPort: http
    name: http

Sample Deployment: Backend API Service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-v1
  namespace: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
      version: v1
  template:
    metadata:
      labels:
        app: api
        version: v1
    spec:
      containers:
      - name: api
        image: my-api:1.0
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: VERSION
          value: "v1"
---
apiVersion: v1
kind: Service
metadata:
  name: api-svc
  namespace: my-app
spec:
  selector:
    app: api
  ports:
  - port: 8080
    targetPort: http
    name: http

After applying these manifests, check that sidecars have been injected:

kubectl get pods -n my-app
# Each pod should show 2/2 containers ready (app + nginx-mesh-sidecar)
# NAME                        READY   STATUS    RESTARTS   AGE
# frontend-xxxxx-yyyy         2/2     Running   0          10s
# frontend-xxxxx-zzzz         2/2     Running   0          10s
# api-v1-xxxxx-yyyy           2/2     Running   0          10s
# api-v1-xxxxx-zzzz           2/2     Running   0          10s
# api-v1-xxxxx-wwww           2/2     Running   0          10s

# Inspect the sidecar details
kubectl describe pod -n my-app frontend-xxxxx-yyyy | grep -A10 "nginx-mesh-sidecar"

Traffic Management: Splitting and Canary Deployments

One of the most powerful capabilities of NGINX Service Mesh is fine-grained traffic control. Using the TrafficSplit custom resource, you can route percentages of requests to different service versions without touching application code.

Deploying a New Version for Canary Testing

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-v2
  namespace: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
      version: v2
  template:
    metadata:
      labels:
        app: api
        version: v2
    spec:
      containers:
      - name: api
        image: my-api:2.0
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: VERSION
          value: "v2"

Note that the v2 deployment shares the same app: api label as v1 but adds version: v2. The existing Service selector app: api will now match both versions — but we'll use TrafficSplit to control distribution.

Creating a TrafficSplit Resource

apiVersion: spec.nginx-service-mesh/v1alpha1
kind: TrafficSplit
metadata:
  name: api-canary
  namespace: my-app
spec:
  service: api-svc
  backends:
  - serviceName: api-svc
    serviceNamespace: my-app
    weight: 90
    filter:
      label:
        version: v1
  - serviceName: api-svc
    serviceNamespace: my-app
    weight: 10
    filter:
      label:
        version: v2
  rules:
  - type: http-header
    condition:
      header: x-canary
      value: "true"
    action:
      routeTo:
        serviceName: api-svc
        serviceNamespace: my-app
        filter:
          label:
            version: v2

This configuration achieves several goals simultaneously:

90% of traffic flows to v1 pods, 10% to v2 pods (gradual canary rollout)
Requests with the HTTP header x-canary: true are always routed to v2 (for developer testing)
The split is enforced by the sidecar proxies transparently — no application logic changes

Apply and verify the TrafficSplit:

kubectl apply -f traffic-split.yaml

# Check the status
kubectl get trafficsplit -n my-app
# NAME         SERVICE     AGE
# api-canary   api-svc     5s

# Describe the split for details
kubectl describe trafficsplit api-canary -n my-app

Testing the Traffic Distribution

# From the frontend pod, make 100 requests and count versions
kubectl exec -n my-app frontend-xxxxx-yyyy -c frontend -- \
  sh -c 'for i in $(seq 1 100); do curl -s api-svc:8080/version; done' | sort | uniq -c

# Expected output (approximate):
#   89 v1
#   11 v2

# Test the canary header override
kubectl exec -n my-app frontend-xxxxx-yyyy -c frontend -- \
  curl -s -H "x-canary: true" api-svc:8080/version
# Output: v2 (always, regardless of weight)

Security: Mutual TLS Configuration

By default, NGINX Service Mesh operates in strict mTLS mode, meaning all service-to-service traffic is encrypted. The mesh automatically provisions X.509 certificates from its internal certificate authority and rotates them every 24 hours.

Understanding mTLS Modes

NGINX Service Mesh supports two mTLS modes:

Strict — all traffic within the mesh is encrypted; plaintext requests are rejected
Permissive — encrypted traffic is accepted, but plaintext is also allowed (useful during migration)

You can set the mode at installation time or change it later:

# Install with permissive mode for gradual adoption
nginx-meshctl install --mtls-mode permissive

# Later, switch to strict mode
nginx-meshctl configure --mtls-mode strict

Verifying mTLS in Action

# Capture traffic on the frontend sidecar to verify encryption
kubectl exec -n my-app frontend-xxxxx-yyyy -c nginx-mesh-sidecar -- \
  tcpdump -i any -A port 8080 -c 5

# You should see TLS handshake packets, not plaintext HTTP

# Check certificate details on a sidecar
kubectl exec -n my-app api-v1-xxxxx-yyyy -c nginx-mesh-sidecar -- \
  openssl s_client -connect localhost:8080 -showcerts 2>/dev/null | \
  grep -A5 "subject="

# Expected: subject= /O=NGINX Service Mesh/CN=api-v1-xxxxx-yyyy

Excluding Services from mTLS (when necessary)

For external services or databases that cannot handle mTLS, you can create exclusion rules:

apiVersion: spec.nginx-service-mesh/v1alpha1
kind: ExternalService
metadata:
  name: external-database
  namespace: my-app
spec:
  hostname: db.external.example.com
  port: 5432
  protocol: tcp
  mtls:
    mode: plaintext
---
apiVersion: spec.nginx-service-mesh/v1alpha1
kind: TrafficPolicy
metadata:
  name: db-plaintext
  namespace: my-app
spec:
  destination:
    serviceName: external-database
  mtls:
    mode: off

Resilience: Circuit Breaking and Rate Limiting

NGINX Service Mesh provides circuit breaking and rate limiting to protect services from cascading failures and excessive load.

Circuit Breaker Configuration

apiVersion: spec.nginx-service-mesh/v1alpha1
kind: CircuitBreaker
metadata:
  name: api-circuit-breaker
  namespace: my-app
spec:
  destination:
    serviceName: api-svc
  rules:
  - maxConnections: 100
    maxPendingRequests: 50
    maxRequests: 200
    maxRetries: 3
    maxRequestTime: 10s
    interval: 30s
    consecutiveFailures: 5
    rejectionProbability: 50

This configuration limits the API service to 100 concurrent connections and 200 maximum requests. After 5 consecutive failures within 30 seconds, the circuit opens and 50% of subsequent requests are immediately rejected (fast-fail) to allow the service to recover.

Rate Limiting Policies

apiVersion: spec.nginx-service-mesh/v1alpha1
kind: RateLimit
metadata:
  name: api-rate-limit
  namespace: my-app
spec:
  destination:
    serviceName: api-svc
  rules:
  - limit: 100
    burst: 20
    delay: 5
    unit: second
    source:
      serviceName: frontend-svc

This limits the frontend service to 100 requests per second to the API service, with a burst allowance of 20 and a delay of 5 seconds when the burst is exceeded.

Observability: Metrics, Tracing, and Logging

NGINX Service Mesh automatically collects metrics from every sidecar proxy and exposes them via Prometheus-compatible endpoints. Tracing propagation uses OpenTelemetry headers (or W3C Trace Context) out of the box.

Prometheus Metrics Collection

The mesh deploys a metrics aggregator that scrapes all sidecars. To integrate with your existing Prometheus stack:

# The mesh metrics service is available at:
# nginx-mesh-metrics-svc.nginx-mesh.svc.cluster.local:8080

# Add this scrape config to your Prometheus configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    scrape_configs:
    - job_name: 'nginx-mesh-sidecars'
      kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
          - my-app
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: keep
        regex: nginx-mesh-sidecar
      - source_labels: [__address__]
        action: replace
        regex: (.+):.+$
        replacement: ${1}:8080
        target_label: __address__
      metric_relabel_configs:
      - source_labels: [service]
        action: keep
        regex: '.+'

Key Metrics Available

nginx_mesh_total_requests — total request count per service
nginx_mesh_request_duration_seconds — latency histogram (p50, p90, p99)
nginx_mesh_active_connections — current connection count
nginx_mesh_tls_handshake_errors — mTLS handshake failures
nginx_mesh_circuit_breaker_open — circuit breaker state (0=closed, 1=open)

Grafana Dashboard Quickstart

# Import a pre-built NGINX Service Mesh dashboard
# Dashboard JSON is available in the mesh repository
curl -sL https://raw.githubusercontent.com/nginxinc/nginx-service-mesh/main/grafana/dashboard.json | \
  kubectl create configmap nginx-mesh-dashboard -n monitoring \
  --from-file=dashboard.json=/dev/stdin

# Then import via Grafana's HTTP API or UI
# The dashboard shows:
# - Service-level request rates
# - Latency percentiles
# - mTLS status indicators
# - Circuit breaker events
# - Sidecar resource consumption

Distributed Tracing with OpenTelemetry

Sidecars automatically propagate trace headers (W3C traceparent or OpenTelemetry trace-id). To send traces to Jaeger or Zipkin:

# Configure tracing export at install time
nginx-meshctl install \
  --tracing-enabled \
  --tracing-backend jaeger \
  --tracing-endpoint http://jaeger-collector.monitoring:14268/api/traces \
  --tracing-sampling-rate 0.1

# Or update an existing mesh
nginx-meshctl configure \
  --tracing-enabled true \
  --tracing-backend zipkin \
  --tracing-endpoint http://zipkin.monitoring:9411/api/v2/spans

Accessing Sidecar Logs

# View sidecar access logs for a specific pod
kubectl logs -n my-app frontend-xxxxx-yyyy -c nginx-mesh-sidecar

# Sample output:
# [2024-01-15T10:23:45.123Z] "GET /api/users HTTP/1.1" 200 234 "-" "curl/7.68.0" \
#   "api-svc.my-app.svc.cluster.local:8080" "10.0.1.5:8080" \
#   upstream=10.0.1.5:8080 upstream_status=200 request_time=0.045 \
#   bytes_sent=234 bytes_received=1024 tls_version=TLSv1.3

# Enable debug logging for troubleshooting
kubectl exec -n my-app frontend-xxxxx-yyyy -c nginx-mesh-sidecar -- \
  kill -SIGUSR1 1  # Reload with debug logging enabled

Best Practices for Production Deployments

1. Namespace Organization

Label namespaces strategically. Avoid injecting sidecars into system namespaces (kube-system, nginx-mesh itself) unless you have a specific reason. Use a dedicated namespace for mesh-eligible applications and gradually expand:

# Phase 1: Single namespace
kubectl label namespace frontend-apps nginx-mesh.inject=true

# Phase 2: Expand to more namespaces after validation
kubectl label namespace backend-services nginx-mesh.inject=true
kubectl label namespace data-pipeline nginx-mesh.inject=true

2. Resource Limits for Sidecars

Always set resource limits on the sidecar container to prevent noisy-neighbor issues. NGINX sidecars are lightweight but still need CPU for TLS handshakes and traffic processing:

# Configure sidecar resource limits at installation
nginx-meshctl install \
  --sidecar-cpu-limit 200m \
  --sidecar-cpu-request 50m \
  --sidecar-memory-limit 64Mi \
  --sidecar-memory-request 16Mi

3. Gradual mTLS Adoption

Start with permissive mTLS mode when introducing the mesh to an existing cluster. This allows both encrypted and plaintext traffic, preventing service disruptions:

# Step 1: Install in permissive mode
nginx-meshctl install --mtls-mode permissive

# Step 2: Monitor for TLS adoption metrics
kubectl port-forward -n nginx-mesh svc/nginx-mesh-metrics-svc 8080:8080 &
curl -s http://localhost:8080/metrics | grep nginx_mesh_tls

# Step 3: Switch to strict after confirming all services use mTLS
nginx-meshctl configure --mtls-mode strict

4. Canary Deployments with Confidence

Combine TrafficSplit with health checks and automated rollbacks. Start with a small weight (5%) and gradually increase while monitoring error rates:

# Progressive canary script example
for weight in 5 10 25 50 100; do
  echo "Setting canary weight to ${weight}%"
  kubectl patch trafficsplit api-canary -n my-app --type='json' \
    -p="[{\"op\":\"replace\",\"path\":\"/spec/backends/1/weight\",\"value\":${weight}}]"
  
  # Wait 2 minutes and check error rate
  sleep 120
  
  # Query Prometheus for error rate
  ERROR_RATE=$(curl -s 'http://prometheus:9090/api/v1/query?query=rate(nginx_mesh_total_requests{service="api-svc",status=~"5.."}[2m])' | jq '.data.result[0].value[1]')
  
  if (( $(echo "$ERROR_RATE > 0.05" | bc -l) )); then
    echo "Error rate too high, rolling back"
    kubectl patch trafficsplit api-canary -n my-app --type='json' \
      -p="[{\"op\":\"replace\",\"path\":\"/spec/backends/1/weight\",\"value\":0}]"
    exit 1
  fi
done

5. Monitoring the Mesh Itself

The mesh control plane components need monitoring too. Set up alerts for the controller and certificate authority:

# Critical alerts to configure in Prometheus AlertManager
groups:
- name: nginx_mesh_health
  rules:
  - alert: MeshControllerDown
    expr: up{job="nginx-mesh-controller"} == 0
    for: 2m
    severity: critical
    annotations:
      summary: "NGINX Mesh controller is not responding"
      
  - alert: SidecarInjectionFailure
    expr: rate(nginx_mesh_injection_failures_total[5m]) > 0.1
    for: 5m
    severity: warning
    annotations:
      summary: "Elevated sidecar injection failure rate"
      
  - alert: CertificateExpiry
    expr: nginx_mesh_cert_expiry_seconds < 86400
    for: 1h
    severity: critical
    annotations:
      summary: "Mesh certificates expiring within 24 hours"

6. Upgrade Strategy

NGINX Service Mesh supports rolling upgrades. Always back up your custom resources before upgrading:

# Backup all mesh configuration
kubectl get trafficsplit,circuitbreaker,ratelimit,externalService -A -o yaml > mesh-backup-$(date +%Y%m%d).yaml

# Upgrade the mesh
nginx-meshctl upgrade --version 2.2.0

# Verify all sidecars are updated
kubectl get pods -A -o json | jq '.items[] | select(.spec.containers[].name == "nginx-mesh-sidecar") | .status.containerStatuses[] | select(.name == "nginx-mesh-sidecar") | .image'

7. Security Considerations

Always run the mesh controller with minimal RBAC permissions — the installer sets these up automatically, but audit them periodically
Rotate the mesh root CA certificate every 90 days in high-security environments
Use network policies alongside mesh policies for defense-in-depth
Never expose the mesh metrics endpoint to the public internet
Regularly audit sidecar versions and ensure they match the control plane version

Troubleshooting Common Issues

Sidecar Not Injecting

If pods appear with only 1/1 containers (missing the sidecar), check:

# Verify namespace label
kubectl get namespace my-app --show-labels | grep nginx-mesh.inject

# Check webhook is operational
kubectl get mutatingwebhookconfiguration nginx-mesh-sidecar-injector -o yaml

# Look for injection errors in controller logs
kubectl logs -n nginx-mesh deployment/nginx-mesh-controller --tail=50 | grep -i error

# Common fix: re-label the namespace and restart the deployment
kubectl label namespace my-app nginx-mesh.inject=true --overwrite
kubectl rollout restart deployment -n my-app

mTLS Handshake Failures

Symptoms include 502 Bad Gateway or connection timeouts between services:

# Check TLS errors on sidecars
kubectl logs -n my-app deployment/frontend -c nginx-mesh-sidecar | grep -i "tls"

# Verify certificate rotation is working
kubectl exec -n my-app deployment/api-v1 -c nginx-mesh-sidecar -- \
  openssl x509 -in /etc/nginx-mesh/certs/client.pem -text -noout | grep "Not After"

# Common fix: restart the certificate authority and force certificate refresh
kubectl rollout restart deployment -n nginx-mesh nginx-mesh-cert-ca
kubectl delete pods -n my-app --all  # Force new certificates on restart

High Latency in the Mesh

# Profile sidecar latency
kubectl exec -n my-app deployment/frontend -c nginx-mesh-sidecar -- \
  curl -s http://localhost:8080/metrics | grep nginx_mesh_request_duration_seconds

# Check if sidecars are CPU-throttled
kubectl top pods -n my-app --containers

# Tune sidecar worker processes
nginx-meshctl configure --sidecar-workers 4  # Default is 2, increase for high-throughput services

Conclusion

NGINX Service Mesh delivers a compelling balance of capability and simplicity for Kubernetes-native microservices networking. By leveraging NGINX's proven proxy technology in a lightweight sidecar architecture, it provides transparent mTLS encryption, intelligent traffic routing, circuit breaking, and rich observability without the heavy operational burden associated with more complex mesh platforms. Its minimal resource footprint makes it suitable for environments ranging from large cloud clusters to resource-constrained edge deployments.

The installation process is streamlined through the nginx-meshctl CLI, and configuration is expressed entirely through Kubernetes-native custom resources — TrafficSplit for canary deployments, CircuitBreaker and RateLimit for resilience, and namespace labels for injection control. By following the best practices outlined in this guide — gradual mTLS adoption, progressive canary rollouts with automated health checks, proper resource limits, and thorough monitoring — teams can confidently deploy NGINX Service Mesh in production and realize the full benefits of a service mesh without unnecessary complexity. Whether you're modernizing a monolith, securing a microservices ecosystem, or building a new cloud-native platform from scratch, NGINX Service Mesh provides a pragmatic, high-performance foundation for service-to-service communication.