← Back to DevBytes
Kubernetes July 14, 2026 16 min read

Kubernetes API Performance: Profiling and Optimization

Understanding Kubernetes API Performance Profiling

The Kubernetes API server sits at the heart of every cluster, processing every request—from pod creations and service updates to custom resource definitions and admission webhooks. API performance profiling is the systematic practice of measuring, analyzing, and optimizing how the API server handles this relentless stream of requests. It encompasses latency distributions, throughput limits, resource consumption patterns, and the identification of bottlenecks across the entire request lifecycle.

Profiling goes beyond simple metric collection. It involves capturing detailed traces of request handling, analyzing CPU and memory profiles of the API server process itself, examining etcd interaction patterns, and understanding how watch mechanisms, list operations, and admission chains impact overall responsiveness. The goal is to build a quantitative understanding of API behavior under realistic and extreme load conditions, then use that data to drive targeted optimizations.

Key Performance Dimensions

Why API Performance Profiling Matters

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

A sluggish API server cascades into every corner of the cluster. Controllers fail to reconcile objects in time, causing stale state. Schedulers slow down, delaying pod placement. Autoscalers react late, risking capacity shortfalls. Service endpoints may become outdated, breaking network routing. Operators and end users experience timeouts in kubectl commands, CI/CD pipelines stall, and platform reliability erodes. Profiling the API server proactively prevents these failures from reaching production.

Real-World Impact Scenarios

How to Profile the Kubernetes API Server

Step 1: Enable and Access pprof Endpoints

The Kubernetes API server exposes Go's net/http/pprof endpoints for CPU profiling, heap profiling, goroutine dumps, and more. These endpoints are typically bound to the API server's health-check or metrics port (often 8443 or a dedicated loopback interface). You must ensure the --profiling flag is set to true on the API server (it is enabled by default in most distributions).

# Check if profiling is enabled on the API server
kubectl get pod -n kube-system -l component=kube-apiserver -o yaml | grep -A 2 profiling

# For managed clusters (GKE, EKS, AKS), profiling endpoints may be restricted.
# On self-managed clusters with kubeadm, you can port-forward to the API server
# after locating its IP and the profiling port.

# Example: Port-forward to a kube-apiserver pod's profiling port
kubectl port-forward -n kube-system pod/kube-apiserver-control-plane-node 8083:8443

# Then access pprof endpoints:
curl -s http://localhost:8083/debug/pprof/
curl -s http://localhost:8083/debug/pprof/profile?seconds=30 -o cpu-profile.pb.gz
curl -s http://localhost:8083/debug/pprof/heap -o heap-profile.pb.gz
curl -s http://localhost:8083/debug/pprof/goroutine -o goroutine.txt

Step 2: Capture CPU Profiles Under Load

CPU profiling is most useful when the API server is under realistic or peak load. Before capturing, generate traffic using tools like kubectl in a loop, the official kubernetes/perf-tests suite, or a custom load generator. Then capture a 30-60 second CPU profile. The resulting pprof data reveals which functions consume the most CPU time—whether it's JSON serialization, etcd request handling, admission evaluation, or something unexpected like excessive garbage collection.

# Generate representative load while profiling
# Terminal 1: Load generation loop
for i in $(seq 1 10000); do
  kubectl get pods --all-namespaces --output=json > /dev/null 2>&1
  kubectl create deployment test-$i --image=nginx --replicas=1 --dry-run=client > /dev/null 2>&1
done

# Terminal 2: Capture 30-second CPU profile during load
curl -s "http://localhost:8083/debug/pprof/profile?seconds=30" -o high-load-cpu.pb.gz

# Analyze the profile with go tool pprof
go tool pprof high-load-cpu.pb.gz

# Inside pprof interactive mode, common commands:
# top20 — shows top 20 CPU-consuming functions
# list encode — shows line-level breakdown of encoding functions
# web — generates a visual call graph (requires graphviz)
# peek etcd — shows functions with 'etcd' in their name or path

Step 3: Analyze Heap and Memory Profiles

Heap profiles reveal memory allocation patterns. The API server can accumulate significant memory from watch caches, serialized objects in flight, and admission webhook request/response buffers. A heap profile helps identify memory leaks, excessive allocations in hot paths, and objects retained unexpectedly due to lingering goroutines or unfinished watches.

# Capture heap profile
curl -s "http://localhost:8083/debug/pprof/heap" -o heap-profile.pb.gz

# Analyze with pprof, focusing on in-use memory (inuse_space)
go tool pprof -inuse_space heap-profile.pb.gz

# Useful pprof queries for heap analysis:
# top20 — top 20 memory-holding functions
# list WatchCache — examine memory used by watch caches
# list Admission — examine admission-related allocations
# pprof -alloc_space shows total allocations over time, useful for GC pressure
go tool pprof -alloc_space heap-profile.pb.gz

Step 4: Examine Goroutine Profiles

A goroutine dump shows all running goroutines with their full stack traces. This is invaluable for finding leaked goroutines from stuck watches, unfinished etcd transactions, or admission webhook calls that hang. In large clusters, thousands of idle goroutines waiting on watch channels is expected; goroutines stuck in sync.Mutex.Lock or net/http send operations are red flags.

# Capture goroutine profile as text
curl -s "http://localhost:8083/debug/pprof/goroutine?debug=2" -o goroutine-dump.txt

# Analyze common goroutine states
grep -c "IO wait" goroutine-dump.txt       # goroutines waiting on network I/O
grep -c "select" goroutine-dump.txt        # goroutines in channel operations (normal for watchers)
grep -c "sync.Mutex.Lock" goroutine-dump.txt # goroutines blocked on mutex contention
grep -c "running" goroutine-dump.txt       # actively running goroutines

# Look for goroutine leaks by counting goroutines over time
for i in $(seq 1 10); do
  curl -s http://localhost:8083/debug/pprof/goroutine -o /tmp/goroutine-$i.txt
  sleep 10
done
# Compare goroutine counts across files to detect monotonic growth

Step 5: Leverage Prometheus Metrics for API Performance

The Kubernetes API server exposes a rich set of Prometheus metrics on its metrics endpoint (typically /metrics on port 8443 or a dedicated metrics port). These metrics provide the time-series view necessary for understanding performance trends, spotting regressions, and correlating API behavior with cluster events. The following metrics are essential for profiling.

# Port-forward to the API server's metrics endpoint
kubectl port-forward -n kube-system pod/kube-apiserver-control-plane-node 8443:8443

# Fetch and examine key performance metrics
curl -s https://localhost:8443/metrics | grep -E "(apiserver_request_duration_seconds|apiserver_request_total|apiserver_current_inflight_requests|etcd_request_duration_seconds|apiserver_watch_events_total|apiserver_storage_objects)"

# Key metrics explained:

# apiserver_request_duration_seconds_bucket — histogram of request latencies
#   Labels: verb, resource, subresource, scope (cluster/namespace), code
#   Use to calculate p50, p95, p99 per resource and verb

# apiserver_current_inflight_requests — gauge of in-flight requests
#   Labels: mutating (read/write), readOnly
#   Critical: if this approaches max-requests-inflight, clients will be throttled

# apiserver_request_total — counter of requests broken down by verb, resource, code
#   Use rate() to calculate throughput per second

# etcd_request_duration_seconds_bucket — histogram of etcd operation latencies
#   Labels: operation (get, put, delete, txn), resource

# apiserver_watch_events_total — counter of watch events by resource
#   High values indicate many objects changing, stressing watch channels

# apiserver_storage_objects — gauge of stored object counts by resource
#   Helps correlate latency with object counts

Step 6: Build Custom Profiling Dashboards with PromQL

Raw metrics become actionable when queried with PromQL. The following queries form the backbone of API performance profiling dashboards.

# P99 latency for LIST pods across all namespaces over the last 5 minutes
histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket{verb="LIST", resource="pods", scope="cluster"}[5m]))

# P50 latency for CREATE secrets
histogram_quantile(0.50, rate(apiserver_request_duration_seconds_bucket{verb="CREATE", resource="secrets"}[5m]))

# Request rate (throughput) per second by verb
sum(rate(apiserver_request_total[5m])) by (verb)

# In-flight mutating requests as a percentage of max capacity
# (max-requests-inflight defaults to 400, but check your cluster config)
apiserver_current_inflight_requests{request_kind="mutating"} / 400 * 100

# etcd P99 latency for PUT operations on pods
histogram_quantile(0.99, rate(etcd_request_duration_seconds_bucket{operation="put", resource="pods"}[5m]))

# Watch event rate per second by resource
sum(rate(apiserver_watch_events_total[5m])) by (resource)

# Correlation: API server CPU seconds per request type
sum(rate(apiserver_request_duration_seconds_sum{verb="LIST"}[5m])) / sum(rate(apiserver_request_duration_seconds_count{verb="LIST"}[5m]))

Step 7: Distributed Tracing Integration

For deep request-level profiling, enable distributed tracing on the API server using the --tracing-config-file flag to point to a tracing configuration that exports spans to Jaeger, Zipkin, or OpenTelemetry collectors. This provides end-to-end visibility into how a single API request spends time across serialization, authentication, authorization, admission, and etcd interaction.

# Example tracing configuration file (tracing-config.yaml)
apiVersion: apiserver.config.k8s.io/v1
kind: TracingConfiguration
endpoint: http://jaeger-collector.observability.svc:4317
samplingRatePerMillion: 10000  # 1% sampling rate

# Pass this to the API server via:
# kube-apiserver --tracing-config-file=/etc/kubernetes/tracing-config.yaml

# Verify tracing is active by checking API server logs for:
# "Tracing is enabled"

Optimization Techniques Based on Profile Findings

Optimization 1: Tuning the Watch Cache

When profiling reveals high LIST latency and elevated etcd reads, the watch cache is often the first optimization target. The API server maintains an in-memory cache of objects for each resource, updated via etcd watches. LIST requests served from this cache avoid etcd entirely, dramatically reducing latency. The --watch-cache-sizes flag controls the cache size per resource.

# Default watch cache sizes are often too small for large clusters.
# Increase cache sizes for resources with many LIST requests.
# Example: Set watch cache size for pods to 5000 and secrets to 2000
kube-apiserver --watch-cache-sizes=pods#5000,secrets#2000,deployments#1000

# Verify watch cache effectiveness via metrics:
# apiserver_list_cache_miss_total — counts LIST requests that missed the cache
# apiserver_list_cache_hit_total — counts LIST requests served from cache
# Aim for a high hit ratio (above 90%) for frequently listed resources

curl -s https://localhost:8443/metrics | grep list_cache

Optimization 2: Adjusting Request Limits

Profiling under load may reveal that the API server hits its concurrency limits, causing request throttling with HTTP 429 responses. The --max-requests-inflight and --max-mutating-requests-inflight flags control how many concurrent non-mutating and mutating requests the API server processes. Setting these too low starves throughput; setting them too high can overwhelm etcd or cause out-of-memory conditions.

# Default values are typically 400 for non-mutating, 200 for mutating.
# For clusters with high throughput needs and sufficient API server replicas:
kube-apiserver --max-requests-inflight=800 --max-mutating-requests-inflight=400

# Monitor the effect:
# apiserver_current_inflight_requests should stabilize below the new limits
# apiserver_request_total with code="429" should drop to near zero
# Watch etcd_request_duration_seconds for any degradation from higher concurrency

Optimization 3: Optimizing Admission Webhooks

Profiling frequently identifies admission webhooks as a dominant latency source. Each mutating or validating webhook adds serial round-trip latency to every matching write request. Optimization strategies include reducing webhook count, enforcing strict timeouts, moving logic to post-commit controllers where possible, and ensuring webhook servers are co-located with minimal network hops.

# Audit webhook latency via API server metrics
# apiserver_admission_webhook_admission_duration_seconds_bucket
# Labels: name, type (mutating/validating), operation

# P99 latency for a specific mutating webhook
histogram_quantile(0.99, rate(apiserver_admission_webhook_admission_duration_seconds_bucket{name="my-pod-webhook.example.com", type="mutating"}[5m]))

# Configure webhook timeouts in the admission webhook configuration
# Short timeouts prevent slow webhooks from stalling the entire write path
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: example-webhook
webhooks:
  - name: my-pod-webhook.example.com
    timeoutSeconds: 2  # Default is 10; reduce for performance-critical paths
    failurePolicy: Ignore  # Allow requests to proceed if webhook times out

# Consider using ValidatingAdmissionPolicy (CEL-based) instead of webhooks
# for common validation cases — evaluated in-process with minimal overhead

Optimization 4: Reducing LIST Payload Sizes

When profiling shows high CPU time in JSON serialization and high memory allocations for LIST responses, reducing payload sizes provides immediate relief. Encourage the use of field selectors, label selectors, and paginated LIST calls with resourceVersion continuation tokens instead of full, unqualified LIST operations.

# Inefficient: Full LIST with large JSON response
kubectl get pods --all-namespaces --output=json

# Efficient: Filtered LIST with label selector
kubectl get pods --all-namespaces --selector=app=myapp --output=json

# Efficient: Paginated LIST (client-side continuation via resourceVersion)
# First call gets a resourceVersion, subsequent calls use that for consistency
kubectl get pods --all-namespaces --chunk-size=500

# For Go clients using client-go, always use pagination:
listOptions := metav1.ListOptions{
    Limit: 500,
    Continue: continueToken,  // from previous response
}
pods, err := clientset.CoreV1().Pods("").List(ctx, listOptions)

Optimization 5: API Priority and Fairness Configuration

The API Priority and Fairness (APF) system, enabled by default since Kubernetes 1.29, allows fine-grained control over how concurrent requests are queued and dispatched. Profiling may reveal that certain request patterns (like frequent LIST calls from a chatty controller) starve more important requests (like scheduler binding decisions). APF lets you define priority levels, flow schemas, and queue configurations to ensure critical operations always get throughput.

# Example APF configuration: Prioritize scheduler and controller requests
# over user-facing LIST operations
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: PriorityLevelConfiguration
metadata:
  name: system-priority
spec:
  type: System
---
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: FlowSchema
metadata:
  name: scheduler-leader-election
spec:
  priorityLevelConfiguration:
    name: system-priority
  matchingPrecedence: 1000
  rules:
    - subjects:
        - kind: ServiceAccount
          serviceAccount:
            name: kube-scheduler
            namespace: kube-system
      resourceRules:
        - verbs: ["create", "update"]
          resources: ["leases", "endpoints"]
      nonResourceRules: []
---
# Lower priority for user LIST operations
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: PriorityLevelConfiguration
metadata:
  name: user-list-low-priority
spec:
  type: Limited
  limited:
    assuredConcurrencyShares: 10
    limitResponse:
      type: Queue
      queuing:
        queues: 4
        handSize: 2
        queueLengthLimit: 50

# Verify APF metrics:
# apiserver_flowcontrol_request_concurrency_limit — concurrency limits per priority level
# apiserver_flowcontrol_current_inqueue_requests — queued requests per priority level

Optimization 6: etcd Tuning and Compaction

API server profiling often reveals that latency originates in etcd, not in the API server itself. Common culprits include etcd compaction pauses, high disk I/O latency, or excessive object counts in a single etcd key range. Optimizing etcd involves regular defragmentation, adjusting etcd's snapshot and compaction intervals, and ensuring the etcd cluster has dedicated SSD storage with sufficient IOPS.

# Check etcd compaction status and database size
ETCDCTL_API=3 etcdctl --endpoints=https://etcd-server:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status --write-out=table

# Trigger manual compaction to reclaim space
ETCDCTL_API=3 etcdctl compact $(etcdctl endpoint status --write-out=json | jq -r '.[].Response.Index')

# Defragment etcd after compaction
ETCDCTL_API=3 etcdctl defrag --cluster

# etcd flags for automatic compaction (set on etcd server, not API server)
# --auto-compaction-mode=periodic --auto-compaction-retention=30m
# This compacts every 30 minutes, preventing unbounded growth

# Monitor etcd metrics alongside API server metrics:
# etcd_server_proposals_applied_total — write throughput
# etcd_disk_backend_commit_duration_seconds_bucket — disk write latency
# etcd_server_slow_read_indexes_total — slow reads indicating compaction issues

Optimization 7: API Server Replica Scaling and Load Balancing

When profiling shows CPU or memory saturation on individual API server instances, horizontal scaling distributes load. However, scaling must be paired with intelligent load balancing because LIST and watch requests are stateful (tied to specific etcd revisions). Ensure your load balancer uses consistent hashing or that clients use resourceVersion to handle bouncing between replicas.

# Scale API server deployment (for self-managed clusters)
kubectl scale deployment kube-apiserver -n kube-system --replicas=5

# Verify load distribution across replicas
curl -s https://localhost:8443/metrics | grep apiserver_request_total | \
  jq -r '.data.result[] | "\(.metric.pod) \(.value[1])"'

# For client-go users, configure connection to use multiple API servers
# by providing multiple endpoints in kubeconfig or using DNS round-robin

Best Practices for Ongoing API Performance Management

Establish Performance Baselines

Capture API server performance profiles when the cluster is healthy and operating normally. Store these baselines—CPU profiles, heap profiles, Prometheus metric snapshots, and goroutine counts—alongside cluster version and configuration information. When a performance regression is suspected, compare current profiles against baselines to pinpoint changes. A 20% increase in P99 LIST latency after a cluster upgrade, for example, can be traced to a specific code change in the API server or a new default setting.

# Automate baseline capture with a cron job
# Capture daily profiles during a representative time window
0 10 * * * curl -s "http://api-server:8083/debug/pprof/profile?seconds=30" \
  -o /var/baselines/$(date +%Y-%m-%d)-cpu.pb.gz
0 10 * * * curl -s "http://api-server:8083/debug/pprof/heap" \
  -o /var/baselines/$(date +%Y-%m-%d)-heap.pb.gz

Profile Before and After Every Change

Any significant cluster change—upgrading Kubernetes versions, adding new CRDs, deploying new admission webhooks, increasing node count, or onboarding new tenants—should trigger a profiling session. The cost of profiling (a few seconds of CPU and memory overhead) is negligible compared to the risk of undetected performance degradation. Integrate profiling into your change management process.

Set Alerting on Key Performance Indicators

Prometheus metrics enable proactive alerting on API performance degradation. Configure alerts for P99 latency exceeding thresholds, in-flight request counts approaching limits, watch cache miss ratios climbing, and admission webhook latencies spiking. These alerts catch performance issues before users notice.

# Example Prometheus alerting rules for API server performance

groups:
  - name: api-server-performance
    rules:
      - alert: APIListLatencyHigh
        expr: histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket{verb="LIST", scope="cluster"}[5m])) > 3
        for: 10m
        annotations:
          summary: "P99 LIST latency exceeds 3 seconds"

      - alert: APIMutatingInflightNearLimit
        expr: apiserver_current_inflight_requests{request_kind="mutating"} > 350
        for: 5m
        annotations:
          summary: "Mutating in-flight requests approaching default limit of 400"

      - alert: WatchCacheMissRateHigh
        expr: rate(apiserver_list_cache_miss_total[5m]) / rate(apiserver_list_cache_hit_total[5m]) > 0.2
        for: 15m
        annotations:
          summary: "Watch cache miss ratio exceeds 20%"

      - alert: AdmissionWebhookLatencyHigh
        expr: histogram_quantile(0.99, rate(apiserver_admission_webhook_admission_duration_seconds_bucket[5m])) > 1
        for: 10m
        annotations:
          summary: "P99 admission webhook latency exceeds 1 second"

Use Continuous Profiling

For production clusters, consider deploying continuous profiling infrastructure (such as Parca, Pyroscope, or cloud-provider profiling services) that captures CPU and heap profiles on a rolling basis without manual intervention. Continuous profiling provides the historical context needed to attribute performance changes to specific deployments, configuration changes, or code regressions.

Document Performance Characteristics Per Resource

Not all resources behave the same way. Pods are numerous and change frequently, stressing watch paths. Secrets are small but sensitive, triggering encryption/decryption overhead. CRDs with large schemas stress the storage path. Document the performance profile of each critical resource in your cluster, including expected P50/P99 latency per verb, typical object counts, and any tuning applied (watch cache sizes, pagination requirements, APF flow schemas). This documentation accelerates troubleshooting when performance deviates.

Test with Realistic Load Profiles

Synthetic benchmarks that generate uniform, evenly distributed requests do not reflect real-world API traffic, which often exhibits bursts, skewed resource access patterns, and long-running watches. Use load profiles derived from production metrics—capture a representative hour of API traffic and replay it at varying speeds using tools like vegeta or the Kubernetes SIG-Scale testing suite. This reveals bottlenecks that uniform load generators miss.

# Example: Capture production API traffic patterns and replay
# Step 1: Extract request counts per resource/verb from production metrics
curl -s https://api-server-prod:8443/metrics | \
  grep apiserver_request_total | \
  grep -v '#' | \
  sort > production-request-patterns.txt

# Step 2: Build a load generator that reproduces this distribution
# Focus on the top 10 most frequent resource/verb combinations
# Match their relative proportions in your test suite

Conclusion

Kubernetes API performance profiling is not a one-time task—it is a continuous discipline that pays dividends in cluster reliability, user experience, and operational predictability. By systematically capturing CPU profiles, heap profiles, goroutine dumps, and Prometheus metrics, you build a quantitative understanding of how your API server behaves under real-world conditions. This understanding guides concrete optimizations: tuning watch cache sizes, adjusting request concurrency limits, streamlining admission webhook chains, configuring API Priority and Fairness, and ensuring etcd health. The practices outlined in this tutorial—from enabling pprof endpoints and crafting PromQL queries to setting performance alerts and maintaining baselines—form a complete framework for mastering API server performance. In a platform where every millisecond of API latency propagates to controllers, schedulers, and ultimately end users, disciplined profiling and optimization is not optional; it is foundational.

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles