Kubernetes Server Bottleneck Detection and Resolution

Understanding Kubernetes Server Bottlenecks

A Kubernetes server bottleneck occurs when one or more resources within your cluster become saturated, causing degraded application performance, increased latency, or complete service unavailability. These bottlenecks can manifest at multiple layers — from the physical node level (CPU, memory, disk I/O, network bandwidth) to the Kubernetes control plane components (API server, etcd, scheduler) and even within your application containers themselves.

Bottleneck detection in Kubernetes is the systematic process of identifying which specific resource or component is constraining your workload's performance. Resolution involves applying targeted fixes — scaling, tuning, restructuring workloads, or adjusting cluster configuration — to eliminate the identified constraint.

Key Bottleneck Categories

• Compute bottlenecks: CPU saturation where containers are throttled or starved of CPU cycles
• Memory bottlenecks: RAM exhaustion leading to OOMKilled pods or excessive swapping
• I/O bottlenecks: Disk throughput or IOPS saturation affecting databases and stateful workloads
• Network bottlenecks: Bandwidth limits, connection tracking table exhaustion, or CNI plugin overhead
• Scheduling bottlenecks: Pods stuck in Pending state due to insufficient node resources or affinity rule conflicts
• Control plane bottlenecks: Slow API server responses, etcd latency spikes, or controller manager backlogs

Why Bottleneck Detection Matters

In production Kubernetes environments, undetected bottlenecks cause cascading failures. A single memory-leaking pod can trigger node-level OOM events, evicting critical system daemons and destabilizing the entire node. CPU-starved API servers delay admission webhooks, slowing pod scheduling cluster-wide. Ignoring these signals means your Mean Time To Recovery (MTTR) stretches from minutes to hours, directly impacting Service Level Objectives (SLOs) and potentially causing significant revenue loss during peak traffic periods.

Proactive bottleneck detection enables:

• Predictable autoscaling: HPA and VPA can react before users experience latency
• Cost optimization: Identifying over-provisioned nodes lets you right-size your infrastructure
• Capacity planning: Historical bottleneck data informs future cluster scaling decisions
• Root cause analysis: Rapid isolation of performance issues reduces debugging time

Setting Up Detection Infrastructure

Before you can detect bottlenecks, you need observability tooling in place. The following stack provides comprehensive visibility into Kubernetes resource bottlenecks.

Step 1: Deploy the Kubernetes Metrics Server

The Metrics Server is the foundation for resource-based autoscaling and basic bottleneck identification via kubectl top commands. It collects CPU and memory metrics from kubelets and exposes them through the Kubernetes API.

# Install Metrics Server from the official repository
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Verify metrics server is running
kubectl get deployment metrics-server -n kube-system

# Wait 30 seconds, then check node metrics
kubectl top nodes

# Check pod resource consumption across a namespace
kubectl top pods -n production --sort-by=cpu

Step 2: Install Prometheus and Node Exporter

While Metrics Server gives point-in-time snapshots, Prometheus provides historical time-series data essential for trend analysis and bottleneck prediction. Node Exporter captures low-level system metrics including disk I/O, network throughput, and kernel memory pressure.

# Add the Prometheus community Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install kube-prometheus-stack (includes Prometheus, Alertmanager, Grafana, Node Exporter)
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.retention=30d \
  --set grafana.adminPassword=SecureAdminPass2024 \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=100Gi

Step 3: Deploy Node Problem Detector

Node Problem Detector runs as a DaemonSet on every node, detecting kernel-level issues like read-only filesystems, OOM events, hung tasks, and disk pressure — problems that raw metrics alone might miss.

# Clone the Node Problem Detector repository
git clone https://github.com/kubernetes/node-problem-detector.git
cd node-problem-detector

# Deploy as a DaemonSet
kubectl apply -f deployment/node-problem-detector-config.yaml
kubectl apply -f deployment/node-problem-detector.yaml

# Verify it's running on all nodes
kubectl get daemonset node-problem-detector -n kube-system

# Check node conditions added by NPD
kubectl get nodes -o custom-columns=NAME:.metadata.name,CONDITIONS:.status.conditions[*].type | grep -E "Kernel|Disk|Memory"

Detection Techniques and Practical Commands

CPU Bottleneck Detection

CPU bottlenecks manifest as high node CPU utilization, container CPU throttling, and increased request latency. Kubernetes enforces CPU limits via the Completely Fair Scheduler (CFS) in the kernel. When a container exceeds its limit, it gets throttled — even if the node has free CPU cycles.

# Identify nodes with high CPU utilization (>80%)
kubectl top nodes | awk 'NR>1 {if ($3+0 > 80) print $1 " CPU: " $3 "%"}' 

# Find pods exceeding their CPU limits (throttled)
kubectl top pods -A | awk 'NR>1 {print $0}' | sort -k3 -rn | head -20

# Check actual throttling metrics with PromQL
# Run in Grafana or via Prometheus API
curl -s 'http://prometheus-service:9090/api/v1/query?query=rate(container_cpu_cfs_throttled_seconds_total{namespace="production"}[5m])' | jq .

# Inspect detailed container resource usage with cAdvisor metrics
kubectl exec -it deployment/prometheus-server -n monitoring -- sh -c \
  'curl -s http://localhost:9090/api/v1/query?query=rate(container_cpu_usage_seconds_total{container!="POD"}[5m]) > 0.8' | jq .

Memory Bottleneck Detection

Memory pressure causes pods to be evicted or killed (OOMKilled). The node's kubelet monitors memory usage and applies eviction thresholds defined by the --eviction-hard flag. Detecting memory bottlenecks early prevents workload disruption.

# Check for OOMKilled pods across the cluster
kubectl get pods -A -o json | jq -r '.items[] | select(.status.containerStatuses[]?.state.terminated.reason=="OOMKilled") | [.metadata.namespace, .metadata.name] | @tsv'

# Monitor node memory pressure conditions
kubectl get nodes -o custom-columns=NAME:.metadata.name,MEMORY_PRESSURE:.status.conditions[?(@.type=="MemoryPressure")].status

# Find pods with memory usage exceeding 90% of their limit
kubectl top pods -A --sort-by=memory | awk 'NR>1 {print $0}' | head -20

# Prometheus query for memory saturation (usage vs request/limit)
# Execute via Grafana or direct API call
curl -s 'http://prometheus-service:9090/api/v1/query?query=sum(container_memory_working_set_bytes{container!="POD"}) by (pod) / sum(label_join(kube_pod_container_resource_limits{resource="memory"},"pod","","namespace","pod")) by (pod) > 0.85' | jq .

Disk I/O Bottleneck Detection

Disk I/O bottlenecks are particularly dangerous for databases and stateful workloads. They cause write amplification, increased commit latency, and can trigger node-level disk pressure evictions.

# Check node disk pressure conditions
kubectl get nodes -o custom-columns=NAME:.metadata.name,DISK_PRESSURE:.status.conditions[?(@.type=="DiskPressure")].status

# Monitor per-pod filesystem usage
for pod in $(kubectl get pods -n production -o name | head -10); do
  echo "=== $pod ==="
  kubectl exec -it $pod -n production -- df -h / 2>/dev/null || true
done

# Prometheus query for disk I/O utilization on nodes
# Shows percentage of device utilization over 5 minutes
curl -s 'http://prometheus-service:9090/api/v1/query?query=rate(node_disk_io_time_seconds_total{device!~"ram.*|loop.*"}[5m]) * 100' | jq .

# Identify pods with high filesystem write rates
curl -s 'http://prometheus-service:9090/api/v1/query?query=topk(10, sum(rate(fs_writes_bytes_total{container!="POD"}[5m])) by (pod))' | jq .

Network Bottleneck Detection

Network bottlenecks arise from bandwidth saturation, connection tracking table exhaustion, or misconfigured CNI plugins. Symptoms include TCP retransmissions, dropped packets, and increased inter-pod latency.

# Check for connection tracking table issues (common with high-pod-density nodes)
# Run on a node directly or via privileged pod
kubectl run netshoot --rm -it --image=nicolaka/netshoot --restart=Never -- /bin/bash -c \
  'echo "Conntrack max: $(cat /proc/sys/net/netfilter/nf_conntrack_max)"; echo "Conntrack count: $(cat /proc/sys/net/netfilter/nf_conntrack_count)"'

# Monitor network errors via Node Exporter metrics
curl -s 'http://prometheus-service:9090/api/v1/query?query=rate(node_network_receive_errors_total[5m]) + rate(node_network_transmit_errors_total[5m]) > 0' | jq .

# Check TCP retransmission rate per node
curl -s 'http://prometheus-service:9090/api/v1/query?query=rate(node_netstat_Tcp_RetransSegs[5m])' | jq .

# Test inter-pod network latency with a debug container
kubectl run latency-test --rm -it --image=alpine/curl --restart=Never -- /bin/sh -c \
  'for i in $(seq 1 100); do curl -so /dev/null -w "Connect: %{time_connect}s Total: %{time_total}s\n" http://my-service.production.svc.cluster.local:8080/health; sleep 0.1; done'

Control Plane Bottleneck Detection

Control plane bottlenecks are harder to detect but equally critical. Slow etcd responses cascade into API server latency, which delays all cluster operations including pod scheduling and service endpoint updates.

# Check etcd database size (large etcd DB causes slowness)
kubectl exec -it etcd-master-node -n kube-system -- etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status --write-out=table

# Monitor API server request latency
curl -s 'http://prometheus-service:9090/api/v1/query?query=histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m])) > 1' | jq .

# Check etcd request duration
curl -s 'http://prometheus-service:9090/api/v1/query?query=histogram_quantile(0.99, rate(etcd_request_duration_seconds_bucket[5m])) > 0.5' | jq .

# Verify scheduler health and pending pod queue
kubectl get pods -A --field-selector=status.phase=Pending | wc -l
kubectl get --raw /api/v1/namespaces/kube-system/pods/scheduler-node:healthz 2>/dev/null || echo "Check scheduler pod directly"

Resolution Strategies

Horizontal Pod Autoscaling (HPA)

For CPU and memory bottlenecks at the application level, HPA dynamically scales pod replicas based on observed metrics. This is the first-line defense against predictable resource saturation.

# Create an HPA that scales based on CPU utilization
cat <

# Deploy VPA recommender, updater, and admission controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/master/vertical-pod-autoscaler/deploy/vpa-beta-crd.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/master/vertical-pod-autoscaler/deploy/vpa-beta-deployment.yaml

# Create a VPA resource for a deployment
cat <

# Taint and label nodes to control workload placement
kubectl taint nodes worker-node-1 workload=high-cpu:NoSchedule
kubectl label nodes worker-node-1 node-role=compute-intensive

# Create a pod with toleration and node affinity
cat <

# Create a StorageClass with IOPS provisioning for cloud environments
cat <

# Apply sysctl tuning for connection tracking via a privileged init container
cat <

# Scale the API server deployment (on managed clusters, adjust through cloud provider settings)
# For self-managed clusters:
kubectl scale deployment kube-apiserver -n kube-system --replicas=3

# Optimize etcd by defragmenting and compacting
# Execute on etcd node
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  defrag --cluster

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  compaction $(ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status --write-out=json | jq -r '.[].Status.header.revision') 

# Reduce object churn by tuning controller manager intervals
kubectl patch deployment kube-controller-manager -n kube-system --type=json -p \
  '[{"op":"add","path":"/spec/template/spec/containers/0/command/-","value":"--node-monitor-grace-period=60s"}]'

# Enable API priority and fairness to protect critical requests
cat <

# Create PrometheusRule for bottleneck alerts
cat < 85
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Node CPU usage above 85% for 10 minutes"
        description: "Node {{ \$labels.instance }} CPU utilization is {{ \$value | printf \"%.1f\" }}%"
    
    - alert: NodeMemoryHigh
      expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Node memory usage above 90%"
        description: "Node {{ \$labels.instance }} memory utilization is {{ \$value | printf \"%.1f\" }}%"
    
    - alert: PodOOMKilled
      expr: increase(kube_pod_container_status_terminated_reason{reason="OOMKilled"}[1h]) > 0
      labels:
        severity: critical
      annotations:
        summary: "Pod killed due to OOM"
        description: "Pod {{ \$labels.pod }} in namespace {{ \$labels.namespace }} was OOMKilled"
    
    - alert: DiskPressureNode
      expr: kube_node_status_condition{condition="DiskPressure",status="true"} == 1
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Node experiencing disk pressure"
        description: "Node {{ \$labels.node }} has DiskPressure condition active"
    
    - alert: CPUTrottlingHigh
      expr: rate(container_cpu_cfs_throttled_seconds_total{container!="POD"}[5m]) > 0.1
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Container CPU throttling detected"
        description: "Container {{ \$labels.container }} in pod {{ \$labels.pod }} is being CPU throttled"
    
    - alert: NetworkErrors
      expr: rate(node_network_receive_errors_total[5m]) + rate(node_network_transmit_errors_total[5m]) > 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Network errors on node interface"
        description: "Node {{ \$labels.instance }} interface {{ \$labels.device }} has network errors"
EOF

# Use LimitRange to enforce defaults across a namespace
cat <

cat <

cat <

# Simple load test using a Kubernetes job with vegeta
cat <

# Example kubelet configuration for a node with 64GB RAM
# Applied via node configuration file or cloud-init
cat < /var/lib/kubelet/config.yaml
evictionHard:
  memory.available: "1Gi"
  nodefs.available: "5%"
  nodefs.inodesFree: "5%"
  imagefs.available: "10%"
evictionSoft:
  memory.available: "2Gi"
  nodefs.available: "10%"
evictionSoftGracePeriod:
  memory.available: "2m"
  nodefs.available: "5m"
evictionMaxPodGracePeriod: 60
EOF

cat <