Understanding Docker API Bottlenecks
A Docker API bottleneck occurs when the communication between Docker clients and the Docker daemon becomes a performance-limiting factor in your containerized infrastructure. The Docker Engine API—whether accessed through the Unix socket (/var/run/docker.sock) or a TCP port—has finite capacity for concurrent requests, and when that capacity is exceeded, operations queue up, timeouts occur, and overall system responsiveness degrades.
At its core, the bottleneck manifests as increased latency on Docker operations such as container listing, image pulls, container creation, or log streaming. The daemon processes requests sequentially or with limited parallelism per operation type, meaning a flood of docker ps commands from multiple monitoring agents can starve out critical orchestration tasks. Understanding this bottleneck requires familiarity with the Docker daemon's internal architecture: it uses a gRPC-based API (for newer versions) or HTTP-based REST API, both subject to connection limits, memory pressure from large responses, and I/O contention when multiple operations touch the same on-disk structures like the layer database or containerd's metadata store.
Why Docker API Bottlenecks Matter
🚀 Deploy your AI agent in 10 minutes
Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.
Try it free →Ignoring Docker API bottlenecks leads to cascading failures across your infrastructure:
- Orchestrator starvation: Kubernetes kubelets, Nomad agents, or Swarm managers rely on rapid Docker API calls. When the API slows, pod scheduling stalls, readiness probes time out, and nodes get marked as unhealthy.
- CI/CD pipeline degradation: Build pipelines that spin up and tear down containers for each job will see exponential build-time increases as the daemon struggles to keep up.
- Monitoring blindness: Prometheus exporters, Datadog agents, and custom monitoring scripts that query the Docker API for metrics will return stale or partial data, creating gaps in observability precisely when you need it most.
- Resource exhaustion: The daemon itself consumes increasing CPU and memory buffering queued requests, sometimes leading to daemon crashes that take down all running containers on the host.
- Security implications: Failed health checks may cause load balancers to route traffic to unhealthy containers, while delayed container stops prevent timely remediation of compromised instances.
In production environments with hundreds of containers per host, the Docker API becomes a shared, contended resource—much like a database connection pool—that requires deliberate management and monitoring.
Detection Methods and Tools
Monitoring Docker Daemon Metrics
The Docker daemon exposes Prometheus metrics when started with the --metrics-addr flag. These metrics provide direct insight into API latency and request volumes:
# Start Docker daemon with metrics endpoint
dockerd --metrics-addr=0.0.0.0:9323 --experimental
# Scrape the metrics endpoint
curl -s http://localhost:9323/metrics | grep -E "engine_daemon|api"
# Key metrics to watch:
# engine_daemon_container_actions_seconds - Histogram of container action durations
# engine_daemon_image_actions_seconds - Histogram of image operation durations
# engine_daemon_engine_info - Daemon information gauge
# engine_daemon_failed_connections - Count of failed connection attempts
Set up a Prometheus alert for high API latency percentiles:
# Prometheus alert rule for Docker API bottleneck detection
groups:
- name: docker_api_bottleneck
rules:
- alert: DockerAPILatencyHigh
expr: histogram_quantile(0.95, rate(engine_daemon_container_actions_seconds_bucket[5m])) > 2
for: 5m
labels:
severity: warning
annotations:
summary: "Docker API p95 latency exceeds 2 seconds"
description: "The 95th percentile latency for Docker container actions is {{ $value }}s, indicating API bottleneck"
Analyzing API Response Times
You can profile API response times directly by instrumenting calls to the Docker socket. This approach works even without daemon metrics enabled:
#!/bin/bash
# Profile Docker API response time via the Unix socket
SOCKET="/var/run/docker.sock"
ENDPOINT="/containers/json"
# Time a single API call with curl using the Unix socket
time curl --unix-socket "$SOCKET" -s -o /dev/null -w \
"HTTP Code: %{http_code}\nTime Total: %{time_total}s\nTime Connect: %{time_connect}s\nTime TTFB: %{time_starttransfer}s\n" \
"http://localhost$ENDPOINT"
# Sample output:
# HTTP Code: 200
# Time Total: 3.247s
# Time Connect: 0.002s
# Time TTFB: 3.244s
# A high TTFB indicates daemon-side processing delay
For continuous monitoring, run the check periodically and log anomalies:
#!/bin/bash
# Continuous Docker API latency monitor
THRESHOLD_SECONDS=2.0
SOCKET="/var/run/docker.sock"
while true; do
START_TIME=$(date +%s.%N)
# Use curl with detailed timing
RESULT=$(curl --unix-socket "$SOCKET" -s -o /dev/null -w "%{time_total}" \
"http://localhost/containers/json?all=true&limit=100" 2>/dev/null)
if (( $(echo "$RESULT > $THRESHOLD_SECONDS" | bc -l) )); then
echo "[$(date -Iseconds)] WARNING: API latency ${RESULT}s exceeds threshold ${THRESHOLD_SECONDS}s"
# Capture diagnostic snapshot when bottleneck detected
TOP_MEMORY=$(curl --unix-socket "$SOCKET" -s \
"http://localhost/containers/json?all=true" | jq '[.[]] | sort_by(-.Memory) | .[0:5] | .[] | {Names: .Names, Memory: (.Memory / 1024 / 1024 | floor)}')
echo "Top memory consumers: $TOP_MEMORY"
fi
sleep 10
done
Using Docker Events for Anomaly Detection
The Docker events stream provides real-time visibility into daemon activity. A sudden burst of events often correlates with API bottlenecks, especially when many containers start or stop simultaneously:
#!/bin/bash
# Monitor Docker events for burst patterns indicative of API stress
EVENT_THRESHOLD=50 # Events per second threshold
WINDOW_SECONDS=5
count_events_in_window() {
docker events --since "${WINDOW_SECONDS}s ago" --until "0s ago" --format '{{.Action}}' 2>/dev/null | wc -l
}
while true; do
EVENT_COUNT=$(count_events_in_window)
EVENTS_PER_SECOND=$(( EVENT_COUNT / WINDOW_SECONDS ))
if [ $EVENTS_PER_SECOND -gt $EVENT_THRESHOLD ]; then
echo "[$(date -Iseconds)] BOTTLENECK WARNING: ${EVENTS_PER_SECOND} events/sec detected"
echo "Last 10 events:"
docker events --since "10s ago" --until "0s ago" --format '{{.Time}} {{.Action}} {{.Type}}' 2>/dev/null | tail -10
fi
sleep $WINDOW_SECONDS
done
Common Bottleneck Scenarios
Container Sprawl and List Operations
The most common bottleneck arises from excessive docker ps or equivalent API calls to /containers/json. Each call requires the daemon to iterate through all containers, read their state from disk, and serialize the response. With hundreds of containers and dozens of concurrent callers (monitoring agents, orchestrators, CI tools), the daemon's single-threaded request processing becomes overwhelmed.
Detection signature: High latency on /containers/json endpoints, rising CPU usage on the dockerd process, and increasing number of ESTABLISHED connections to /var/run/docker.sock.
# Check connection count to Docker socket
ss -lx | grep docker.sock
lsof /var/run/docker.sock | wc -l
# Typical problematic pattern: dozens of processes holding the socket open
# dockerd 1234 root 42u unix 0x... /var/run/docker.sock
# containerd 1235 root 12u unix 0x... /var/run/docker.sock
# prometheus 5678 root 8u unix 0x... /var/run/docker.sock
# cadvisor 9012 root 15u unix 0x... /var/run/docker.sock
# Repeated many times over...
Image Pull Contention
Concurrent image pulls to the same daemon create contention on the layer storage subsystem. The Docker daemon must coordinate layer downloads, verify checksums, and unpack layers—operations that are I/O and CPU intensive. When multiple CI jobs pull large images simultaneously, API calls for other operations queue behind these expensive image operations.
Detection signature: Spikes in engine_daemon_image_actions_seconds metrics, growing /var/lib/docker/tmp directory, and increased iowait in system CPU metrics.
Volume and Network API Overhead
Operations involving volume listing (/volumes) and network inspection (/networks) also contribute to bottlenecks, particularly in environments using Docker Compose extensively. Each docker-compose up invocation triggers a cascade of API calls to inspect networks, volumes, and containers, creating multiplicative load when many Compose projects run simultaneously.
Resolution Strategies
Caching and Connection Pooling
Implement client-side caching to reduce redundant API calls. Most monitoring queries (container list, image list) can be cached for 5-30 seconds without significant staleness:
#!/usr/bin/env python3
"""
Docker API client with integrated caching to prevent bottleneck amplification.
Uses a TTL-based in-memory cache to coalesce identical requests.
"""
import json
import time
import threading
import requests_unixsocket
from collections import OrderedDict
from typing import Any, Optional
class DockerAPICache:
def __init__(self, socket_path: str = '/var/run/docker.sock',
default_ttl: float = 15.0, max_size: int = 100):
self.socket_path = socket_path
self.base_url = f'http://localhost'
self.default_ttl = default_ttl
self.max_size = max_size
self._cache: OrderedDict = OrderedDict()
self._lock = threading.Lock()
self.session = requests_unixsocket.Session()
self.session.mount('http://localhost',
requests_unixsocket.UnixAdapter(socket_path))
def _cache_key(self, endpoint: str, params: str) -> str:
return f"{endpoint}:{params}"
def _evict_expired(self):
now = time.monotonic()
expired_keys = []
for key, (timestamp, _) in self._cache.items():
if now - timestamp > self.default_ttl:
expired_keys.append(key)
else:
break # OrderedDict, oldest first
for key in expired_keys:
del self._cache[key]
def get(self, endpoint: str, params: str = '', ttl: Optional[float] = None) -> Any:
cache_key = self._cache_key(endpoint, params)
ttl = ttl or self.default_ttl
with self._lock:
self._evict_expired()
if cache_key in self._cache:
timestamp, data = self._cache[cache_key]
if time.monotonic() - timestamp <= ttl:
# Move to end (most recently used)
self._cache.move_to_end(cache_key)
return data
# Cache miss or expired - fetch from daemon
url = f"{self.base_url}{endpoint}"
if params:
url += f"?{params}"
try:
response = self.session.get(url, timeout=10)
response.raise_for_status()
data = response.json()
except Exception as e:
print(f"API call failed: {e}")
# Return stale data if available as fallback
with self._lock:
if cache_key in self._cache:
_, stale_data = self._cache[cache_key]
return stale_data
raise
with self._lock:
# Evict oldest if at capacity
while len(self._cache) >= self.max_size:
self._cache.popitem(last=False)
self._cache[cache_key] = (time.monotonic(), data)
self._cache.move_to_end(cache_key)
return data
def list_containers(self, all: bool = True, limit: int = 100) -> list:
params = f"all={str(all).lower()}&limit={limit}"
return self.get('/containers/json', params, ttl=10)
def list_images(self) -> list:
return self.get('/images/json', ttl=60)
def inspect_container(self, container_id: str) -> dict:
return self.get(f'/containers/{container_id}/json', ttl=5)
# Usage example
cache = DockerAPICache()
# First call hits the daemon
containers = cache.list_containers()
print(f"Found {len(containers)} containers")
# Second call within TTL returns cached data instantly
containers_again = cache.list_containers()
print(f"Cache hit: {len(containers_again)} containers (instant)")
Batching API Calls
Instead of making N individual container inspect calls, batch them into a single query where possible. The Docker API supports querying multiple containers with filters:
#!/usr/bin/env python3
"""
Demonstrates batching Docker API calls to reduce request count.
Instead of looping and inspecting each container individually,
use filter parameters to fetch relevant data in one call.
"""
import requests_unixsocket
import time
def inefficient_approach(socket_path: str, container_ids: list) -> dict:
"""Makes N individual inspect calls - causes bottleneck under load."""
session = requests_unixsocket.Session()
session.mount('http://localhost', requests_unixsocket.UnixAdapter(socket_path))
results = {}
for cid in container_ids:
try:
resp = session.get(f'http://localhost/containers/{cid}/json', timeout=5)
results[cid] = resp.json()
except Exception as e:
results[cid] = {'error': str(e)}
return results
def efficient_approach(socket_path: str, label_filter: str) -> dict:
"""Uses a single filtered list call to get all matching containers at once."""
session = requests_unixsocket.Session()
session.mount('http://localhost', requests_unixsocket.UnixAdapter(socket_path))
# Fetch all containers matching the label filter in one API call
url = f'http://localhost/containers/json?all=true&filters={{"label":["{label_filter}"]}}'
resp = session.get(url, timeout=10)
containers = resp.json()
return {c['Id']: c for c in containers}
# Benchmark comparison
def benchmark():
socket = '/var/run/docker.sock'
# Simulate 50 container IDs
test_ids = ['abc' + str(i) for i in range(50)]
start = time.monotonic()
inefficient_approach(socket, test_ids)
inefficient_time = time.monotonic() - start
start = time.monotonic()
efficient_approach(socket, 'com.example.project=myapp')
efficient_time = time.monotonic() - start
print(f"Inefficient (50 individual calls): {inefficient_time:.3f}s")
print(f"Efficient (1 filtered call): {efficient_time:.3f}s")
print(f"Reduction: {((inefficient_time - efficient_time) / inefficient_time * 100):.1f}% fewer API calls")
benchmark()
Docker Daemon Tuning
Configure the Docker daemon for higher throughput by adjusting its runtime parameters. Create or modify /etc/docker/daemon.json:
{
"max-concurrent-downloads": 10,
"max-concurrent-uploads": 10,
"max-download-attempts": 5,
"metrics-addr": "0.0.0.0:9323",
"log-level": "warn",
"containerd-namespace": "docker",
"default-ulimits": {
"nofile": {
"Name": "nofile",
"Hard": 64000,
"Soft": 64000
}
},
"builder": {
"max-concurrent-builds": 4
}
}
Key tuning parameters explained:
- max-concurrent-downloads: Controls parallel layer pulls. Increase on hosts with fast storage and network to reduce image pull latency, but be cautious as too many concurrent pulls can saturate disk I/O.
- max-concurrent-uploads: Similar for pushing images. Tune based on your push frequency from CI systems.
- log-level warn: Reduces daemon CPU overhead from debug logging. In production, avoid debug level which adds significant overhead per API call.
- builder max-concurrent-builds: Limits parallel image builds. Set based on available CPU cores and memory.
- default-ulimits nofile: Increases file descriptor limits for containers, indirectly reducing "too many open files" errors that cause API retries.
After modifying daemon.json, restart the Docker daemon:
# Reload systemd configuration and restart
sudo systemctl daemon-reload
sudo systemctl restart docker
# Verify the new settings took effect
docker info | grep -A 5 "Concurrent"
Rate Limiting and Circuit Breakers
Implement client-side rate limiting and circuit breakers to protect the Docker API from overload. This is especially important for monitoring tools that might otherwise hammer the API during incident conditions:
#!/usr/bin/env python3
"""
Rate-limited Docker API client with circuit breaker pattern.
Prevents clients from overwhelming the Docker daemon during
high-load scenarios or when the daemon is already degraded.
"""
import time
import threading
from datetime import datetime, timedelta
from collections import deque
import requests_unixsocket
from enum import Enum
class CircuitState(Enum):
CLOSED = "closed" # Normal operation
OPEN = "open" # Failing, reject immediately
HALF_OPEN = "half_open" # Testing if recovered
class DockerAPICircuitBreaker:
def __init__(self, socket_path: str = '/var/run/docker.sock',
failure_threshold: int = 5,
recovery_timeout: float = 60.0,
half_open_max_requests: int = 3,
rate_limit_per_second: float = 10.0):
self.socket_path = socket_path
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout
self.half_open_max_requests = half_open_max_requests
# Circuit breaker state
self.state = CircuitState.CLOSED
self.failure_count = 0
self.last_failure_time: Optional[datetime] = None
self.half_open_requests = 0
self._state_lock = threading.Lock()
# Rate limiter state (token bucket)
self.rate = rate_limit_per_second
self.tokens = rate_limit_per_second
self.last_refill = time.monotonic()
self._rate_lock = threading.Lock()
# Session
self.session = requests_unixsocket.Session()
self.session.mount('http://localhost',
requests_unixsocket.UnixAdapter(socket_path))
def _refill_tokens(self):
now = time.monotonic()
elapsed = now - self.last_refill
self.tokens = min(self.rate, self.tokens + elapsed * self.rate)
self.last_refill = now
def _acquire_token(self) -> bool:
with self._rate_lock:
self._refill_tokens()
if self.tokens >= 1.0:
self.tokens -= 1.0
return True
return False
def _check_circuit(self) -> bool:
"""Returns True if request can proceed, False if circuit is open."""
with self._state_lock:
now = datetime.now()
if self.state == CircuitState.CLOSED:
if self.failure_count >= self.failure_threshold:
self.state = CircuitState.OPEN
self.last_failure_time = now
print(f"CIRCUIT OPEN: {self.failure_count} consecutive failures")
return False
return True
elif self.state == CircuitState.OPEN:
if self.last_failure_time and \
(now - self.last_failure_time).total_seconds() >= self.recovery_timeout:
self.state = CircuitState.HALF_OPEN
self.half_open_requests = 0
print("CIRCUIT HALF_OPEN: Testing daemon recovery")
return True
return False
elif self.state == CircuitState.HALF_OPEN:
if self.half_open_requests < self.half_open_max_requests:
self.half_open_requests += 1
return True
return False
return True
def _record_success(self):
with self._state_lock:
if self.state == CircuitState.HALF_OPEN:
self.state = CircuitState.CLOSED
self.failure_count = 0
print("CIRCUIT CLOSED: Daemon recovered successfully")
self.failure_count = 0
def _record_failure(self):
with self._state_lock:
self.failure_count += 1
self.last_failure_time = datetime.now()
if self.state == CircuitState.HALF_OPEN:
self.state = CircuitState.OPEN
print("CIRCUIT REOPENED: Half-open test failed")
def request(self, endpoint: str, method: str = 'GET', timeout: float = 10.0):
"""Make a rate-limited, circuit-breaker-protected API call."""
# Check circuit breaker first
if not self._check_circuit():
raise Exception(f"Docker API circuit is OPEN - rejecting request to {endpoint}")
# Rate limit
while not self._acquire_token():
time.sleep(0.1)
url = f"http://localhost{endpoint}"
try:
if method == 'GET':
response = self.session.get(url, timeout=timeout)
elif method == 'POST':
response = self.session.post(url, timeout=timeout)
else:
raise ValueError(f"Unsupported method: {method}")
if response.status_code >= 500:
self._record_failure()
raise Exception(f"Server error: {response.status_code}")
self._record_success()
return response.json()
except requests_unixsocket.ConnectionError as e:
self._record_failure()
raise Exception(f"Connection failed: {e}")
except Exception as e:
self._record_failure()
raise
# Usage with adaptive behavior
client = DockerAPICircuitBreaker(
rate_limit_per_second=5.0,
failure_threshold=3,
recovery_timeout=30.0
)
def safe_container_list():
try:
containers = client.request('/containers/json?all=true&limit=50')
return containers
except Exception as e:
print(f"Protected call failed: {e}")
# Fallback to cached data or degraded mode
return []
containers = safe_container_list()
print(f"Safely retrieved {len(containers) if containers else 0} containers")
Practical Code Examples
Python Bottleneck Detector
This comprehensive detector combines latency monitoring, connection counting, and event burst analysis into a single diagnostic tool that you can run on any Docker host:
#!/usr/bin/env python3
"""
Docker API Bottleneck Detector - Comprehensive diagnostic tool.
Monitors latency, connection saturation, event bursts, and daemon health.
"""
import json
import os
import time
import subprocess
import threading
import requests_unixsocket
from datetime import datetime
from collections import deque
from typing import Dict, List, Tuple
class BottleneckDetector:
"""Real-time Docker API bottleneck detection and reporting."""
def __init__(self, socket_path: str = '/var/run/docker.sock'):
self.socket_path = socket_path
self.session = requests_unixsocket.Session()
self.session.mount('http://localhost',
requests_unixsocket.UnixAdapter(socket_path))
# Rolling windows for trend analysis
self.latency_window = deque(maxlen=60) # 60 samples
self.error_window = deque(maxlen=60)
self.event_window = deque(maxlen=60)
# Thresholds
self.latency_warning = 2.0 # seconds
self.latency_critical = 5.0 # seconds
self.error_rate_warning = 0.1 # 10% error rate
self.connection_warning = 20 # concurrent socket connections
self.event_burst_warning = 30 # events per second
# State
self.running = True
self.last_event_count = 0
self.last_event_time = time.monotonic()
def measure_latency(self, endpoint: str = '/containers/json',
params: str = 'all=true&limit=50') -> Tuple[float, int]:
"""Measure API call latency for a given endpoint."""
url = f"http://localhost{endpoint}?{params}"
start = time.monotonic()
try:
resp = self.session.get(url, timeout=15)
latency = time.monotonic() - start
return latency, resp.status_code
except Exception as e:
latency = time.monotonic() - start
return latency, 0 # 0 indicates connection failure
def count_socket_connections(self) -> int:
"""Count active connections to the Docker socket."""
try:
result = subprocess.run(
['lsof', self.socket_path, '-t'],
capture_output=True, text=True, timeout=5
)
return len(result.stdout.strip().split('\n'))
except Exception:
# Fallback: count via /proc/net/unix
try:
with open('/proc/net/unix', 'r') as f:
content = f.read()
return content.count('docker.sock')
except Exception:
return -1
def get_event_rate(self) -> float:
"""Calculate Docker event rate per second."""
try:
result = subprocess.run(
['docker', 'events', '--since', '5s', '--until', '0s',
'--format', '{{.Action}}'],
capture_output=True, text=True, timeout=10
)
event_count = len(result.stdout.strip().split('\n'))
return event_count / 5.0
except Exception:
return 0.0
def check_daemon_health(self) -> Dict[str, any]:
"""Check Docker daemon health via /info endpoint."""
try:
resp = self.session.get('http://localhost/info', timeout=5)
if resp.status_code == 200:
info = resp.json()
return {
'status': 'healthy',
'containers': info.get('Containers', 0),
'images': info.get('Images', 0),
'mem_total': info.get('MemTotal', 0),
'cpu_count': info.get('NCPU', 0),
'driver': info.get('Driver', 'unknown')
}
return {'status': 'unhealthy', 'code': resp.status_code}
except Exception as e:
return {'status': 'unreachable', 'error': str(e)}
def diagnose(self) -> Dict[str, any]:
"""Run full diagnostic suite."""
report = {
'timestamp': datetime.now().isoformat(),
'socket_path': self.socket_path,
'checks': {}
}
# Latency check on multiple endpoints
endpoints = [
('/containers/json', 'all=true&limit=10'),
('/images/json', ''),
('/info', ''),
('/version', '')
]
latencies = {}
for endpoint, params in endpoints:
latency, status = self.measure_latency(endpoint, params)
name = endpoint.split('/')[-1] or 'root'
latencies[name] = {'latency': round(latency, 3), 'status': status}
self.latency_window.append(latency)
report['checks']['latencies'] = latencies
# Calculate latency statistics
if self.latency_window:
avg_latency = sum(self.latency_window) / len(self.latency_window)
p95_index = int(len(self.latency_window) * 0.95)
sorted_latencies = sorted(self.latency_window)
p95_latency = sorted_latencies[min(p95_index, len(sorted_latencies) - 1)]
report['checks']['latency_stats'] = {
'avg': round(avg_latency, 3),
'p95': round(p95_latency, 3),
'max': round(max(self.latency_window), 3)
}
# Connection saturation check
connections = self.count_socket_connections()
report['checks']['socket_connections'] = connections
# Event rate check
event_rate = self.get_event_rate()
report['checks']['event_rate_per_sec'] = round(event_rate, 2)
# Daemon health
report['checks']['daemon_health'] = self.check_daemon_health()
# Generate alerts
alerts = []
# Latency alerts
if latencies.get('json', {}).get('latency', 0) > self.latency_critical:
alerts.append({
'severity': 'CRITICAL',
'message': f"Container list latency {latencies['json']['latency']}s exceeds critical threshold {self.latency_critical}s"
})
elif latencies.get('json', {}).get('latency', 0) > self.latency_warning:
alerts.append({
'severity': 'WARNING',
'message': f"Container list latency {latencies['json']['latency']}s exceeds warning threshold {self.latency_warning}s"
})
# Connection alerts
if connections > self.connection_warning * 2:
alerts.append({
'severity': 'CRITICAL',
'message': f"{connections} concurrent socket connections - socket saturation likely"
})
elif connections > self.connection_warning:
alerts.append({
'severity': 'WARNING',
'message': f"{connections} concurrent socket connections approaching saturation"
})
# Event burst alerts
if event_rate > self.event_burst_warning:
alerts.append({
'severity': 'WARNING',
'message': f"Event rate {event_rate:.1f}/sec indicates possible container churn bottleneck"
})
# Daemon health alerts
daemon_status = report['checks']['daemon_health'].get('status', 'unknown')
if daemon_status != 'healthy':
alerts.append({
'severity': 'CRITICAL',
'message': f"Docker daemon health check failed: {daemon_status}"
})
report['alerts'] = alerts
report['alert_count'] = len(alerts)
return report
def run_continuous(self, interval: float = 15.0):
"""Run continuous monitoring with periodic diagnostics."""
print(f"Starting Docker API Bottleneck Detector")
print(f"Socket: {self.socket_path}")
print(f"Check interval: {interval}s")
print("-" * 60)
while self.running:
report = self.diagnose()
# Print status
status_char = "✓" if report['alert_count'] == 0 else "⚠"
print(f"\n{status_char} [{report['timestamp']}]")
latency_stats = report['checks'].get('latency_stats', {})
print(f" Latency: avg={latency_stats.get('avg', 'N/A')}s "
f"p95={latency_stats.get('p95', 'N/A')}s")
print(f" Connections: {report['checks'].get('socket_connections', 'N/A')}")
print(f" Event Rate: {report['checks'].get('event_rate_per_sec', 'N/A')}/s")
if report['alerts']:
print(f" ALERTS ({report['alert_count']}):")
for alert in report['alerts']:
print(f" [{alert['severity']}] {alert['message']}")
time.sleep(interval)
# Run the detector
if __name__ == '__main__':
import argparse
parser = argparse.ArgumentParser(description='Docker API Bottleneck Detector')
parser.add_argument('--socket', default='/var