Docker Compose GPU Access: Production Guide

Understanding GPU Access in Docker Compose

Docker Compose simplifies multi-container application management, but when your services rely on GPU acceleration — common in machine learning inference, training, video transcoding, or scientific computing — you need explicit GPU access. Modern Docker Compose (version 3.x and later) integrates GPU support directly into the service definition using the deploy.resources block, which aligns with the Docker Swarm resource model but works equally well with local Docker engine instances when combined with the appropriate container runtime.

This guide covers everything you need to configure GPU access in production Docker Compose stacks, from driver prerequisites to advanced multi-GPU and MIG setups, along with battle-tested best practices.

Why GPU Access Matters in Production

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

In production environments, GPU acceleration delivers critical performance gains:

Machine Learning Inference & Training: Reduces latency from seconds to milliseconds and training from days to hours.
Video Processing: Real-time transcoding, frame analysis, and streaming.
Scientific Simulations: Molecular dynamics, CFD, weather prediction.
Data Analytics: GPU-accelerated dataframe operations (cuDF, cuML).

Running these workloads inside containers ensures reproducibility, portability, and resource isolation, but only if the GPU is properly exposed. Docker Compose GPU support lets you declaratively assign GPUs to services, making your stack self-documenting and ready for orchestration.

Prerequisites for Docker Compose GPU Support

Before you can use GPU devices in Compose files, your host must meet these requirements:

NVIDIA Drivers: Properly installed and running. Verify with nvidia-smi.
NVIDIA Container Toolkit: This replaces the older nvidia-docker2. Install the nvidia-container-toolkit package and configure the Docker runtime.
Docker Engine >= 19.03: The GPU device request feature was introduced in this version.
Docker Compose v2.x or v1.27+: Full support for deploy.resources in non-Swarm contexts.

Configure the default Docker runtime to use nvidia by editing /etc/docker/daemon.json:

{
  "runtimes": {
    "nvidia": {
      "path": "nvidia-container-runtime",
      "runtimeArgs": []
    }
  },
  "default-runtime": "nvidia"
}

Restart Docker after the change. Now all containers can access GPUs by default, but we’ll control access per service inside Compose files.

Enabling GPU Access in Docker Compose: The deploy Block

The official way to request GPUs in a Compose service is through the deploy.resources.reservations.devices array. This is part of the Docker Compose Specification, compatible with both local development and Swarm clusters. A minimal GPU-accelerated service looks like:

version: '3.8'

services:
  gpu-app:
    image: tensorflow/tensorflow:latest-gpu
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    environment:
      - TF_ENABLE_ONEDNN_OPTS=0

Key points:

The devices list contains one or more device reservation objects.
driver must be set to nvidia (or amd for ROCm, intel for GPU, but NVIDIA is the most common).
count specifies how many GPUs to allocate from the available pool.
capabilities defines the required GPU features — usually [gpu].

Device Reservation Options Deep Dive

The full device specification supports:

driver: nvidia, amd, intel, or custom runtime driver.
count: Integer number of GPUs, or all to use every GPU on the host.
device_ids: Array of specific GPU UUIDs or indices (e.g., ["0", "2"] or ["GPU-abc123"]).
capabilities: List of capability strings. For NVIDIA, typical values are gpu, compute, utility, display, mig (for Multi-Instance GPU).
options: Driver-specific key-value pairs (rarely needed).

Specifying Exact GPUs

To pin a service to particular GPUs, use device_ids:

services:
  inference:
    image: my-bert-server:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["GPU-ebc4a5d2", "GPU-9f7c1a3e"]
              capabilities: [gpu]

This ensures the service uses only those two physical GPUs, leaving others free for different workloads. Using UUIDs is more stable than indices because indices can change after reboots.

Using All Available GPUs

services:
  distributed-trainer:
    image: pytorch/pytorch:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Multi-Instance GPU (MIG) Support

For NVIDIA A100 or H100 GPUs in MIG mode, you can request a specific MIG instance by adding the mig capability or using device IDs that correspond to MIG slices. To request a MIG compute instance:

services:
  mig-app:
    image: my-cuda-app:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [mig]

You can also target a particular MIG slice by its index (e.g., device_ids: ["0:0"]) after enumerating them with nvidia-smi mig -lci.

Production-Grade Compose File Examples

Let’s build a complete production stack with a GPU-accelerated inference API, a Redis cache, and a monitoring sidecar.

1. Single-GPU Inference Service

version: '3.8'

services:
  llm-server:
    image: ghcr.io/myrepo/llm-server:v2.1.3
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    ports:
      - "8080:8080"
    environment:
      MODEL_PATH: /models/llama-7b-q4
      CUDA_VISIBLE_DEVICES: "0"  # Redundant with device_ids but provides clarity
    volumes:
      - /mnt/models:/models:ro
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

Note: Even though the deploy.reservations block handles GPU assignment, setting CUDA_VISIBLE_DEVICES explicitly can help frameworks that don't fully rely on the container runtime.

2. Multi-GPU Training with Device Affinity

version: '3.8'

services:
  trainer-0:
    image: pytorch/pytorch:2.0.1-cuda11.8-cudnn8-devel
    hostname: worker-0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]
    volumes:
      - ./data:/workspace/data
    command: python train.py --rank 0 --master_addr scheduler
    networks:
      - training-net

  trainer-1:
    image: pytorch/pytorch:2.0.1-cuda11.8-cudnn8-devel
    hostname: worker-1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["1"]
              capabilities: [gpu]
    volumes:
      - ./data:/workspace/data
    command: python train.py --rank 1 --master_addr scheduler
    networks:
      - training-net

networks:
  training-net:
    driver: overlay  # Use bridge for local single-node

Each service is pinned to a distinct GPU, avoiding resource contention. In production, you might replace these with replicated services and dynamic GPU assignment using environment variables and an orchestrator.

3. Fractional GPU Sharing with MIG

version: '3.8'

services:
  mig-inference-1:
    image: nvcr.io/nvidia/tritonserver:23.10-py3
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0:1"]  # MIG slice 1 on GPU 0
              capabilities: [mig]
    environment:
      MIG_DEVICE: "0:1"

  mig-inference-2:
    image: nvcr.io/nvidia/tritonserver:23.10-py3
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0:2"]  # MIG slice 2 on GPU 0
              capabilities: [mig]

Best Practices for GPU Access in Production Compose Stacks

Explicitly specify device_ids in production: Avoid count: all unless you intend to monopolize the host. Use UUIDs to survive reboots.
Combine with environment variables: Some libraries (TensorFlow, PyTorch) honor CUDA_VISIBLE_DEVICES. Set it to match your device_ids for defense in depth.
Use health checks: GPU services can fail silently. A health endpoint that verifies GPU availability (e.g., runs a tiny CUDA kernel) ensures restarts.
Pin versions: GPU driver, CUDA runtime, and container toolkit versions must be compatible. Use specific image tags and document the driver version required.
Resource limits: GPU memory isn’t managed by Docker cgroups. Use nvidia-container-runtime environment variables like NVIDIA_REQUIRE_CUDA or application-level memory limits.
Isolate sensitive workloads: Place different services on separate GPUs or MIG slices. In Compose, you can use device_ids to enforce this partitioning.
Monitor GPU utilization: Integrate with nvidia-smi exporters (e.g., dcgm-exporter) as a sidecar container in the same Compose file to feed Prometheus.
Secure the runtime: Ensure the nvidia-container-toolkit is configured with no-cdi or properly restricted to prevent container escape.
Use Compose profiles: Define GPU and non-GPU variants of services using Compose profiles to maintain a single file that adapts to hardware availability.

Example: Monitoring Sidecar for GPU Metrics

services:
  app:
    image: my-gpu-app:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]

  dcgm-exporter:
    image: nvidia/dcgm-exporter:3.3.5
    pid: "host"
    volumes:
      - /run/prometheus:/run/prometheus
    environment:
      - DCGM_EXPORTER_INTERVAL=30000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ports:
      - "9400:9400"

The exporter collects metrics from all GPUs, including the one reserved by the app, providing a holistic view of cluster GPU usage.

Handling GPU Access in Docker Swarm Mode

When using Docker Compose with Swarm (via docker stack deploy), the deploy.resources.reservations.devices block is fully supported. However, GPU scheduling in Swarm requires careful planning:

Swarm does not automatically schedule containers onto nodes that have the requested GPUs. You must use node labels and placement constraints.
Label nodes with gpu=true or specific GPU IDs, then use placement.constraints in the Compose file.

services:
  gpu-service:
    image: my-image
    deploy:
      placement:
        constraints:
          - node.labels.gpu == true
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

For multi-node GPU clusters, consider using the generic-resource feature or an external scheduler like Kubernetes. Docker Compose remains ideal for single-node production or development setups.

Common Pitfalls and Troubleshooting

Missing NVIDIA runtime: If you see Error response from daemon: could not select device driver "nvidia", install the toolkit and set the default runtime.
GPU not visible inside container: Run nvidia-smi inside the container to check. If it fails, verify the capabilities list and that the driver is correctly specified.
Compose file version mismatch: The deploy key is ignored in docker-compose up for version 2 files. Use version 3.x (or omit version, relying on the Compose Specification).
Conflicting CUDA_VISIBLE_DEVICES: Some images set this environment variable at build time. Override it in your Compose environment to match your device reservation.

Conclusion

GPU access in Docker Compose has matured into a first-class feature that brings the convenience of declarative configuration to accelerated computing. By leveraging the deploy.resources.reservations.devices block, you can precisely assign GPUs, enforce isolation, and build reproducible production stacks. Combine it with device IDs, MIG slicing, health checks, and monitoring sidecars, and you have a powerful, portable platform for any GPU-dependent workload. Start with the examples above, adapt them to your hardware, and let your containers unlock the full potential of your accelerators.