Docker Compose GPU Access: Best Practices and Common Pitfalls

What Is Docker Compose GPU Access?

Docker Compose GPU access refers to the ability to expose physical or logical GPU devices from the host machine to containers defined in a docker-compose.yml file. This allows containerized workloads—such as machine learning training, inference servers, video transcoding pipelines, or scientific simulations—to leverage the massive parallel computing power of NVIDIA, AMD, or Intel GPUs directly, without the overhead of virtualization layers.

At its core, GPU access in Docker Compose is about making the host's GPU devices and their driver stacks visible inside the container namespace. This is accomplished through a combination of kernel-level device exposure, driver library mounting, and container runtime configuration. For NVIDIA GPUs, this is orchestrated by the NVIDIA Container Toolkit (formerly nvidia-docker2), which integrates with Docker's runtime layer to inject the necessary device nodes and libraries automatically.

A Brief History of GPU Containerization

Docker's GPU support has evolved significantly:

Docker 1.x era (pre-2017): GPUs were accessed by manually mounting /dev/nvidia* devices and library paths—fragile and error-prone.
nvidia-docker (2017): NVIDIA introduced a dedicated Docker runtime that automated device injection and library mounting.
Docker 19.03+ (2019): The --gpus flag was introduced natively in Docker, making GPU access a first-class feature.
Docker Compose 2.3+ / v3.x: GPU support was integrated into the Compose specification through the deploy stanza and runtime properties, though with important version-specific nuances.

Why GPU Access Matters in Docker Compose

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Modern development workflows increasingly rely on containerization for reproducibility and portability. When those workflows involve GPU-accelerated computing, proper Compose-level GPU configuration becomes critical for several reasons:

Multi-service orchestration: A typical ML pipeline might include a training service, a TensorBoard visualization service, a Redis caching layer, and a PostgreSQL metadata store—all defined in a single Compose file. Only some services need GPU access, and Compose lets you selectively assign GPUs.
Resource isolation and allocation: With multiple GPU-capable services (e.g., an inference API and a background training job), you can pin specific GPUs to specific containers, preventing resource contention.
Reproducible environments: Compose files committed alongside code ensure every developer and CI/CD pipeline uses identical GPU configurations, eliminating the "it works on my machine" problem for GPU workloads.
Fractional GPU sharing: With technologies like NVIDIA MIG (Multi-Instance GPU) or MPS (Multi-Process Service), Compose can define containers that use only a slice of a GPU, enabling efficient multi-tenant GPU utilization.

Without proper GPU configuration in Compose, containers default to CPU-only execution, which can turn a 2-hour GPU training job into a 2-day CPU slog—or worse, cause the application to crash when CUDA libraries are missing.

How to Enable GPU Access in Docker Compose

There are three primary approaches to configure GPU access in a docker-compose.yml file, and the choice depends on your Docker version, Compose specification version, and whether you're operating in Swarm mode or standalone mode. Below I'll walk through each method with complete, runnable examples.

Prerequisites Check

Before diving into Compose configurations, verify your host is properly set up:

# 1. Verify NVIDIA drivers are installed and GPU is visible
nvidia-smi

# 2. Verify Docker is installed (19.03 or newer recommended)
docker --version

# 3. Install NVIDIA Container Toolkit
# Debian/Ubuntu:
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

# 4. Verify the nvidia runtime is registered
docker info | grep -i nvidia
# Should show: "runtimes: nvidia runc" or similar

# 5. Test GPU access in a standalone container
docker run --rm --gpus all nvidia/cuda:12.1-base nvidia-smi

Method 1: The `deploy` Resource Reservation (Recommended for Compose v3.x)

This is the most modern and portable approach, defined under the deploy key which aligns with the Docker Swarm resource specification. It works with docker-compose (standalone) and docker stack deploy (Swarm). The key is the devices array under resources.reservations.

# docker-compose.yml
version: '3.8'

services:
  trainer:
    image: nvidia/cuda:12.1-runtime-ubuntu22.04
    command: nvidia-smi
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    # Optional: specify exact GPU UUIDs or device indices
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           device_ids: ['0', '2']  # Use only GPU 0 and GPU 2
    #           capabilities: [gpu]

  web-api:
    image: my-inference-api:latest
    ports:
      - "8080:8080"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    environment:
      - NVIDIA_VISIBLE_DEVICES=0  # Redundancy: restrict to GPU 0

The driver field must be set to nvidia for NVIDIA GPUs. The count field specifies how many GPUs to allocate (use all or an integer). The capabilities array defines what the container can do—[gpu] gives full access, while [utility, compute] or [graphics, compute, display] can be used for more granular control depending on the driver version.

Important: When using this method, you must run docker-compose up with no special flags—the reservations are processed automatically. However, if you're on an older Docker version (<19.03), you may need to fall back to Method 2.

Method 2: The `runtime` Property (Compose v2.x / Legacy)

Before the deploy.resources.reservations.devices syntax was widely supported in standalone Compose, the runtime property was the standard way to enable GPUs. This requires your Compose file to use a version 2.x schema (or a hybrid approach).

# docker-compose.yml
version: '2.3'

services:
  gpu-app:
    image: tensorflow/tensorflow:latest-gpu
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    command: python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
    volumes:
      - ./models:/models

When using runtime: nvidia, Docker invokes the NVIDIA Container Toolkit runtime which automatically mounts /usr/bin/nvidia-container-runtime as the OCI runtime. This method is straightforward but less portable—it ties your Compose file to a specific runtime name and doesn't work in Swarm mode.

Caveat: If you attempt to use runtime: nvidia in a Compose file with version: '3.x', Docker Compose will silently ignore the runtime property (or throw a warning). This is one of the most common pitfalls developers encounter.

Method 3: Explicit Device Mapping (Universal Fallback)

The most low-level approach—directly mapping GPU device files and library paths—works across all Docker versions and Compose specifications but requires you to know exactly which devices and libraries your application needs.

# docker-compose.yml
version: '3.8'

services:
  gpu-app:
    image: nvidia/cuda:12.1-runtime-ubuntu22.04
    command: nvidia-smi
    devices:
      - /dev/nvidia0:/dev/nvidia0
      - /dev/nvidia1:/dev/nvidia1
      - /dev/nvidiactl:/dev/nvidiactl
      - /dev/nvidia-uvm:/dev/nvidia-uvm
      - /dev/nvidia-modeset:/dev/nvidia-modeset
    volumes:
      - /usr/bin/nvidia-smi:/usr/bin/nvidia-smi
      - /usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1
      - /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1
    environment:
      - LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu

While this method is brittle and maintenance-heavy (driver updates can change library paths), it's invaluable when debugging why other methods fail or when working in heavily restricted environments where the NVIDIA Container Toolkit cannot be installed.

Hybrid Approach: Using `environment` Variables for Fine-Grained Control

The NVIDIA Container Toolkit respects several environment variables that give you per-container control over GPU visibility and capabilities, even when using the deploy or runtime methods:

# docker-compose.yml
version: '3.8'

services:
  inference-server:
    image: my-triton-server:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    environment:
      # Restrict to specific GPU UUIDs (overrides deploy count)
      - NVIDIA_VISIBLE_DEVICES=GPU-3ab5c9f2-1a2b-4c3d-5e6f-7a8b9c0d1e2f
      # Enable specific driver capabilities
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
      # Disable automatic library mounting if you want manual control
      - NVIDIA_DISABLE_REQUIRE=1
    volumes:
      - /custom/cuda/libs:/usr/local/cuda/lib64:ro

This hybrid approach combines the convenience of the deploy reservation with the granularity of environment variables, giving you the best of both worlds.

Best Practices for Docker Compose GPU Access

1. Pin Your CUDA and Driver Versions

GPU container images are tightly coupled to the host's NVIDIA driver version due to CUDA's forward-compatibility model. A container built with CUDA 12.1 requires a host driver that supports CUDA 12.1 (driver version >= 530). Always pin both the image tag and document the minimum driver requirement:

# docker-compose.yml
version: '3.8'

services:
  training-job:
    # Pin exact CUDA version—don't use :latest
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    # Document driver requirement for ops team
    labels:
      - "nvidia.driver.minimum=530.41.03"
      - "cuda.version=12.1"

2. Use Specific Device IDs for Multi-GPU Setups

On multi-GPU machines, avoid using count: all for every service. Instead, explicitly assign GPU indices or UUIDs to prevent resource contention and enable predictable performance:

services:
  inference-gpu0:
    image: inference-server:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
    ports:
      - "8000:8000"

  inference-gpu1:
    image: inference-server:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['1']
              capabilities: [gpu]
    ports:
      - "8001:8000"

  background-trainer:
    image: training-job:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['2', '3']  # GPUs 2 and 3 for training
              capabilities: [gpu]

3. Always Include Health Checks for GPU Services

GPU containers can appear "running" even when CUDA initialization fails. Add a health check that verifies actual GPU accessibility:

services:
  gpu-app:
    image: my-cuda-app:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "nvidia-smi", "||", "exit", "1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s

4. Prefer the `deploy` Syntax Over `runtime`

The deploy.resources.reservations.devices approach is the direction Docker is heading. It's Swarm-compatible, more expressive (supports device_ids, count, capabilities), and doesn't require consumers of your Compose file to know about runtime names. Use version: '3.8' or higher and avoid runtime: nvidia unless you have a specific legacy constraint.

5. Separate GPU and Non-GPU Services

Not every service in your stack needs a GPU. Only attach GPUs to services that actually use them—this conserves GPU resources and makes your Compose file self-documenting about which components are accelerated:

services:
  # GPU-accelerated services
  llm-inference:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  # Non-GPU services—no deploy.resources block needed
  redis-cache:
    image: redis:7-alpine

  postgres:
    image: postgres:16

  api-gateway:
    image: nginx:alpine
    ports:
      - "80:80"

6. Validate Your Configuration Before Deployment

Use docker-compose config to render and validate your Compose file, catching syntax errors early:

# Render the effective Compose configuration
docker-compose config > rendered-compose.yml

# Check that GPU reservations appear correctly
grep -A 10 "devices:" rendered-compose.yml

# Dry-run with a GPU check container
docker-compose run --rm gpu-app nvidia-smi

Common Pitfalls and How to Avoid Them

Pitfall 1: Using `runtime: nvidia` with Compose v3.x

The problem: Developers often copy legacy examples and place runtime: nvidia inside a version: '3.x' Compose file. Docker Compose silently ignores the runtime key in v3.x schemas (it's only recognized in v2.x), and the container runs without GPU access.

The fix: Either switch to version: '2.3' if you must use runtime, or (better) migrate to the deploy.resources.reservations.devices syntax:

# ❌ BROKEN: runtime ignored in v3.x
version: '3.8'
services:
  app:
    runtime: nvidia  # This does NOTHING in v3.x
    image: nvidia/cuda:latest

# ✅ FIXED: Use deploy reservation
version: '3.8'
services:
  app:
    image: nvidia/cuda:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Pitfall 2: NVIDIA Container Toolkit Not Installed or Wrong Version

The problem: The deploy reservation syntax requires the NVIDIA Container Toolkit (specifically nvidia-container-runtime) to be registered as a Docker runtime. If it's missing, you'll see errors like:

Error response from daemon: could not select device driver "nvidia" 
with capabilities: [[gpu]]

The fix: Install the toolkit properly and restart Docker:

# Add NVIDIA package repositories
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-docker.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

# Verify runtime registration
docker run --rm --gpus all nvidia/cuda:12.1-base nvidia-smi

Pitfall 3: Driver-Version Mismatch Between Host and Container

The problem: You pull a container with CUDA 12.2, but your host driver only supports up to CUDA 12.1. The container starts but GPU-accelerated code crashes with:

CUDA error: no kernel image is available for execution on the device

The fix: Always check compatibility. Use nvidia-smi on the host to see the supported CUDA version:

# Host check
nvidia-smi | grep "CUDA Version"
# Output: "CUDA Version: 12.1"

# Then choose a container with matching or lower CUDA version
# ✅ Compatible: cuda:12.1-runtime, cuda:11.8-runtime
# ❌ Incompatible: cuda:12.2-runtime (requires newer driver)

Pitfall 4: Assuming `count: all` Shares GPUs Across Services

The problem: When multiple services request count: all, each container gets access to all GPUs, but there's no orchestration-level isolation. Two containers can simultaneously allocate memory on the same GPU, leading to out-of-memory (OOM) errors or performance thrashing.

The fix: Use device_ids to partition GPUs explicitly, or leverage NVIDIA MIG to create hardware-isolated GPU instances:

# Partition GPUs explicitly
services:
  service-a:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0', '1']
              capabilities: [gpu]
  service-b:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['2', '3']
              capabilities: [gpu]

Pitfall 5: GPU Access Works in `docker run` but Not in Compose

The problem: You successfully test GPU access with docker run --gpus all ..., but the same image fails in Compose. This typically happens because Compose uses a different code path for resource allocation.

The fix: Debug incrementally—first verify the Compose syntax is correct, then test with a minimal Compose file:

# Minimal GPU Compose test
version: '3.8'
services:
  test:
    image: nvidia/cuda:12.1-base
    command: nvidia-smi
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

# Run with explicit GPU flag (some Docker versions need this)
docker-compose up --force-recreate
# Or, if using older Docker Compose, try:
DOCKER_OPTS="--gpus all" docker-compose up

Pitfall 6: SELinux/AppArmor Blocking GPU Device Access

The problem: On systems with SELinux (CentOS, RHEL, Fedora) or AppArmor (Ubuntu with hardened profiles), container access to /dev/nvidia* can be blocked, resulting in "permission denied" errors even though the NVIDIA runtime is correctly configured.

The fix: For SELinux, apply the appropriate context label. For AppArmor, ensure the NVIDIA profiles are loaded:

# SELinux: Add security context to allow GPU access
# Option A: Use the container-selinux policy
sudo setenforce 0  # Temporarily test if SELinux is the culprit

# Option B: Apply proper label in Compose
services:
  gpu-app:
    security_opt:
      - label=type:container_t  # Or use :Z on volume mounts
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

# AppArmor: Load NVIDIA profiles
sudo apparmor_parser -r /etc/apparmor.d/nvidia-container-runtime

Pitfall 7: Using Compose GPU Syntax in CI/CD Environments Without GPUs

The problem: Your Compose file with GPU reservations works perfectly on your workstation but fails in a CI/CD pipeline that runs on CPU-only nodes.

The fix: Make GPU reservations conditional using Compose override files or environment-specific profiles:

# Base Compose file: docker-compose.yml (no GPU)
version: '3.8'
services:
  app:
    image: my-app:latest
    command: python main.py
    profiles:
      - cpu-only

# GPU override: docker-compose.gpu.yml
version: '3.8'
services:
  app:
    profiles:
      - gpu-enabled
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

# Usage:
# CPU-only: docker-compose --profile cpu-only up
# GPU: docker-compose -f docker-compose.yml -f docker-compose.gpu.yml --profile gpu-enabled up

Putting It All Together: A Complete Production-Ready Example

Here's a complete, production-oriented docker-compose.yml that incorporates all the best practices discussed above. It defines an LLM inference service with GPU access, a Redis cache for embeddings, and a monitoring sidecar—all with proper GPU isolation, health checks, and version pinning:

# docker-compose.yml
version: '3.8'

# Networks for service isolation
networks:
  inference-net:
    driver: bridge
  monitoring-net:
    driver: bridge

# Volumes for persistent data
volumes:
  model-cache:
  redis-data:

services:
  # =============================================
  # GPU-Accelerated LLM Inference Service
  # =============================================
  llm-inference:
    image: ghcr.io/myorg/llm-server:v2.1.0-cuda12.1  # Pinned version + CUDA
    container_name: llm-inference-prod
    restart: unless-stopped
    
    # GPU configuration: dedicated GPU 0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
    
    # Environment for fine-grained GPU control
    environment:
      - NVIDIA_VISIBLE_DEVICES=0
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
      - CUDA_VISIBLE_DEVICES=0  # Framework-level restriction
      - MODEL_PATH=/models/llama-70b
      - REDIS_URL=redis://embeddings-cache:6379
      - LOG_LEVEL=INFO
    
    # Health check that verifies real GPU access
    healthcheck:
      test: ["CMD-SHELL", "nvidia-smi -L | grep GPU || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    
    ports:
      - "8080:8080"   # REST API
      - "9090:9090"   # gRPC
    
    volumes:
      - model-cache:/models:ro  # Read-only model mount
      - /run/nvidia-topology:/run/nvidia-topology:ro  # NVLink topology
    
    networks:
      - inference-net
    
    # Resource limits beyond GPU
    mem_limit: 32g
    cpus: 8
    
    labels:
      - "nvidia.driver.minimum=530.41.03"
      - "service.type=gpu-inference"
      - "monitoring.enable=true"

  # =============================================
  # Embeddings Cache (CPU-only)
  # =============================================
  embeddings-cache:
    image: redis:7.2-alpine
    container_name: redis-embeddings
    restart: unless-stopped
    
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5
    
    volumes:
      - redis-data:/data
    
    networks:
      - inference-net
    
    # No GPU reservation needed—clean separation
    mem_limit: 4g
    cpus: 2

  # =============================================
  # GPU Monitoring Sidecar
  # =============================================
  gpu-exporter:
    image: nvidia/dcgm-exporter:3.3.5
    container_name: gpu-metrics
    restart: unless-stopped
    
    # Access all GPUs for monitoring, but read-only
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [utility]  # Utility only—no compute
    
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=utility
    
    ports:
      - "9400:9400"
    
    networks:
      - monitoring-net
    
    volumes:
      - /run/nvidia-topology:/run/nvidia-topology:ro

  # =============================================
  # Prometheus scraper for GPU metrics
  # =============================================
  prometheus:
    image: prom/prometheus:v2.48.0
    container_name: gpu-prometheus
    restart: unless-stopped
    
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    
    ports:
      - "9091:9090"
    
    networks:
      - monitoring-net
    
    # No GPU needed

Running the Complete Stack

# 1. Validate the Compose file
docker-compose config | grep -A 15 "devices:"

# 2. Start the stack
docker-compose up -d

# 3. Verify GPU access in the inference service
docker-compose exec llm-inference nvidia-smi

# 4. Check GPU metrics
curl http://localhost:9400/metrics | grep DCGM

# 5. View logs to ensure no CUDA errors
docker-compose logs llm-inference | grep -i cuda

# 6. Stop everything
docker-compose down --volumes

Conclusion

Docker Compose GPU access has matured from a fragile, manual-device-mapping hack into a robust, declarative configuration model that integrates cleanly with modern container orchestration. The key to success is understanding which configuration method applies to your Docker version and Compose specification, and then layering on best practices around version pinning, explicit GPU assignment, health validation, and environment-based fine-tuning.

Start with the deploy.resources.reservations.devices syntax as your default—it's the future-proof, Swarm-compatible approach. Fall back to runtime: nvidia only when legacy constraints demand it, and use explicit device mapping as a last resort or debugging tool. Always validate your setup with nvidia-smi from inside the container before trusting that GPU acceleration is actually working. By following the patterns and avoiding the pitfalls outlined in this tutorial, you'll build GPU-accelerated microservice stacks that are reproducible, predictable, and production-hardened.