← Back to DevBytes
Docker July 07, 2026 14 min read

Docker with NVIDIA GPU Acceleration: Production Guide

Introduction to Docker with NVIDIA GPU Acceleration

Running GPU-accelerated workloads inside Docker containers has transformed how machine learning engineers, data scientists, and DevOps teams deploy production-grade AI infrastructure. NVIDIA GPU acceleration in Docker allows containers to access the full power of CUDA-enabled GPUs directly, eliminating the performance overhead traditionally associated with virtualized environments. This guide covers everything from foundational concepts to production-ready deployment patterns.

What Is NVIDIA GPU Acceleration in Docker?

NVIDIA GPU acceleration in Docker refers to the ability of Docker containers to access physical NVIDIA GPUs installed on the host machine. This is achieved through the NVIDIA Container Toolkit (formerly known as nvidia-docker), which exposes GPU device files and CUDA driver libraries from the host into the container's runtime environment. Unlike CPU-only containers that are completely isolated from host hardware, GPU-enabled containers receive a controlled passthrough of specific GPU resources, allowing frameworks like TensorFlow, PyTorch, and custom CUDA applications to execute with near-native performance.

The core components involved are:

Why GPU Acceleration in Containers Matters for Production

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Production environments demand consistency, reproducibility, and efficient resource utilization. Running GPU workloads directly on bare metal or inside virtual machines introduces dependency conflicts, driver mismatches, and configuration drift. Containerizing GPU workloads solves these problems while unlocking additional benefits:

Prerequisites and Installation

Before diving into production configurations, ensure the host machine has the required components installed. This section walks through a complete setup on Ubuntu 22.04 LTS, which serves as a reference for other Linux distributions.

Step 1: Install NVIDIA Drivers on the Host

First, identify the GPU model and install the appropriate driver. For data center GPUs like the A100, H100, or consumer cards like the RTX 4090, use the NVIDIA official repository:

# Check for existing NVIDIA hardware
lspci | grep -i nvidia

# Add the NVIDIA driver repository
sudo apt update
sudo apt install -y ubuntu-drivers-common

# Auto-detect and install the recommended driver
sudo ubuntu-drivers autoinstall

# Reboot the system
sudo reboot

# After reboot, verify the driver loaded correctly
nvidia-smi

The nvidia-smi command should display the GPU model, driver version, CUDA version, and available memory. Note the CUDA Driver Version reported here — this is the maximum CUDA version supported by the host driver and determines which CUDA-compatible container images you can run.

Step 2: Install Docker Engine

If Docker is not already installed, set it up using the official Docker repository to ensure compatibility with the NVIDIA Container Toolkit:

# Remove old versions if present
sudo apt remove -y docker docker-engine docker.io containerd runc

# Install dependencies
sudo apt update
sudo apt install -y ca-certificates curl gnupg lsb-release

# Add Docker's official GPG key
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

# Add the repository
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker Engine
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

# Start Docker and verify
sudo systemctl enable docker
sudo systemctl start docker
docker --version

Step 3: Install the NVIDIA Container Toolkit

The NVIDIA Container Toolkit integrates with Docker's runtime layer, enabling GPU device passthrough. Install it as follows:

# Add the NVIDIA Container Toolkit package repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Install the toolkit
sudo apt update
sudo apt install -y nvidia-container-toolkit

# Configure Docker to use the NVIDIA runtime
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Verify the runtime is registered
docker info | grep -i runtime

The output should list nvidia among the available runtimes. At this point, Docker is fully configured to launch GPU-accelerated containers.

Running Your First GPU Container

With the infrastructure in place, test the setup with a simple CUDA container that queries GPU information:

# Pull a CUDA-enabled base image
docker pull nvidia/cuda:12.4.0-runtime-ubuntu22.04

# Run a container that executes nvidia-smi
docker run --rm --runtime=nvidia --gpus all \
  nvidia/cuda:12.4.0-runtime-ubuntu22.04 \
  nvidia-smi

If everything is configured correctly, you will see the same GPU information as running nvidia-smi directly on the host. The container successfully accessed the host GPU through the NVIDIA runtime bridge.

Key flags explained:

Understanding NVIDIA Container Image Variants

NVIDIA publishes several CUDA image flavors on Docker Hub. Choosing the right one is critical for production efficiency:

For production inference, prefer the runtime or cudnn-runtime variants. They are smaller, faster to pull, and contain only what the application needs at execution time.

Building Production-Ready GPU Docker Images

A production Dockerfile for GPU workloads must balance image size, security, and framework compatibility. Here is a well-structured example for a PyTorch inference service:

# syntax=docker/dockerfile:1
# ---- Base Image ----
FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04 AS base

# Prevent interactive prompts during package installation
ENV DEBIAN_FRONTEND=noninteractive

# Install system dependencies required by the application
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3.11 \
    python3.11-venv \
    python3-pip \
    libgl1-mesa-glx \
    libglib2.0-0 \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get clean

# Create a non-root user for security
RUN groupadd -r appuser && useradd -r -g appuser -m -d /home/appuser appuser

# ---- Build Dependencies Stage ----
FROM base AS build

# Install build tools and compile requirements
COPY requirements.txt /tmp/requirements.txt
RUN pip3 install --no-cache-dir --upgrade pip setuptools wheel \
    && pip3 install --no-cache-dir -r /tmp/requirements.txt

# ---- Final Production Image ----
FROM base AS production

# Copy only the installed packages from the build stage
COPY --from=build /usr/local/lib/python3.11/dist-packages /usr/local/lib/python3.11/dist-packages
COPY --from=build /usr/local/bin /usr/local/bin

# Copy application code
COPY --chown=appuser:appuser ./src /home/appuser/src
WORKDIR /home/appuser/src

# Switch to non-root user
USER appuser

# Preload common CUDA kernels to reduce first-inference latency
ENV CUDA_CACHE_PATH=/home/appuser/.cache/cuda
RUN python3 -c "import torch; torch.zeros(1).cuda()" 2>/dev/null || true

# Health check for GPU availability
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD python3 -c "import torch; assert torch.cuda.is_available(), 'GPU not available'" || exit 1

# Expose the inference API port
EXPOSE 8080

ENTRYPOINT ["python3", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

This multi-stage build pattern separates dependency compilation from the final image, keeping the production artifact lean. Key production considerations include:

GPU Resource Management in Production

Production deployments often share GPUs across multiple containers or require fine-grained resource control. Docker provides several mechanisms to manage GPU allocation:

Selecting Specific GPUs

# Assign only GPU 0 and GPU 2 to this container
docker run --rm --runtime=nvidia --gpus '"device=0,2"' \
  nvidia/cuda:12.4.0-runtime-ubuntu22.04 \
  nvidia-smi -L

Limiting GPU Memory and Compute Utilization

For multi-tenant scenarios where multiple containers share a GPU, use NVIDIA's MIG (Multi-Instance GPU) on supported hardware like the A100 or H100, or enforce soft limits via environment variables:

# Cap visible GPU memory to 8GB using NVIDIA_VISIBLE_DEVICES override
docker run --rm --runtime=nvidia \
  -e NVIDIA_VISIBLE_DEVICES=0 \
  -e NVIDIA_MEMORY_LIMIT=8192 \
  nvidia/cuda:12.4.0-runtime-ubuntu22.04 \
  nvidia-smi

For strict isolation on data center GPUs, configure MIG instances on the host first, then assign containers to specific MIG slices:

# On the host, enable MIG mode and create instances
sudo nvidia-smi -mig 1
sudo nvidia-smi mig -cgi 9,9,9,9,9,9,9  # Create 7 equal slices on an A100

# Launch container bound to a specific MIG instance
docker run --rm --runtime=nvidia \
  -e NVIDIA_VISIBLE_DEVICES=MIG- \
  nvidia/cuda:12.4.0-runtime-ubuntu22.04 \
  nvidia-smi

Docker Compose for Multi-Service GPU Workloads

In production, GPU workloads often run alongside companion services like Redis for caching, PostgreSQL for metadata, or NGINX for reverse proxying. Docker Compose orchestrates these multi-container deployments while ensuring GPU resources are properly declared:

# docker-compose.yml
version: '3.8'

services:
  inference-api:
    build:
      context: ./inference
      dockerfile: Dockerfile
    image: myregistry/inference-api:latest
    runtime: nvidia
    environment:
      NVIDIA_VISIBLE_DEVICES: '0'
      CUDA_VISIBLE_DEVICES: '0'
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    ports:
      - "8080:8080"
    volumes:
      - model-cache:/home/appuser/.cache
    healthcheck:
      test: ["CMD", "python3", "-c", "import torch; assert torch.cuda.is_available()"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

  redis-cache:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    restart: unless-stopped

  nginx-proxy:
    image: nginx:1.25-alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/certs:/etc/nginx/certs:ro
    depends_on:
      - inference-api
    restart: unless-stopped

volumes:
  model-cache:
  redis-data:

The deploy.resources.reservations.devices block is the Compose-native way to request GPU resources, compatible with both standalone Docker Compose and Docker Swarm mode. The capabilities: [gpu] field informs the orchestrator that this service requires GPU acceleration.

Kubernetes Integration for GPU Orchestration

For larger production clusters, Kubernetes provides GPU-aware scheduling. After installing the NVIDIA device plugin on your cluster, pods can request GPUs as first-class resources:

# gpu-inference-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: inference-api
  template:
    metadata:
      labels:
        app: inference-api
    spec:
      containers:
      - name: inference-container
        image: myregistry/inference-api:latest
        ports:
        - containerPort: 8080
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            nvidia.com/gpu: 1
        env:
        - name: CUDA_VISIBLE_DEVICES
          value: "0"
        readinessProbe:
          exec:
            command:
            - python3
            - -c
            - "import torch; assert torch.cuda.is_available()"
          initialDelaySeconds: 10
          periodSeconds: 30
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 60
      nodeSelector:
        nvidia.com/gpu.product: Tesla-A100-SXM4-40GB
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule

The nvidia.com/gpu: 1 resource request triggers the Kubernetes scheduler to place the pod on a node with an available GPU. Node selectors and tolerations further refine placement to specific GPU models and ensure pods land on GPU-enabled nodes.

Monitoring and Observability

Production GPU containers require continuous monitoring to detect performance degradation, memory leaks, and thermal issues. Integrate the following observability patterns:

GPU Metrics Collection with Prometheus

Deploy NVIDIA's Data Center GPU Manager (DCGM) exporter alongside your containers to expose GPU metrics in Prometheus format:

# Run DCGM exporter as a background container on each GPU node
docker run -d --name dcgm-exporter \
  --runtime=nvidia \
  --gpus all \
  -p 9400:9400 \
  nvidia/dcgm-exporter:3.3.5-3.4.0-ubuntu22.04

# Metrics available at http://localhost:9400/metrics include:
# - DCGM_FI_DEV_GPU_UTIL (GPU utilization percentage)
# - DCGM_FI_DEV_FB_USED (framebuffer memory used)
# - DCGM_FI_DEV_GPU_TEMP (GPU temperature in Celsius)
# - DCGM_FI_DEV_POWER_USAGE (power draw in watts)

Container-Level GPU Logging

Instrument application code to log GPU metrics at regular intervals. This example uses PyTorch's built-in utilities:

import torch
import logging
import time
from threading import Thread

logger = logging.getLogger("gpu-monitor")

def log_gpu_metrics(interval_seconds=30):
    """Periodically logs GPU memory and utilization."""
    while True:
        if torch.cuda.is_available():
            allocated = torch.cuda.memory_allocated() / 1024**3
            reserved = torch.cuda.memory_reserved() / 1024**3
            max_allocated = torch.cuda.max_memory_allocated() / 1024**3
            logger.info(
                f"GPU Memory: allocated={allocated:.2f}GB, "
                f"reserved={reserved:.2f}GB, "
                f"peak={max_allocated:.2f}GB"
            )
            torch.cuda.reset_peak_memory_stats()
        time.sleep(interval_seconds)

# Start the monitoring thread
monitor_thread = Thread(target=log_gpu_metrics, daemon=True)
monitor_thread.start()

Security Best Practices for GPU Containers

GPU containers inherit the same security principles as standard containers but introduce additional considerations due to direct hardware access:

Example of a hardened container launch:

docker run --rm --runtime=nvidia --gpus '"device=0"' \
  --read-only \
  --cap-drop=ALL \
  --cap-add=NET_BIND_SERVICE \
  --security-opt=no-new-privileges \
  --user 1000:1000 \
  -p 8080:8080 \
  myregistry/inference-api:latest

Optimizing Container Images for Production

Large container images slow down deployment velocity and increase storage costs. Apply these optimization techniques to GPU images:

Layer Caching Strategy

# Place rarely-changing layers first, frequently-changing layers last
# This maximizes Docker build cache utilization

FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04 AS base

# 1. System packages (rarely change)
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3.11 python3.11-venv && \
    rm -rf /var/lib/apt/lists/*

# 2. Framework wheels (change occasionally)
RUN pip3 install torch==2.3.0 torchvision==0.18.0

# 3. Application dependencies (change moderately)
COPY requirements.txt /tmp/
RUN pip3 install -r /tmp/requirements.txt

# 4. Application code (changes frequently)
COPY ./src /app/src

Image Size Comparison

Monitor image sizes to quantify optimization impact:

# Check image sizes
docker images --format "table {{.Repository}}:{{.Tag}}\t{{.Size}}" \
  | grep -E "cuda|inference"

# Typical sizes:
# nvidia/cuda:12.4.0-devel-ubuntu22.04  -> ~3.2GB (avoid in production)
# nvidia/cuda:12.4.0-runtime-ubuntu22.04 -> ~1.1GB
# Optimized multi-stage inference image   -> ~650MB

Common Pitfalls and Troubleshooting

Even well-planned GPU container deployments encounter issues. Here are frequent problems and their solutions:

CUDA Driver Version Mismatch

Symptom: Container exits with CUDA driver version is insufficient for CUDA runtime version.

Solution: The host NVIDIA driver imposes a maximum CUDA version. Use nvidia-smi on the host to check the driver version, then select a container image with a CUDA version less than or equal to that value. The CUDA runtime in the container can be older than the host driver, but never newer.

# Check host CUDA compatibility
nvidia-smi | grep "CUDA Version"

# If host shows "CUDA Version: 12.2", you can use:
# nvidia/cuda:12.2.0-runtime-ubuntu22.04
# nvidia/cuda:11.8.0-runtime-ubuntu22.04  (older is fine)
# But NOT: nvidia/cuda:12.4.0-runtime-ubuntu22.04 (too new)

GPU Not Visible Inside Container

Symptom: torch.cuda.is_available() returns False or nvidia-smi fails inside the container.

Diagnostic steps:

# 1. Verify GPU is visible on host
nvidia-smi

# 2. Verify NVIDIA runtime is registered
docker info | grep -i runtime

# 3. Check container runtime flag
docker inspect  | grep -i runtime

# 4. Check environment variables inside container
docker exec  env | grep NVIDIA

# 5. Verify device files are present
docker exec  ls -la /dev/nvidia*

Out of Memory (OOM) on GPU

Symptom: Container crashes with CUDA out of memory errors.

Solutions: Implement gradient checkpointing in training workloads, reduce batch sizes, use mixed-precision training (FP16/BF16), or allocate more GPU memory per container. For shared GPUs, enforce memory limits via NVIDIA_MEMORY_LIMIT or MIG partitioning.

Continuous Integration for GPU Containers

Automated testing of GPU containers in CI pipelines ensures that images are functional before deployment. While most CI runners lack physical GPUs, you can validate GPU-specific code paths using software fallbacks or cloud-based GPU runners:

# .github/workflows/gpu-build.yml (GitHub Actions with GPU runner)
name: GPU Container CI

on:
  push:
    branches: [main]

jobs:
  build-and-test:
    runs-on: gpu-runner  # Custom label for GPU-enabled runner
    steps:
      - uses: actions/checkout@v4

      - name: Build GPU container
        run: |
          docker build -t inference-api:ci-test \
            -f docker/Dockerfile .

      - name: Smoke test GPU availability
        run: |
          docker run --rm --runtime=nvidia --gpus all \
            inference-api:ci-test \
            python3 -c "import torch; assert torch.cuda.is_available()"

      - name: Run inference correctness tests
        run: |
          docker run --rm --runtime=nvidia --gpus all \
            -v ${{ github.workspace }}/test-data:/data \
            inference-api:ci-test \
            python3 -m pytest tests/ -v

      - name: Vulnerability scan
        run: |
          docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
            aquasec/trivy image --severity HIGH,CRITICAL \
            inference-api:ci-test

Conclusion

Docker with NVIDIA GPU acceleration provides a robust, production-tested foundation for deploying AI workloads at scale. By combining the NVIDIA Container Toolkit with disciplined image construction, resource management, and observability practices, teams can achieve consistent GPU performance across development, staging, and production environments. The key takeaways are: always match your container CUDA version to the host driver capability, prefer multi-stage builds with runtime base images, enforce security through non-root users and read-only filesystems, and integrate GPU-aware health checks and monitoring into your deployment topology. With these patterns in place, your GPU-accelerated container infrastructure will be ready to serve inference traffic, train models, and scale horizontally across Kubernetes clusters with minimal operational friction.

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles