What is Docker BuildKit

Docker BuildKit is a next-generation build engine that ships with Docker but was originally an opt-in feature. Starting from Docker 23.0, BuildKit became the default builder for the docker build command. It replaces the legacy builder with a highly performant, concurrent, and extensible architecture designed to solve many long-standing pain points in Docker image construction.

BuildKit introduces a new processing model based on a directed acyclic graph (DAG) of build operations. Rather than executing Dockerfile instructions sequentially from top to bottom, BuildKit analyzes the full dependency graph and executes independent stages in parallel. This fundamental shift unlocks a suite of advanced capabilities: efficient layer caching, secure secret injection, SSH agent forwarding, cache mounts for package managers, and output customization—all while maintaining backward compatibility with existing Dockerfiles.

Why BuildKit Matters

The legacy Docker builder processes your Dockerfile as a linear script. Every instruction runs in strict order, even when there is no logical dependency between them. This leads to several practical problems:

• Slow builds — steps that could run concurrently (like installing unrelated packages and copying separate source directories) block each other unnecessarily.
• Cache invalidation pain — changing a single line near the top of the Dockerfile invalidates every subsequent layer, forcing a full rebuild even when downstream steps are unaffected.
• Secret sprawl — passing API keys, tokens, or credentials during build time forces developers to resort to risky workarounds like multi-stage copy tricks or --build-arg values that leak into image history.
• No build-time network isolation control — you cannot selectively grant network access to specific RUN instructions without exposing the entire build.
• Bulky final images — without fine-grained control over what gets exported, artifacts like intermediate layers, build caches, and unnecessary files bloat production images.

BuildKit addresses each of these issues systematically. It is not just a performance upgrade—it is a fundamental redesign that enables safer, faster, and more expressive Docker builds.

Enabling and Configuring BuildKit

Enabling BuildKit on Docker Engine

On Docker 23.0 and later, BuildKit is the default builder. You can verify this by running:

docker buildx version

If you are on an older Docker version (18.09+), you can enable BuildKit via two methods:

Method 1: Set the environment variable before every build command:

DOCKER_BUILDKIT=1 docker build -t myapp:latest .

Method 2: Permanently enable it in the Docker daemon configuration file /etc/docker/daemon.json:

{
  "features": {
    "buildkit": true
  }
}

After editing the daemon configuration, restart Docker:

sudo systemctl restart docker

To confirm BuildKit is active, look for the BuildKit version string in the build output. The legacy builder does not display this information.

Using Docker Buildx (Recommended)

The modern, cross-platform way to invoke BuildKit is through docker buildx, which provides additional features like multi-platform builds and builder instance management:

docker buildx build -t myapp:latest .

Create a dedicated builder instance for advanced features like cross-compilation:

docker buildx create --name mybuilder --use
docker buildx inspect --bootstrap

The --bootstrap flag ensures the builder is fully initialized before you start using it.

Key BuildKit Features with Practical Examples

1. Parallel Build Execution

BuildKit automatically parallelizes independent Dockerfile stages. Consider a multi-stage Dockerfile where two stages copy different source directories and install different dependencies. BuildKit detects the lack of dependencies between these stages and runs them concurrently, dramatically reducing total build time on multi-core machines.

# Dockerfile — BuildKit runs stage A and stage B in parallel
FROM node:18 AS builder-a
WORKDIR /app/admin
COPY admin-package.json package.json
RUN npm install

FROM node:18 AS builder-b
WORKDIR /app/public
COPY public-package.json package.json
RUN npm install

# Final stage depends on both, so it waits for them
FROM nginx:alpine
COPY --from=builder-a /app/admin/dist /usr/share/nginx/html/admin
COPY --from=builder-b /app/public/dist /usr/share/nginx/html/public

In the legacy builder, these stages would execute one after the other. BuildKit runs them simultaneously, potentially cutting build time in half.

2. Secrets Management with --secret

One of the most impactful BuildKit features is the ability to mount secrets into the build environment without baking them into the image layers. Secrets are mounted as temporary files that exist only during the execution of the RUN instruction that references them.

Example: Using an npm auth token without leaking it:

# Dockerfile
FROM node:18-alpine
RUN --mount=type=secret,id=npm_token,dst=/root/.npmrc,target=/root/.npmrc \
    npm install --production

Build the image with the secret passed at build time:

docker build --secret id=npm_token,src=.npmrc -t myapp:secure .

Alternatively, pass the secret from an environment variable without writing it to disk:

export NPM_TOKEN="npm_abc123def456"
docker build --secret id=npm_token,env=NPM_TOKEN -t myapp:secure .

Inside the RUN instruction, the secret file appears at the mount point (/root/.npmrc in this example). Once the RUN command completes, the secret is unmounted and never appears in any image layer. This is a massive security improvement over --build-arg, which persists values in the build history and layer metadata.

Common Pitfall: The dst and target mount options behave differently. Use dst to specify the exact path where the secret file should appear inside the container. The target option is an alias and works identically in most contexts—just pick one convention and stick with it to avoid confusion.

3. SSH Agent Forwarding

BuildKit allows forwarding your SSH agent socket into build containers, enabling secure access to private Git repositories and other SSH-protected resources without embedding keys in the image.

# Dockerfile
FROM alpine:3.18
RUN --mount=type=ssh \
    ssh -o StrictHostKeyChecking=no git@github.com 2>&1 | tee /tmp/git-check

# Actual use case: clone a private repository
RUN --mount=type=ssh \
    git clone git@github.com:myorg/private-repo.git /app

Build the image while forwarding your SSH agent:

docker build --ssh default -t myapp:latest .

The default value tells BuildKit to use the SSH agent socket from your current environment (the one available via $SSH_AUTH_SOCK). You can also specify custom SSH configuration files or keys:

docker build --ssh default --ssh custom_key=/path/to/private/key -t myapp:latest .

Common Pitfall: The SSH mount is only available for the duration of the single RUN instruction that declares it. If you need SSH access across multiple steps, you must declare the mount on each one individually. Also, ensure your SSH agent is actually running and has the required keys loaded before the build—an empty agent socket causes mysterious authentication failures mid-build.

4. Cache Mounts for Package Managers

Cache mounts are one of the most practical BuildKit features for day-to-day development. They allow you to persist package manager caches across repeated builds without including the cache data in the final image. This drastically speeds up iterative builds by avoiding redundant downloads of packages, dependencies, and indices.

Example for apt (Debian/Ubuntu):

# Dockerfile
FROM ubuntu:22.04
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y \
    build-essential \
    curl \
    git

Example for apk (Alpine):

# Dockerfile
FROM alpine:3.18
RUN --mount=type=cache,target=/var/cache/apk \
    apk add --update \
    python3 \
    py3-pip

Example for pip (Python):

# Dockerfile
FROM python:3.11-slim
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --no-cache-dir \
    flask \
    gunicorn \
    numpy

Example for go modules:

# Dockerfile
FROM golang:1.21
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    go build -o /app/server ./cmd/server

Example for npm (Node.js):

# Dockerfile
FROM node:18-alpine
RUN --mount=type=cache,target=/root/.npm \
    npm install --production

The sharing=locked option ensures that concurrent builds using the same cache volume do not corrupt each other. Use sharing=shared for read-only access across builds, or sharing=private when each build needs an isolated cache. The default sharing mode is shared, which works well for most package manager scenarios.

Common Pitfall: Cache mounts persist between builds on the same machine, but they do NOT persist across different BuildKit instances or CI runner ephemeral environments. In CI pipelines, you need to configure remote cache storage (like an S3 bucket or a Docker registry) to benefit from cache mounts across pipeline runs. We cover remote caching later in this article.

5. Output Customization with --output

BuildKit decouples the build process from the output format. You can export build artifacts as a Docker image (the default), a tarball, a filesystem directory on your host, or even multiple outputs simultaneously.

Export the full image as a tarball:

docker build --output type=tar,dest=myimage.tar .

Export only specific files from a stage (bypassing the image format entirely):

# Dockerfile
FROM node:18 AS builder
WORKDIR /app
COPY package.json .
RUN npm install
COPY src/ src/
RUN npm run build

# No final FROM stage needed — BuildKit can export directly from builder

Then extract only the compiled output to your host filesystem:

docker build --target builder --output type=local,dest=./dist .

This pattern is incredibly useful for static site generators, compiled binaries, or any build where you want the artifact on your host without the overhead of creating and extracting a full container image.

Export an OCI image tarball for loading into another runtime:

docker build --output type=oci,dest=myapp-oci.tar .

Common Pitfall: When using --output type=local, BuildKit exports the entire root filesystem of the target stage. If your builder stage installs gigabytes of development dependencies, those will all appear in your host directory. To export only specific files, structure your Dockerfile so the target stage contains exactly what you want exported, or use a COPY instruction in a dedicated export stage that assembles only the desired artifacts.

6. Remote and Registry Caching

BuildKit supports exporting and importing layer caches to and from remote locations. This is essential for CI/CD pipelines where each build runs on a fresh ephemeral worker with no local cache.

Export cache to a container registry:

docker build \
  --cache-to type=registry,ref=myregistry.example.com/myapp:cache,mode=max \
  -t myapp:latest .

Import cache from a registry in subsequent builds:

docker build \
  --cache-from type=registry,ref=myregistry.example.com/myapp:cache \
  -t myapp:latest .

Use both cache import and export in a single CI step:

docker build \
  --cache-from type=registry,ref=myregistry.example.com/myapp:cache \
  --cache-to type=registry,ref=myregistry.example.com/myapp:cache,mode=max \
  -t myapp:latest .

The mode=max option exports cache entries for all layers, including intermediate ones. Without mode=max, only the final image layers are cached. For cache mounts (the --mount=type=cache feature), remote caching requires using a shared cache backend like an S3 bucket or a local network volume—registry caching does not cover cache mounts.

Cache to/from S3-compatible storage:

docker build \
  --cache-to type=s3,region=us-east-1,bucket=my-build-cache,name_prefix=myapp/ \
  --cache-from type=s3,region=us-east-1,bucket=my-build-cache,name_prefix=myapp/ \
  -t myapp:latest .

Common Pitfall: Registry cache is not a full image and cannot be pulled with docker pull. It is a separate manifest stored alongside the image tags. Also, cache entries are specific to the build context—changing the build context (e.g., different source files) invalidates the corresponding cache layers even if the Dockerfile instructions themselves haven't changed.

7. Heredoc Syntax for Multi-Line RUN Instructions

BuildKit supports heredoc syntax inside Dockerfiles, allowing you to write multi-line scripts cleanly without backslash chains or awkward concatenation:

# Dockerfile
FROM ubuntu:22.04
RUN <

FROM alpine:3.18
ARG TARGETARCH
RUN <

FROM nginx:alpine
COPY <
Generated at build time

Common Pitfall: Heredoc delimiters must not appear in the content body. If your script contains the delimiter word on its own line, the heredoc terminates prematurely. Use unique delimiters like EOT, SCRIPT_END, or a random string for safety. Also, heredocs in COPY instructions are a BuildKit-specific feature and will fail on the legacy builder.

8. Network Control per RUN Instruction

BuildKit allows you to selectively disable or enable network access for individual RUN instructions. This is valuable for security hardening and for ensuring reproducible builds that do not accidentally depend on external resources.

# Dockerfile
FROM python:3.11-slim

# This step needs network access to download packages
RUN --network=default pip install flask

# This step runs offline for security and determinism
RUN --network=none python -c "print('No network here')"

# Another step that needs network
RUN --network=default curl -s https://example.com/version.txt > /tmp/version

Common Pitfall: The --network=none flag blocks ALL network access, including DNS resolution and connections to localhost services. If a command expects to bind to a port for an internal check (like curl localhost:8080), it will fail even though the target is local. Use --network=default unless you explicitly need network isolation.

Best Practices for BuildKit Workflows

1. Structure Dockerfiles for Maximum Parallelism

Break your build into multiple independent stages rather than one long sequential chain. BuildKit can parallelize stages that do not depend on each other. Use named stages liberally to create clear dependency boundaries.

# Good: Independent stages that can run in parallel
FROM node:18 AS deps-install
COPY package.json package-lock.json ./
RUN npm ci

FROM node:18 AS unit-tests
COPY --from=deps-install node_modules node_modules/
COPY src/ src/
COPY test/ test/
RUN npm test

FROM node:18 AS build
COPY --from=deps-install node_modules node_modules/
COPY src/ src/
RUN npm run build

FROM node:18-slim AS production
COPY --from=build dist/ /app/dist/
COPY --from=deps-install node_modules/ /app/node_modules/
CMD ["node", "/app/dist/server.js"]

In this structure, unit-tests and build can execute simultaneously after deps-install completes. The legacy builder would run them sequentially regardless.

2. Use Cache Mounts Aggressively

Every package manager invocation in your Dockerfile should use a cache mount. The performance improvement on iterative builds is dramatic—often reducing package installation time from minutes to seconds. Combine multiple cache mounts when a tool uses multiple cache directories:

RUN --mount=type=cache,target=/root/.cache/pip \
    --mount=type=cache,target=/root/.cache/wheel \
    pip install -r requirements.txt

3. Never Pass Secrets via ARG or ENV

With BuildKit, there is zero excuse for leaking secrets into image layers. Always use --secret for credentials, tokens, and keys:

# Bad — secret baked into layer history forever
ARG GITHUB_TOKEN
RUN git clone https://token:${GITHUB_TOKEN}@github.com/myorg/repo.git

# Good — secret mounted only for the duration of the RUN
RUN --mount=type=secret,id=github_token,dst=/tmp/token \
    GITHUB_TOKEN=$(cat /tmp/token) \
    git clone https://token:${GITHUB_TOKEN}@github.com/myorg/repo.git

Even if you delete the secret file in the same RUN instruction, the layer history retains the ARG value. BuildKit secrets solve this fundamentally.

4. Leverage Heredocs for Readability

Multi-line scripts with heredocs are easier to read, maintain, and debug than inline one-liners or backslash-continued chains. They also survive syntax highlighting and code review better:

# Hard to read and maintain
RUN apt-get update && apt-get install -y \
    package1 \
    package2 \
    package3 \
    && rm -rf /var/lib/apt/lists/*

# Clean and maintainable with heredoc
RUN <


5. Implement Remote Caching Early in CI Pipelines

If you are using GitHub Actions, GitLab CI, Jenkins, or any CI system with ephemeral runners, invest time in setting up remote caching. The cost savings and speed improvements compound quickly. For GitHub Actions, combine docker buildx with the actions/cache backend or a registry cache:

# GitHub Actions workflow excerpt
- name: Build and push with cache
  run: |
    docker buildx build \
      --cache-from type=registry,ref=ghcr.io/${{ github.repository }}:build-cache \
      --cache-to type=registry,ref=ghcr.io/${{ github.repository }}:build-cache,mode=max \
      --push \
      -t ghcr.io/${{ github.repository }}:latest .

6. Use --output for Artifact Extraction

For static assets, compiled binaries, or any build output you need on the host, prefer --output type=local over docker cp from a temporary container. It is faster, more reliable, and avoids the overhead of creating intermediate containers:

# Extract compiled frontend assets directly to ./dist
docker build --target build-stage --output type=local,dest=./dist .

7. Pin BuildKit Versions for CI Determinism

While Docker's bundled BuildKit is generally up to date, you can use a specific BuildKit version via the moby/buildkit container image for reproducible CI environments:

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
  moby/buildkit:v0.12.0 \
  build --frontend=dockerfile.v0 --local context=. --output type=image,name=myapp

Common Pitfalls and How to Avoid Them

Pitfall 1: Assuming Legacy Builder Behavior

The most frequent source of confusion is writing Dockerfiles that depend on legacy builder quirks. For example, the legacy builder preserves file ownership from the build context, while BuildKit may apply different default permissions. Always explicitly set ownership and permissions with chown and chmod when they matter:

COPY --chown=1000:1000 --chmod=755 app.sh /usr/local/bin/app.sh

Pitfall 2: Forgetting That Cache Mounts Are Local

Cache mounts (--mount=type=cache) persist on the local machine where the build runs but are NOT included in remote registry caches. Developers often assume their CI builds will benefit from cache mounts automatically. In CI, cache mounts require either persistent build runners or a shared volume backend (like S3 or a network filesystem). Without these, each CI run starts with an empty cache.

Pitfall 3: Confusing --secret File Paths

The dst option on --mount=type=secret specifies the path inside the container where the secret file will appear. If your application expects the secret at a different path, you must either align the dst or explicitly copy/read the secret file within your RUN script. A common mistake is providing a dst path that the subsequent command never actually reads:

# Secret appears at /run/secrets/token, but npm config expects ~/.npmrc
RUN --mount=type=secret,id=npm_token,dst=/run/secrets/token \
    npm install  # npm never looks at /run/secrets/token!

Fix it by either pointing dst to the correct path or explicitly copying:

RUN --mount=type=secret,id=npm_token,dst=/root/.npmrc \
    npm install

Pitfall 4: Mismatched Cache Sharing Modes

Using sharing=locked on a cache mount that multiple parallel builds need to write to simultaneously can cause builds to serialize on the lock, defeating the purpose of parallelization. Use sharing=shared for read-heavy caches (like package indices) and reserve sharing=locked for caches where write conflicts could corrupt data (like compiled object caches). When in doubt, sharing=shared is the safer default.

Pitfall 5: Not Handling Heredoc Whitespace Properly

Heredocs capture whitespace literally. If your heredoc content has leading tabs or spaces that you intend as indentation, they become part of the script. Use the <<- syntax (with a hyphen) to strip leading tabs:

# With <<- (tab stripping) — tabs are removed
RUN <<-EOF
	echo "hello"   # This tab-indented line will have its leading tab stripped
	echo "world"
EOF

# Without the hyphen — spaces/tabs are preserved exactly
RUN <


Pitfall 6: SSH Mount Expiry Mid-Build

SSH agent forwarding via --mount=type=ssh is scoped to a single RUN instruction. If you need SSH access for multiple steps, you must repeat the mount declaration on every RUN that requires it. There is no persistent SSH session across RUN boundaries:

# Each RUN gets its own SSH mount
RUN --mount=type=ssh git clone git@github.com:org/repo1.git /app/repo1
RUN --mount=type=ssh git clone git@github.com:org/repo2.git /app/repo2
RUN --mount=type=ssh git clone git@github.com:org/repo3.git /app/repo3

Pitfall 7: Overlooking .dockerignore Impact on Cache

BuildKit cache invalidation is based on the checksums of files copied into the build context. If your .dockerignore file is too permissive, irrelevant file changes (like editor swap files or log outputs) can invalidate the cache unnecessarily. Conversely, an overly aggressive .dockerignore might exclude files that are genuinely needed, causing builds to fail or produce unexpected output. Regularly audit your .dockerignore to keep it precise:

# .dockerignore — exclude noise but include essential build context
*.swp
*.swo
*.log
.git/
node_modules/
dist/
.env
*.md
!important-config-file.md

Pitfall 8: Expecting BuildKit Features in Docker Compose Build

Docker Compose supports BuildKit, but the configuration syntax differs. Secrets and cache mounts must be declared in the Compose file, not passed via CLI flags:

# docker-compose.yml
services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
      secrets:
        - npm_token
secrets:
  npm_token:
    file: .npmrc

Cache mounts in Compose are configured via the cache_from and cache_to keys in the build section, which map to the equivalent BuildKit flags. Check your Compose version for exact syntax support.

Conclusion

Docker BuildKit represents a significant evolution in container image construction. Its graph-based execution model, secure secret management, cache mount system, and flexible output handling solve real-world problems that plagued the legacy builder for years. By adopting BuildKit—now the default in modern Docker—you gain faster builds, smaller and more secure images, and a build pipeline that integrates cleanly with CI/CD infrastructure. The key to success lies in restructuring Dockerfiles to exploit parallelism, using secrets and cache mounts idiomatically, configuring remote caching for ephemeral CI environments, and staying vigilant about the common pitfalls outlined above. With these practices in place, your Docker builds will become not just faster, but fundamentally more robust and maintainable.