Fix Kubernetes 'CrashLoopBackOff' Error

Understanding the CrashLoopBackOff Error

In Kubernetes, a CrashLoopBackOff status indicates that a container inside a pod is repeatedly crashing after being restarted. The kubelet detects that the container has exited with a non-zero exit code, attempts to restart it, and when the crash pattern persists, the pod enters this state. The error message you see when running kubectl get pods looks like this:

NAME                     READY   STATUS             RESTARTS   AGE
my-app-7f8b9c6d4-xq5k2   0/1     CrashLoopBackOff   6          10m

This is not an immediate failure—it's a backoff mechanism. Kubernetes waits progressively longer between restart attempts (10s, 20s, 40s, up to 5 minutes) to avoid overwhelming the system with a perpetually failing container. The status is a symptom, not a root cause, and resolving it requires systematic investigation of the underlying issue.

Why the CrashLoopBackOff Error Matters

Ignoring this error leads to cascading problems in production environments. A pod stuck in CrashLoopBackOff cannot serve traffic, meaning your application is effectively down for that replica. If all replicas enter this state, you face a complete service outage. Beyond immediate downtime, the error can mask deeper configuration problems, resource constraints, or image issues that affect the entire deployment pipeline. Understanding how to diagnose and fix it is a core skill for any developer working with Kubernetes.

Common Root Causes

Before diving into debugging commands, it helps to know the most frequent triggers:

• Application bugs: unhandled exceptions, segmentation faults, or runtime errors that cause the process to exit
• Missing environment variables: the app expects a database URL or secret but receives nothing
• Incorrect entrypoint or command: the container starts and immediately finishes because the main process is misconfigured
• Dependency failures: a required service (database, API endpoint) is unreachable and the app terminates
• Resource limits too tight: OOMKilled (out-of-memory kill) when memory limits are set below what the app needs
• Volume mount issues: missing ConfigMaps, Secrets, or persistent volumes the container expects
• Liveness probe misconfiguration: a probe that is too aggressive kills a healthy container that is still starting up

Step-by-Step Diagnosis Workflow

1. Inspect Pod Logs

The first and most revealing step is to check the container logs. Even a crashed container retains its logs from the previous run. Use kubectl logs with the --previous flag to see the output from the most recently terminated instance:

kubectl logs my-app-7f8b9c6d4-xq5k2 --previous

If the pod has multiple containers, specify the container name with -c:

kubectl logs my-app-7f8b9c6d4-xq5k2 -c app-container --previous

Look for stack traces, error messages, or indications that the process exited unexpectedly. A common log line you might see is:

Error: Unable to connect to database at db-service:5432
Application terminated with exit code 1

This immediately points to a dependency or network issue.

2. Examine Pod Events

Kubernetes records events for every significant action, including container restarts. Run kubectl describe pod to get a chronological view of what happened:

kubectl describe pod my-app-7f8b9c6d4-xq5k2

Scroll to the Events section at the bottom. You will see entries like:

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  10m                  default-scheduler  Successfully assigned default/my-app-7f8b9c6d4-xq5k2 to worker-node-1
  Normal   Pulled     9m45s                kubelet            Container image "my-app:v1.2.3" already present on machine
  Warning  BackOff    8m30s (x4 over 9m)   kubelet            Back-off restarting failed container
  Warning  Failed     30s (x5 over 9m30s)  kubelet            Error: container not found or exited with status 137

Exit status 137 indicates the container was killed by a signal (137 = 128 + 9, signal 9 is SIGKILL), which often points to an OOM kill. Exit status 1 typically means the application itself exited with an error. Exit status 0 means success—but if the container keeps restarting with exit 0, your entrypoint or command likely finishes immediately rather than running as a long-lived process.

3. Check Container Exit Codes and Restart Count

Use kubectl get pod with YAML or JSON output to see the container statuses in detail:

kubectl get pod my-app-7f8b9c6d4-xq5k2 -o jsonpath='{.status.containerStatuses}' | jq

This shows the restartCount, last state with exitCode, and any reason for termination. An example output:

[
  {
    "name": "app-container",
    "restartCount": 12,
    "lastState": {
      "terminated": {
        "exitCode": 137,
        "reason": "OOMKilled",
        "startedAt": "2025-01-15T10:23:45Z",
        "finishedAt": "2025-01-15T10:24:02Z"
      }
    },
    "ready": false,
    "started": false
  }
]

An OOMKilled reason tells you the container exceeded its memory limit and was forcibly terminated by the kernel. This is one of the most common and actionable signals you will encounter.

4. Validate the Entrypoint and Command

Sometimes the container runs a command that exits immediately. For example, if your Dockerfile has:

CMD ["/bin/sh", "-c", "echo hello"]

the container will print "hello" and exit with code 0, then Kubernetes restarts it, creating an infinite loop. To debug this, inspect the running container's command arguments:

kubectl get pod my-app-7f8b9c6d4-xq5k2 -o jsonpath='{.spec.containers[0].command}' && echo
kubectl get pod my-app-7f8b9c6d4-xq5k2 -o jsonpath='{.spec.containers[0].args}' && echo

If no command or args are specified, the image's ENTRYPOINT and CMD from the Dockerfile are used. Ensure they define a long-running process like a web server, not a script that terminates.

5. Reproduce Locally with the Same Image

Pull the exact image tag and run it locally with Docker to see if it crashes the same way:

docker run --rm my-app:v1.2.3

If it crashes locally, you can add debugging tools or inspect the filesystem. If it works locally but not in Kubernetes, the issue is likely environment-specific: missing environment variables, network policies blocking outbound connections, or resource constraints.

6. Check Environment Variables and Secrets

List the environment variables injected into the pod:

kubectl exec my-app-7f8b9c6d4-xq5k2 -- env 2>/dev/null || true

If the container is in CrashLoopBackOff, exec may fail. Instead, inspect the pod spec:

kubectl get pod my-app-7f8b9c6d4-xq5k2 -o jsonpath='{.spec.containers[0].env}' | jq

Verify that required variables like DATABASE_URL, API_KEY, or LOG_LEVEL are present and correctly sourced from Secrets or ConfigMaps. A missing Secret reference will prevent the pod from starting entirely, but a misnamed key inside a Secret will result in an empty value being passed to the container.

7. Review Resource Limits and Requests

If you see OOMKilled in the container status, check the resource configuration:

kubectl get pod my-app-7f8b9c6d4-xq5k2 -o jsonpath='{.spec.containers[0].resources}' | jq

Typical output:

{
  "limits": {
    "memory": "128Mi",
    "cpu": "500m"
  },
  "requests": {
    "memory": "64Mi",
    "cpu": "250m"
  }
}

If the application needs more than 128Mi of memory, the kernel will kill it. Increase the limit:

resources:
  limits:
    memory: "256Mi"
    cpu: "500m"
  requests:
    memory: "128Mi"
    cpu: "250m"

Apply the updated deployment and monitor whether the restarts stop.

8. Check Liveness Probe Configuration

A misconfigured liveness probe can kill a container that is still initializing. Inspect the probe settings:

kubectl get pod my-app-7f8b9c6d4-xq5k2 -o jsonpath='{.spec.containers[0].livenessProbe}' | jq

An overly aggressive probe might look like:

{
  "httpGet": {
    "path": "/healthz",
    "port": 8080
  },
  "initialDelaySeconds": 0,
  "periodSeconds": 5,
  "failureThreshold": 1
}

With zero initial delay and a failure threshold of 1, the probe starts checking immediately and kills the container on the first failure—before the application has a chance to bind to port 8080. A safer configuration adds generous delays:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

This gives the application 30 seconds to start and tolerates up to 3 consecutive probe failures before restarting the container.

Practical Fixes for Common Scenarios

Scenario A: OOMKilled Due to Tight Memory Limits

Symptoms: The container status shows reason: OOMKilled, exit code 137, and restart count keeps climbing.

Fix: Update the deployment's resource limits. Edit the deployment directly:

kubectl edit deployment my-app

Find the resources block and adjust the memory limit upward. Alternatively, patch it:

kubectl patch deployment my-app -p '{"spec":{"template":{"spec":{"containers":[{"name":"app-container","resources":{"limits":{"memory":"512Mi"}}}]}}}}'

Then watch the pod recover:

kubectl rollout status deployment my-app

Scenario B: Missing Environment Variable Causing Application Crash

Symptoms: Logs show Error: DATABASE_URL is not set or similar.

Fix: Add the missing variable to the deployment spec or verify that the Secret exists and is correctly referenced. Create the Secret if needed:

kubectl create secret generic db-secret --from-literal=DATABASE_URL=postgres://user:pass@db-host:5432/mydb

Then reference it in the deployment:

env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: db-secret
        key: DATABASE_URL

Apply the changes and verify:

kubectl apply -f deployment.yaml
kubectl get pods -l app=my-app

Scenario C: Entrypoint Exits Immediately

Symptoms: The container restarts every few seconds with exit code 0. Logs show the application printed something and then stopped.

Fix: Modify the Dockerfile or override the command in the deployment. To override, add a command field to the container spec that runs a long-lived process:

command: ["/bin/sh", "-c"]
args: ["while true; do echo 'Running...'; sleep 60; done"]

For production, ensure your application runs as a foreground process (e.g., a web server like nginx -g 'daemon off;' or a Node.js app with node server.js).

Scenario D: Liveness Probe Kills Healthy Container

Symptoms: The application starts successfully according to logs, but gets killed after a few seconds. Events show repeated probe failures followed by container kills.

Fix: Tune the probe parameters. Update the deployment with realistic delays:

kubectl patch deployment my-app -p '{"spec":{"template":{"spec":{"containers":[{"name":"app-container","livenessProbe":{"initialDelaySeconds":60,"periodSeconds":15,"failureThreshold":5}}]}}}}'

Consider adding a startup probe for slow-starting applications to give them dedicated initialization time before liveness checks begin:

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 0
  periodSeconds: 5
  failureThreshold: 30
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

The startup probe runs first and must succeed before the liveness probe takes over, giving the application up to 150 seconds (30 × 5s) to initialize.

Automated Alerting and Prevention

To catch CrashLoopBackOff errors before they cause outages, integrate monitoring tools. Use Prometheus queries to detect pods in this state:

kube_pod_status_reason{reason="CrashLoopBackOff"} > 0

Set up alerts that trigger when a pod remains in CrashLoopBackOff for more than a few minutes. Additionally, use admission controllers like OPA or Kyverno to enforce minimum resource limits, mandatory liveness probe delays, and required environment variable documentation at deployment time.

Best Practices to Prevent CrashLoopBackOff

• Always define resource requests and limits — Set memory limits with headroom based on observed usage under load. Use Vertical Pod Autoscaler in recommendation mode to gather data before enforcing tight limits.
• Use startup probes for slow-starting applications — Separate initialization concerns from ongoing health checks. A startup probe prevents premature liveness probe failures.
• Log gracefully on failure — Ensure your application logs the reason for exit before calling exit(1). Structured logging makes post-mortem analysis faster.
• Test images locally with the same entrypoint — Run docker run with the exact command and environment variables your Kubernetes pod uses. Catch exit-code-zero bugs before deployment.
• Validate environment variables at container start — Add a startup script or init container that checks required variables are present and non-empty, exiting with a clear message if something is missing.
• Set failureThreshold greater than 1 on liveness probes — A single transient failure should not trigger a restart. Allow at least 3 consecutive failures.
• Monitor restart counts — Track kube_pod_container_status_restarts_total in Prometheus and alert on rapid increases, even before the pod enters CrashLoopBackOff.

Conclusion

The CrashLoopBackOff error is Kubernetes' way of telling you that a container is unstable and needs attention. Rather than treating it as a mysterious failure, approach it methodically: check logs first, examine exit codes, verify resource limits, validate environment configuration, and scrutinize health probe timing. Each piece of diagnostic data narrows the search space until the root cause becomes clear. By combining systematic debugging with preventive practices—appropriate resource limits, startup probes, graceful error handling, and pre-deployment testing—you can eliminate the most common causes of crash loops and keep your deployments stable and reliable.