Fix 'Too many open files' in Linux in Production: Root Cause Analysis

Understanding the 'Too Many Open Files' Error

The error message Too many open files (often appearing as EMFILE or ENFILE in application logs) signals that a process—or the entire system—has reached its limit for simultaneously open file descriptors. This is not just about regular files; it covers network sockets, pipes, Unix domain sockets, inotify watches, and even eventfd handles. In production, this can cause silent failures, dropped connections, or cascading outages.

What Are File Descriptors?

On Linux, every I/O resource a process accesses is represented by an integer called a file descriptor (FD). When you open a file with open(), create a socket with socket(), or accept a connection with accept(), the kernel allocates a new FD. These descriptors are tracked per-process and also constrained by a system-wide limit. The two primary limits are:

• Per-process limit — ulimit -n (soft/hard RLIMIT_NOFILE)
• System-wide limit — /proc/sys/fs/file-max

When a process hits its per-process limit, operations like open() fail with errno EMFILE ("Too many open files"). When the whole system hits file-max, the kernel returns ENFILE ("Too many open files in system").

Why This Error Matters in Production

A "Too many open files" error is never benign in production. It can manifest as:

• Web servers rejecting new connections
• Database connection pools failing to acquire new sockets
• Logging frameworks crashing mid-write, losing critical telemetry
• Message queue consumers stalling
• Service mesh sidecars (like Envoy) dropping traffic

Because the failure is often partial—some FDs remain available until they're exhausted—the problem can go undetected by health checks until it's too late. Root cause analysis must distinguish between a simple misconfiguration (limits too low) and a genuine file descriptor leak (the application forgets to close descriptors).

Root Cause Analysis Methodology

When you see Too many open files in logs or monitoring, follow this systematic approach to pinpoint the exact cause before applying a fix.

Step 1: Identify the Affected Process

Start by locating the process that is throwing the error. If the error appears in application logs, the PID is often logged alongside the message. If not, use journalctl or dmesg to search for the error. For containerized workloads, identify the container ID and then the host PID.

# Find recent occurrences of "Too many open files" in system logs
journalctl -xe | grep -i "Too many open files"
dmesg | grep -i "Too many open files"

# For a specific container, get host PID (example with docker)
docker inspect --format '{{.State.Pid}}' container_name

Step 2: Check System-Wide and Per-Process Limits

Verify the current limits to understand whether the process is hitting a hard ceiling or a soft limit that can be raised temporarily.

# System-wide current usage vs maximum
cat /proc/sys/fs/file-nr          # (allocated, unused, max)
cat /proc/sys/fs/file-max         # system-wide hard limit

# Per-process limits for PID 12345
cat /proc/12345/limits            # look for "Max open files"
# Or use prlimit
prlimit --pid 12345 --nofile

The /proc/PID/limits output shows both Soft and Hard limits. The process can increase its own soft limit up to the hard limit using setrlimit(), but the hard limit is only raised by privileged operations (or during container creation with --ulimit).

Step 3: Inspect Open File Descriptors with lsof

To find out what the process is actually keeping open, list its file descriptors. This step reveals leaks—files or sockets that remain open long after they are needed.

# List all open FDs for PID 12345
lsof -p 12345

# Count total open FDs for the process
ls -1 /proc/12345/fd | wc -l

# Group by file type to spot unusual accumulation
lsof -p 12345 | awk '{print $5}' | sort | uniq -c | sort -rn

# Check socket-specific details (TCP/UDP state)
lsof -p 12345 -i TCP -s TCP:CLOSE-WAIT

Pay close attention to sockets in CLOSE_WAIT state—these indicate the remote side closed the connection, but the local process never called close(), leaking descriptors. Also look for a large number of anonymous inodes (pipes, eventfd) that may be created internally by libraries.

Step 4: Trace File Descriptor Usage with strace

When static inspection isn't enough, attach a tracer to see which system calls are creating FDs and whether corresponding close() calls are missing. This is invasive (adds overhead), so use it briefly in production during a controlled maintenance window or on a replicated canary.

# Trace only file-descriptor-related syscalls for PID 12345 for 30 seconds
strace -p 12345 -e trace=open,openat,creat,socket,accept,close,dup,dup2,dup3 -o /tmp/fd_trace.log &
sleep 30
kill %1

# Summarize open vs close calls
grep -E 'open(at)?\(|socket\(|accept\(' /tmp/fd_trace.log | wc -l
grep 'close(' /tmp/fd_trace.log | wc -l

A significant mismatch between open/accept and close counts suggests a leak. Look for error paths in the code where a descriptor is opened but not closed on exception.

Step 5: Analyze Logs and Application Behavior

Correlate the descriptor usage pattern with application metrics. Many frameworks expose an FD gauge (e.g., Prometheus process_open_fds). If the number of open FDs grows monotonically over time and never decreases under load, you have a leak. If it spikes and then recovers, the limits might simply be too low for legitimate workload peaks.

Fixing the Issue: Immediate and Long-Term Solutions

Immediate Mitigation

In an emergency, raise the process's soft limit to buy time while the root cause is being fixed. This does not solve a leak but prevents immediate crashes.

# Temporarily increase the limit for a running process (requires root/capability)
prlimit --pid 12345 --nofile=65535:65535

# For a systemd service, adjust the limit and reload
systemctl edit myservice.service --full
# Add: LimitNOFILE=65536
systemctl daemon-reload
systemctl restart myservice

Increasing the Limits Permanently

If the workload genuinely needs more descriptors, set higher limits system-wide and per-process through configuration files.

# 1. System-wide limit (persists across reboots)
echo "fs.file-max = 2000000" >> /etc/sysctl.d/99-file-max.conf
sysctl --system

# 2. Per-user/group limits in /etc/security/limits.conf
# Add lines like:
# *                soft    nofile          65536
# *                hard    nofile          65536
# Or for a specific user:
# appuser          soft    nofile          65536
# appuser          hard    nofile          65536

# 3. For containers, set ulimit at container runtime
docker run --ulimit nofile=65536:65536 myimage

Fixing File Descriptor Leaks in Code

The permanent fix is to eliminate leaks. Common pitfalls and their remedies:

• Not closing in error paths — Use RAII (in C++), try-with-resources (Java), or defer/contextlib.closing (Python/Go). Always close FDs in finally blocks.
• Forgetting to close HTTP response bodies — In Go, always call resp.Body.Close(); in Python requests, use a context manager.
• Leaking sockets in CLOSE_WAIT — Detect with lsof and add proper shutdown/close logic for connection errors.
• Infinite accumulation of inotify watches — Audit inotify_add_watch calls and remove watches on exit.
• Forked processes inheriting FDs — Mark FDs as O_CLOEXEC on open, or close unneeded descriptors after fork().

Here's a Python example of a leak and its fix:

# Leaky version – file handle never closed
def read_config_leaky(path):
    f = open(path, 'r')
    return f.read()   # f is leaked when function returns

# Fixed version using context manager
def read_config_fixed(path):
    with open(path, 'r') as f:
        return f.read()

In Go, always drain and close the response body to avoid leaking TCP connections:

// Leaky – body not closed
resp, err := http.Get("http://example.com")
if err != nil { return err }
// ... read some data, but never close

// Fixed
resp, err := http.Get("http://example.com")
if err != nil { return err }
defer resp.Body.Close()
// Optionally drain to allow connection reuse
io.Copy(ioutil.Discard, resp.Body)

Best Practices to Prevent 'Too Many Open Files'

• Set limits proactively — Don't wait for crashes. Use infrastructure-as-code to define LimitNOFILE in systemd units, Kubernetes security contexts, and container runtimes.
• Monitor FD usage — Scrape process_open_fds from Node Exporter or your application's metrics endpoint. Alert when usage exceeds 80% of the soft limit.
• Test for leaks in CI — Run soak tests with FD profiling. Tools like goleak (Go) or file descriptor sniffers can detect unclosed resources after tests.
• Use lsof in health checks — A simple script can detect CLOSE_WAIT sockets and warn before the limit is reached.
• Code review for resource management — Enforce patterns like RAII, defer, and explicit close in error paths during code review.
• Understand library behavior — Some libraries (like Node.js fs.watch) consume FDs per watched file; know their cost and set limits accordingly.

Conclusion

The "Too many open files" error is a classic production pitfall that stems either from inadequate limits or from resource leaks. By following a disciplined root cause analysis—checking limits, inspecting open descriptors with lsof, tracing system calls with strace, and correlating with metrics—you can quickly distinguish between a configuration issue and a software defect. Immediate mitigation via limit increases buys time, but the permanent solution lies in fixing leaks, adjusting limits to realistic values, and embedding preventive monitoring. With these practices, you transform a cryptic kernel error into a manageable, well-understood operational signal.