← Back to DevBytes

Fix 'Disk quota exceeded' in Linux in Production: Root Cause Analysis

Understanding "Disk Quota Exceeded" in Linux

When a Linux system returns the error Disk quota exceeded (typically accompanied by errno EDQUOT), it means a user or group has reached the maximum number of files (inodes) or total storage blocks allowed by the filesystem quota mechanism. This is distinct from a full disk — the filesystem itself may have ample free space, but a particular user, group, or project has hit its enforced limit.

In production environments, this error manifests silently in application logs, causes write failures for databases and web servers, and can trigger cascading outages when critical daemons suddenly cannot create new files, write to log files, or accept uploaded content. Root cause analysis requires inspecting quota limits, current usage, and the specific resource type (blocks vs. inodes) that has been exhausted.

Why Quota Exceeded Matters in Production

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

A production outage caused by quota limits is particularly dangerous because traditional disk-space monitoring often misses it. Teams monitoring df -h see healthy free space and assume storage is fine. Meanwhile, the application fails with cryptic "No space left on device" or "Disk quota exceeded" errors. Common production scenarios include:

Understanding the root cause means differentiating between block quota exhaustion and inode quota exhaustion — each requires a different remediation path.

Step 1: Confirm Quota Is Active and Identify the Filesystem

First, determine whether quota is enabled on the affected filesystem. The error typically appears in application logs or when manually testing writes:

# Reproduce the error as the affected user
touch /home/user123/testfile
# Output: touch: cannot touch '/home/user123/testfile': Disk quota exceeded

Check which filesystem the path resides on and whether quota accounting is active:

# Find the mount point
df -h /home/user123

# Check mount options for quota flags
mount | grep -E 'usrquota|grpquota|prjquota'
# Example output:
# /dev/sdb1 on /home type xfs (rw,relatime,attr2,inode64,prjquota)

The presence of usrquota (user quota), grpquota (group quota), or prjquota (project quota) in mount options confirms quota enforcement. On XFS, quota information is stored internally in the filesystem metadata; on ext4, it lives in aquota.user and aquota.group files at the filesystem root.

Step 2: Inspect Current Quota Usage and Limits

Use quota and repquota to retrieve per-user and per-group statistics. This reveals both soft limits (warning threshold with grace period) and hard limits (absolute ceiling).

# Show quota for a specific user
quota -v user123

# Example output:
# Disk quotas for user user123 (uid 1001):
# Filesystem   blocks   quota   limit   grace   files   quota   limit   grace
# /dev/sdb1    1024000*  1000000 1000000    none   15000*  10000   10000    none
#                                      ^ hard limit                    ^ hard limit

The asterisk * next to blocks or files indicates the hard limit has been reached. In this example, both block usage (1,024,000 blocks = ~1 GB at 1K block size) and inode count (15,000 files) exceed their respective limits.

For a broader view across all users, use repquota:

# Generate full quota report (XFS)
repquota -s /home

# For ext4 with quota files
repquota -s /home

The -s flag shows human-readable sizes. Look for users where used equals or exceeds hard limit. Pay special attention to inode limits — a user with thousands of small cache files, session files, or log files may exhaust the file count limit long before hitting the block limit.

Step 3: Distinguish Block Quota vs. Inode Quota Exhaustion

This is the critical fork in root cause analysis. The remediation differs completely:

Block Quota Exhaustion (Disk Space Per User)

The user has consumed their allocated disk space in bytes/blocks. Check what's consuming space:

# As root, check the user's home directory size
du -sh /home/user123

# Find large files and directories owned by the user
find /home/user123 -type f -size +100M -exec ls -lh {} \; 2>/dev/null

# Use ncdu for interactive exploration (install if needed)
ncdu /home/user123

Common culprits include unrotated log files, core dumps, large datasets, or forgotten backup archives.

Inode Quota Exhaustion (Number of Files)

The user has hit the limit on the number of files they can create, even if total bytes are within limits. This is common with applications that generate many small files:

# Count files owned by the user across the filesystem
find /home -user user123 -type f 2>/dev/null | wc -l

# Find directories with excessive file counts
find /home/user123 -type d -exec sh -c 'echo "$(find "$1" -maxdepth 1 -type f | wc -l) $1"' _ {} \; | sort -rn | head -20

Typical inode hogs include PHP session files, cache metadata, npm/node_modules trees, Git objects, thumbnail caches, or unmaintained cron job output files. The find command above reveals directories with thousands of files.

Step 4: Immediate Remediation — Buy Time

In a production outage, the priority is restoring service. You have several emergency options:

Option A: Temporarily Raise the Hard Limit

# Raise the user's block hard limit (in 1K blocks) — 2GB example
setquota -u user123 1500000 2000000 0 0 /home

# Raise inode hard limit — 20,000 files example
setquota -u user123 0 0 15000 20000 /home

# Verify the change took effect
quota -v user123

The four numeric arguments to setquota are: soft-block-limit, hard-block-limit, soft-inode-limit, hard-inode-limit. Setting soft limits to 0 leaves them unchanged.

Option B: Clear User Files That Are Safe to Delete

# Delete obvious junk: coredumps, cache, temporary files
find /home/user123 -name "*.core" -delete
find /home/user123 -name "*.tmp" -mtime +1 -delete
find /home/user123 -path "*/cache/*" -type f -mtime +7 -delete

# Clear package manager caches (e.g., pip, npm, composer)
rm -rf /home/user123/.cache/pip
rm -rf /home/user123/.npm/_cacache

Option C: Migrate Data to Another Filesystem Without Quotas

# Move large directories to a non-quota partition (e.g., /data)
mkdir -p /data/user123-backup
rsync -av --remove-source-files /home/user123/large_dataset/ /data/user123-backup/large_dataset/
# Then symlink or update application config to point to /data

Option D: Disable Quota Enforcement Temporarily (Last Resort)

# Soft-disable quotas without unmounting (XFS)
xfs_quota -x -c "disable -u" /home    # disables user quota enforcement
xfs_quota -x -c "disable -g" /home    # disables group quota enforcement

# Re-enable later with:
xfs_quota -x -c "enable -u" /home

# For ext4, you must remount without quota options:
mount -o remount,^usrquota,^grpquota /home

Disabling quotas is a nuclear option — only use it when the quota mechanism itself is misconfigured and you need immediate relief. Always re-enable after the incident.

Step 5: Root Cause Analysis — Why Did Usage Spike?

After restoring service, investigate why usage exceeded the limit. This prevents recurrence. The approach depends on whether blocks or inodes were exhausted.

Investigating Block Usage Spikes

Look for recent large files or rapid growth patterns:

# Find files created in the last hour (adjust timeframe as needed)
find /home/user123 -type f -mmin -60 -exec ls -lh {} \; | sort -k5 -hr | head -20

# Check for files that grew rapidly using a before/after snapshot
# If you have a baseline from yesterday:
find /home/user123 -type f -newer /var/tmp/baseline_marker -exec du -sh {} \;

Common patterns: a cron job writing verbose output without log rotation; a debug mode accidentally left enabled; a backup process writing to the wrong directory; or a data pipeline producing unexpected output.

Investigating Inode Usage Spikes

Inode exhaustion is almost always a runaway process generating many small files. Identify the source:

# Find directories with the most files (modify depth as needed)
find /home/user123 -maxdepth 3 -type d -print0 | while IFS= read -r -d '' dir; do
  count=$(find "$dir" -maxdepth 1 -type f 2>/dev/null | wc -l)
  [ "$count" -gt 500 ] && echo "$count $dir"
done | sort -rn

# Check file creation rate over time — snapshot inode counts
# Take two snapshots 10 minutes apart:
find /home/user123 -type f 2>/dev/null | wc -l   # snapshot 1
sleep 600
find /home/user123 -type f 2>/dev/null | wc -l   # snapshot 2
# If difference is large, a process is creating files rapidly

Frequent inode culprits include:

Step 6: Permanent Fix — Adjust Limits and Monitoring

Once the root cause is understood, implement a permanent solution:

Set Appropriate Quota Limits

# Set realistic limits based on historical usage patterns
# Block limit: 2x peak observed usage
setquota -u user123 2000000 3000000 0 0 /home

# Inode limit: 2x peak file count
setquota -u user123 0 0 20000 30000 /home

# Make limits persistent by editing /etc/limits or using setquota in a startup script
# For XFS, quotas survive reboots when enabled in /etc/fstab:
# /dev/sdb1  /home  xfs  defaults,uquota,pquota  0  0

Implement Quota Monitoring

Standard disk monitoring misses quota issues. Add quota-specific checks:

# Cron job to alert on users nearing quota limits (example script)
#!/bin/bash
THRESHOLD_PERCENT=80
repquota -s /home | awk 'NR>5 && $3 != "0" && $4 != "0" {
  used = $2; limit = $4;
  gsub(/[KMG]/, "", used); gsub(/[KMG]/, "", limit);
  if (limit > 0 && (used/limit)*100 > '"$THRESHOLD_PERCENT"') 
    print "WARNING: User " $1 " at " int((used/limit)*100) "% of quota"
}' | mail -s "Quota Alert" admin@example.com

For production-grade monitoring, integrate quota checks into Prometheus, Nagios, or Datadog using custom check scripts that parse repquota output.

Add Filesystem-Level Inode Monitoring

# Monitor overall filesystem inode usage (separate from quotas)
df -i /home
# Filesystem     Inodes   IUsed   IFree IUse% Mounted on
# /dev/sdb1      5.2M     1.1M    4.1M   22% /home

# Alert if IUse% exceeds 90%
df -i /home | awk 'NR==2 {if(int($5)>90) print "CRITICAL: Inode usage >90%"}'

Best Practices for Production Quota Management

Example: XFS Project Quota for a Shared Application Directory

Project quotas are powerful for isolating a specific directory tree. Here's how to set up and diagnose project quota exhaustion for /app/uploads:

# Define a project ID and assign it to the directory
echo "100:uploads" >> /etc/projects
echo "uploads:100" >> /etc/projid
echo "/app/uploads:100" >> /etc/projects

# Apply the project ID to the directory
xfs_quota -x -c "project -s uploads" /app

# Set project quota limits (soft 5GB, hard 10GB)
xfs_quota -x -c "limit -p bsoft=5g bhard=10g uploads" /app

# Check project quota usage
xfs_quota -x -c "report -p -h" /app

When project quota is exceeded, any write to /app/uploads fails with "Disk quota exceeded," regardless of which user writes. Diagnose with:

# Check project quota status
xfs_quota -x -c "quota -p -v uploads" /app

# Find what's consuming space in the project directory
du -sh /app/uploads
find /app/uploads -type f -size +100M -exec ls -lh {} \;

Conclusion

"Disk quota exceeded" errors in production are a storage management failure that standard disk-space monitoring cannot catch. Effective root cause analysis requires methodically checking whether the exhaustion is block-based or inode-based, identifying the specific user or project affected, and tracing the spike in usage back to its source — whether it's a runaway process, missing log rotation, or an application generating excessive small files. The immediate fix involves raising limits, clearing junk, or migrating data, but the permanent solution demands setting realistic limits, implementing quota-aware monitoring, and establishing operational practices that prevent silent exhaustion. By treating quota limits as a first-class monitoring target alongside disk space and inode utilization, you close a critical observability gap that otherwise leads to hard-to-diagnose production outages.

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles