Updated
Linux disk usage spike? Find what filled the disk in 3 steps
Short answer: When Linux disk usage suddenly spikes, run df -h and df -i to see whether space or inodes ran out, then run sudo du -xhd1 on the full filesystem to find the directory that grew. The usual causes are container or system logs, application data or backups, and deleted files that a process still holds open.
A database reports disk full, uploads fail, or a root filesystem approaches capacity. Before deleting anything, identify the affected filesystem and whether the shortage is storage space or inodes. These examples target Linux; paths and available utilities depend on your installation. Only the commands marked CHANGE (deletes files) under "App caches and crash dumps eating disk" delete anything; every other command in this guide is read-only.
1. Separate storage capacity from inode exhaustion
df -h
df -i
Record the mount point and observation time. Free bytes do not rule out inode exhaustion: many small files can exhaust a filesystem's available inodes. Correlate growth with deployments, imports, uploads and backup jobs.
2. Narrow the directory scan
sudo du -xhd1 /var
sudo du -xhd1 /home
These are example starting paths, not every possible data location. Use the path on the affected filesystem. On GNU du, -x avoids crossing into other filesystems and -d1 limits the displayed depth. Large scans can still create substantial I/O even though they do not delete files; start narrowly on busy servers. Permission errors can make the result incomplete.
3. Follow the evidence to the source
- Container storage: inspect
docker system dfanddocker info --format '{{.DockerRootDir}}'. These help orient the investigation, but do not assume one storage summary accounts for every log file. If evidence points to container logs, follow the Docker log rotation guide. - Application data: check the actual upload, backup and database directories. Do not delete database files or Docker volumes to make the alert disappear.
- System logs and caches:
journalctl --disk-usagehelps inspect journal storage on systemd systems. Look for a service repeatedly logging errors before deciding on retention changes. - Deleted but open files: when df and directory totals disagree, an authorized operator can inspect
sudo lsof +L1, if lsof is installed. A process may still hold a deleted file open. This is one possible explanation, not proof; identify the owner before planning a service action.
App caches and crash dumps eating disk
Suspect this category when du points at /var/lib/systemd/coredump, /var/crash or someone's ~/.cache, or when files roughly the size of a process's memory appear in an application directory. What these have in common: individual files are large, and they keep coming back with every crash or build. Find what keeps producing them before you clean up.
Read-only: crash dumps (core dumps)
cat /proc/sys/kernel/core_pattern
coredumpctl list --no-pager 2>/dev/null | tail -n 20
sudo du -sh /var/lib/systemd/coredump /var/crash 2>/dev/null
sudo find / -xdev -type f \( -name 'core' -o -name 'core.[0-9]*' \) -size +10M -exec ls -lh {} + 2>/dev/null
core_patterndecides where dumps go. If it starts with|…systemd-coredump, systemd-coredump stores them in/var/lib/systemd/coredumpandcoredumpctllists them. If it starts with|…apport(common on Ubuntu), crash reports land in/var/crash. If it is justcoreor similar, dumps are written to the crashing process's working directory; the lastfindlocates them.- The same program appearing again and again in
coredumpctl listmeans it keeps crashing and leaves a dump each time. Fix the crash first (check the app's logs, or exit code 137 and OOMKilled), or the disk will fill up again. - A file named
coreis not always a dump. Before removing one, runfile PATHand look forcore filein the output. - The total limit for systemd-coredump is set by
MaxUse=andKeepFree=; view them withsystemd-analyze cat-config systemd/coredump.conf.
Read-only: application caches
sudo du -sh /root/.cache /home/*/.cache 2>/dev/null | sort -h
du -sh ~/.cache/* 2>/dev/null | sort -h | tail -n 10
pip cache info 2>/dev/null
du -sh "$(npm config get cache 2>/dev/null)" 2>/dev/null
sudo du -sh /var/cache/apt /var/cache/dnf /var/cache/yum 2>/dev/null
docker system df
- Common heavy users of
~/.cache: pip, browsers downloaded by automation tools (ms-playwright,puppeteer), model files (huggingface) and the Go build cache (go-build). CI and build accounts can quietly collect tens of gigabytes. - In
docker system df, look at the Build Cache row; theRECLAIMABLEcolumn is what can be freed. - Caches can usually be downloaded or rebuilt again; the cost is slower installs and builds next time. Don't clear them while a build or install is running.
Read-only: Java and Node heap dumps
sudo find / -xdev -type f \( -name '*.hprof' -o -name '*.heapsnapshot' -o -name 'hs_err_pid*.log' -o -name 'report.*.json' \) -size +1M -exec ls -lh {} + 2>/dev/null
ps -eo args | grep -oE -- '-XX:[+]?HeapDump[A-Za-z]*(=[^ ]*)?|--heapsnapshot[^ ]*|--report-on-fatalerror|--diagnostic-dir[^ ]*' | sort | uniq -c
- With
-XX:+HeapDumpOnOutOfMemoryError, Java writes ajava_pid<PID>.hprofon every out-of-memory error. It is about as large as the heap and lands in the process's working directory or the path set by-XX:HeapDumpPath.hs_err_pid*.logfiles are JVM crash logs and are usually small. - Node writes
*.heapsnapshotfiles with--heapsnapshot-near-heap-limitor--heapsnapshot-signal, andreport.*.jsonwith--report-on-fatalerror, to the working directory or--diagnostic-dir. The second command shows which processes run with these flags; flags set through theNODE_OPTIONSenvironment variable won't show up. - For apps in containers, dumps are written inside the container's filesystem or a mounted volume. A container stuck in a restart loop can leave one dump per out-of-memory crash.
- A heap dump is a raw copy of process memory and can contain passwords, tokens and user data. Don't upload or share it, and don't send it to any AI tool. If you need to keep one for analysis, move it somewhere only you can read. To find other places secrets sit in plain text, see Plaintext secrets on your server or in your repo? A 3-step self-check.
CHANGE (deletes files): clean up only what you've confirmed is disposable
sudo coredumpctl dump --output=/mnt/OTHER_DISK/core.APP.keep PID
sudo rm /var/lib/systemd/coredump/core.APP_NAME.*
pip cache purge
npm cache clean --force
sudo apt-get clean
sudo dnf clean packages
docker builder prune --filter until=168h
sudo rm /path/to/app/java_pid12345.hprof
- The first line saves the one dump you still want to analyze (replace
PIDwith the process ID fromcoredumpctl list). Save it to another filesystem or machine, not back onto the full one. The second removes the rest of that program's dumps; replaceAPP_NAMEwith the program name in the file names, andlsthe pattern first to see what it matches. pip cache purgeandnpm cache clean --forceclear the cache of the account that runs them.apt-get clean(Debian/Ubuntu) anddnf clean packages(RHEL family) remove only downloaded package files, not installed software.docker builder prune --filter until=168hremoves build cache older than 7 days. It doesn't touch images, containers or volumes; the next build will just be slower.- There is no undo for deletion. Under
~/.cache, delete only directories of tools you recognize; neverrm -rf ~/.cachewholesale. - After cleaning, deal with the cause: fix the crash, point
HeapDumpPathat a dedicated directory and clean it regularly, or lowerMaxUse=incoredump.conf. Then recheckdf -has in step 4.
4. Review a change before freeing space
Keep the relevant error evidence. Confirm ownership and retention requirements before removing a cache or backup. Do not truncate Docker-managed logs, prune volumes or kill an unfamiliar process as a generic fix. For business data, arrange backups or snapshots on suitable storage before any approved archive or removal operation.
After you make a change, recheck capacity, inodes and the affected application's health. Watch whether growth resumes. An alert clearing does not establish that the underlying fault is resolved.
Illustrative example, not a customer incident
A root filesystem is nearly full while inode usage is normal. Directory checks point toward container storage, then a specific log shows rapidly repeated connection errors. Removing the log alone would not address the repeated errors. Investigate the connection failure and review log rotation. If inodes were exhausted instead, many small files would deserve attention rather than assuming the same log problem.
Ask AI for evidence, not automatic cleanup
Check first without changing anything: compare disk space, inodes and the likely growing directories. Distinguish container logs, application data, system caches, app caches and crash dumps, and deleted files still held open. State what is confirmed and what remains unknown. Do not delete files, truncate logs or restart services. Propose any change with its risk, verification and rollback steps first.
In OpsMate (itops.sh), use the terminal and AI workspace to examine the results. Review actual commands and permissions; a prompt alone is not a security boundary. Each time you ask the AI for help, recent terminal output is sent to the cloud AI along with your message, so remove sensitive information first.
Try OpsMate
500 free AI calls per month and unlimited servers. The desktop app keeps your SSH credentials on your own machine by default.
FAQ
Linux disk usage suddenly spiked. How do I find what filled the disk?
Run df -h and df -i to see whether space or inodes ran out, and note the mount point and time. Then run sudo du -xhd1 on the full filesystem to find the directory that grew; on GNU du, -x stays on one filesystem. Start narrowly on busy servers, because large scans create I/O. The usual causes are container or system logs, application data or backups, and deleted files a process still holds open.
df -h shows free space, so why is the disk still "full"?
The filesystem may have run out of inodes rather than bytes. df -h reports storage capacity and df -i reports inodes, and free bytes do not rule out inode exhaustion: many small files can use up a filesystem's available inodes. In that case, look for directories full of small files instead of assuming one large log is the problem.
df says the disk is full, but du doesn't find the files. Why?
One possible explanation is a deleted file that a process still holds open: the space isn't released, but no directory total shows it. If lsof is installed, an authorized operator can check with sudo lsof +L1. This is one possible explanation, not proof. Identify the process and its owner before planning any service action, and don't kill an unfamiliar process as a generic fix.
Container logs filled my disk. Can I just delete or truncate them?
Don't truncate Docker-managed logs, prune volumes or delete database files just to make the alert disappear. First check docker system df and docker info --format '{{.DockerRootDir}}', keep the relevant error evidence, and look for a service repeatedly logging errors. Removing the log alone doesn't fix those errors. If evidence points to container logs, set up log rotation as described in the Docker log rotation guide.
Need help interpreting the evidence?
OpsMate (itops.sh) puts your SSH terminal and AI on one page, and you review the evidence. Each time you ask the AI for help, recent terminal output is sent to the cloud AI along with your message, so remove sensitive information first.
Start free Desktop with local credentials