Keeping overlay2 Disk Usage Under Control Without Breaking Running Containers

Docker’s overlay2 driver keeps every image layer, every stopped container’s writable layer and every build cache on the same filesystem as /var/lib/docker. On a VPS with a 40 GB disk, a handful of Node builds can fill 30 GB in a week, and the first visible symptom is usually a container that dies at startup or a Postgres that refuses writes. The layout is what makes it confusing: docker system df reports 12 GB while du shows 22 GB, and neither number includes the 7 GB sitting in orphaned layers.

Aggressive cleanup is dangerous on a live host. docker system prune -a will happily remove an image whose running container has a different tag, and --volumes deletes named volumes that no container references — including the one your database is about to mount on restart. The approach below keeps running containers untouched: work out what is actually reclaimable, prune in narrow steps, then put retention policies in place so the problem does not recur. Disk headroom is the reason running containers on a dedicated KVM slice beats sharing an overcommitted host, since you can see and own the whole filesystem.

Reading the real disk number

Start with three different views. docker system df -v breaks usage down by image, container and build cache, and includes the shared layer column that explains most of the discrepancy. du -sh /var/lib/docker/overlay2 gives the on-disk truth including layers no longer referenced by any image. df -h /var/lib/docker tells you whether the block device itself is full, which is a different problem from a full layer store.

docker system df -v | head -40
du -sh /var/lib/docker/overlay2 /var/lib/docker/volumes
df -h /var/lib/docker
find /var/lib/docker/overlay2 -maxdepth 1 -type d -mtime +7 | wc -l

A common misconfiguration is Docker defaulting to the root filesystem when a larger data volume exists. Check docker info | grep 'Docker Root Dir'; if it points at / and you have a separate disk mounted elsewhere, moving the data root is the highest-value fix available, and it is a symlink plus a daemon restart, not a reinstall.

Safe prune sequence

Prune from the least to the most destructive, and check the reported reclaimable figure before confirming. Dangling images are unreferenced first: nothing points to them, so removing them cannot break a running container. Build cache is next and is nearly always safe, though it slows the next build. Stopped containers are third — those only matter if you plan to restart them.

• docker image prune — dangling images only, safe
• docker builder prune --filter until=168h — build cache older than a week
• docker container prune --filter until=72h — stopped containers aged out
• docker image prune -a — every image without a container; inspect first
• docker volume prune — never without docker volume ls review

Why running containers survive, and which do not

A running container holds a reference to its image, so docker image prune -a will not remove it. It will remove the image a stopped container depends on, because a stopped container is not a reference in the same sense, and docker start will then fail with “no such image”. Check docker ps -a --filter status=exited before any -a prune; if the list is non-empty, decide per container rather than in bulk.

Build cache is the safest large win on a development host and the least safe on a build server, because losing it turns a 40-second incremental build into a 12-minute cold one. On a 4 GB VPS the cache is also the largest single consumer after images. Keep it if builds run more than a few times per day; drop it aggressively if this host only runs production containers.

docker builder prune --filter until=168h --keep-storage 5GB
docker container prune --filter until=72h
docker image prune -a --filter until=336h

Volumes are not images and need their own policy

Named volumes live under /var/lib/docker/volumes and are never touched by image prunes. They are also the usual source of a 15 GB database in a container you thought was disposable. Treat them as durable storage: back them up, then decide on retention. docker volume prune only removes volumes not referenced by any container, running or stopped, which means a deleted-but-not-yet-recreated container’s data looks reclaimable.

Command	Touches running containers	Touches volumes
docker image prune	No	No
docker image prune -a	No	No
docker system prune	No	No
docker system prune -a –volumes	No, but breaks restart	Yes
rm -rf /var/lib/docker/*	Destroys everything	Yes

Controlling size before it accumulates

Two daemon settings do most of the work. data-root moves the store to a dedicated disk, and the overlay2.size storage-opt caps each container’s writable layer on XFS with project quotas. On ext4 that option is unavailable, so if per-container limits matter, format the data disk as XFS with pquota mounted. Without a cap, one runaway log file inside a container can consume the whole disk.

# /etc/docker/daemon.json
{
  "data-root": "/srv/docker",
  "storage-driver": "overlay2",
  "storage-opts": ["overlay2.size=20G"],
  "log-driver": "json-file",
  "log-opts": {"max-size": "20m", "max-file": "3"}
}

Log rotation is the other half of the equation. The default json-file driver grows without limit, and container logs regularly reach multiple GB on a quiet service after a crash loop. Setting max-size at 20 MB with three files caps each container’s log at 60 MB. Existing containers keep their old settings until recreated, so a docker compose up -d --force-recreate is required for the change to take effect.

Scheduling cleanup with a hard floor

A weekly cron job is enough if it is guarded by a disk threshold. Prune only when usage exceeds 75 percent, so a normally lean host never loses build cache. Run it against the narrow commands, log the reclaimed figure, and alert if the host is still above 90 percent after the run — that indicates a volume or a base image that needs manual attention.

#!/bin/sh
use=$(df --output=pcent /var/lib/docker | tail -1 | tr -dc 0-9)
[ "$use" -lt 75 ] && exit 0
docker builder prune -f --filter until=168h
docker image prune -f --filter until=336h
docker container prune -f --filter until=72h
df -h /var/lib/docker

What to check after a prune

Confirm every container is still running, then verify the ones with mounts can still reach their data. docker ps --format '{{.Names}} {{.Status}}' should show no “Restarting” states in the minutes after the job. A container that goes into a restart loop post-prune almost always lost its image or its volume, and both are recoverable by re-pulling or restoring, not by rebooting the VPS.

Add docker system events --filter type=image --since 1h to your troubleshooting notes; it shows exactly which layers were removed and when. With a data root on its own disk, per-container size caps, log rotation and a thresholded weekly prune, overlay2 usage on a KVM-based VPS stays bounded and predictable instead of drifting upward until something breaks.