Tuning the dmesg Ring Buffer on a VPS: Why Boot Logs Vanish

Run dmesg | head -5 on a VPS that has been up for three weeks and the timestamps will read a few days ago, not boot. The kernel log ring buffer is a fixed circular allocation, and on a chatty host with virtio, br_netfilter and conntrack messages it wraps quickly. The classic symptom is dmesg | grep -i 'Out of memory' returning nothing because the OOM record scrolled out hours after the incident. sysctl kernel.dmesg_restrict and cat /proc/sys/kernel/printk control access and console verbosity, but the buffer size itself lives in the kernel configuration and, on modern kernels, in a boot parameter.

This article covers how to size the buffer, how to persist it across reboots on a VPS, and how to ship the records elsewhere so that the kernel never becomes your only forensic source again. It is aimed at operators running 1–8 vCPU instances where memory is real and a 16 MiB log buffer is not free.

Reading the Current Ring Buffer Size

The authoritative value is LOG_BUF_LEN, compiled into the kernel and reported at boot. Check it with dmesg | grep -i 'log_buf_len' or, more reliably, by inspecting the kernel command line:

cat /proc/cmdline
dmesg --kernel --level=info | grep -i 'Memory:.*available\|log_buf_len'
zcat /proc/config.gz 2>/dev/null | grep CONFIG_LOG_BUF_SHIFT
# fallback on non-rt kernels:
sysctl kernel.printk

A CONFIG_LOG_BUF_SHIFT of 18 means 2^18 = 262144 bytes, roughly 256 KiB. Distributions such as Debian and Ubuntu ship 17 or 18, which is often under an hour of runtime on a host with multiple cloud-init and virtio_net messages per second. If /proc/config.gz is absent, the boot-time value is still visible in the early dmesg output as log_buf_len: ... bytes when the log_buf_len= parameter was supplied.

Setting log_buf_len on the Kernel Command Line

The buffer can be enlarged at boot without recompiling, via the log_buf_len=N parameter. The value must be a power of two and is rounded up by the kernel. On GRUB-based systems, edit the default entry in /etc/default/grub:

# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="consoleblank=0 log_buf_len=4M"

update-grub            # Debian/Ubuntu
# grub2-mkconfig -o /boot/grub2/grub.cfg   # RHEL family
reboot
dmesg | grep -i 'log_buf_len'   # confirm: should show 4194304

4 MiB is a sensible default for a 2–4 GiB VPS and typically buys 12–48 hours of runtime depending on message rate. On instances with heavy conntrack or a KVM host that passes through a lot of virtio interrupts, 8 MiB is a reasonable ceiling. Above 16 MiB, the trade-off is that the buffer is allocated from low kernel memory at boot and is not reclaimable.

Why Entries Disappear Even with a Larger Buffer

Space is only one reason records vanish. Message rate matters more. A single iptables -L rule set with logging, or a misconfigured conntrack table, can emit thousands of lines per minute. Benchmark your own rate with a quick count of lines per uptime hour:

boot=$(date -d "$(uptime -s)" +%s)
now=$(date +%s)
lines=$(dmesg | wc -l)
echo "lines/min: $(( lines * 60 / (now - boot) ))"

If the answer exceeds roughly 200 lines per minute, no buffer size will hold a week of history. The correct fix is to stop the noisy source: silence xt_LOG rules you no longer need, set kernel.printk_ratelimit and kernel.printk_ratelimit_burst to throttle bursts, and check whether nf_conntrack is logging table-full events repeatedly.

Rate limiting is not the same as dropping the incident you care about. Printk rate limiting applies per emitting site for WARN and INFO levels, so genuine OOM or ext4 errors still come through. Combine it with a console loglevel of 4 (sysctl -w kernel.printk="4 4 1 7") so that only warnings and above reach the buffer from debugging paths.

Persisting Kernel Logs to Disk and Remote

The ring buffer should never be the primary store. systemd-journald reads /dev/kmsg and writes a persistent journal, and rsyslog with kern.* can forward to a central collector. Enable persistent journal storage and cap its size so a VPS disk does not fill:

# /etc/systemd/journald.conf
[Journal]
Storage=persistent
SystemMaxUse=500M
SystemMaxFileSize=50M
RateLimitIntervalSec=30s
RateLimitBurst=2000

systemctl restart systemd-journald
journalctl -k -b -1 --no-pager | tail -20

The -b -1 flag lists messages from the previous boot, which is exactly the archive the ring buffer cannot provide after a reboot. For hosts that reboot unexpectedly, journalctl -k -b -1 -p err after recovery is the fastest path to the cause. If disk space is tight, allow the journal a 500 MiB ceiling and periodically verify with journalctl --disk-usage.

Kernel Panic and Oops Capture

A kernel panic reboots the machine and, unless configured otherwise, leaves no trace in the next boot. Two sysctls matter: kernel.panic sets the seconds to wait before reboot, and kernel.panic_on_oops decides whether an oops escalates. To capture a crash dump, either use kdump with a reserved crashkernel region or persist pstore to a backing device.

sysctl -w kernel.panic=10
sysctl -w kernel.panic_on_oops=1
sysctl -w kernel.panic_print=1
# verify pstore availability
mount | grep pstore
ls -l /sys/fs/pstore/

panic_print=1 prints the task list before reboot, which is invaluable when the console log is the only surviving evidence. On virtualised hardware, pstore is rarely available, so a serial console or netconsole feed to another host is the more reliable channel.

Container and Cloud-Init Noise

Two sources dominate message volume on a fresh VPS image. cloud-init logs each datasource probe at INFO level, and Docker or containerd emit network setup messages per container start. On an instance that runs containers with restart policies, a flapping health check can produce thousands of veth and iptables lines per hour and wrap the buffer on its own.

Cap cloud-init verbosity with log_verbosity: warn in /etc/cloud/cloud.cfg.d/05_logging.cfg, and keep its output journal-routed rather than console-routed. For container noise, retain the messages that matter (OOM, link down) but let the container runtime’s log driver absorb per-start chatter instead of writing it to kmsg.

Verifying the Change Under Load

After a reboot, confirm the new buffer size and measure how long it takes to wrap under realistic conditions. A simple check is to note the oldest timestamp and watch the delta grow:

dmesg -T | head -1
dmesg -T | tail -1
# oldest-to-newest span in seconds
dmesg -T | awk 'NR==1{split($0,a," ");t0=$0} END{}' >/dev/null
journalctl -k --since "-1h" | wc -l

Record the span before and after the change. On a small instance the increase from 256 KiB to 4 MiB typically multiplies retention by 15–20x once noise has been trimmed. Keep an eye on /proc/meminfo Slab growth to confirm the larger buffer is not causing pressure alongside other kernel allocations.

Sizing the buffer is only one part of keeping a small instance observable; the memory cost of the change matters too. If you are budgeting RAM across a fleet, see our cloud VPS benefits breakdown for how fixed kernel overhead compares with workload headroom. And when logs need to outlive the host itself, forwarding to a collector is cheaper than growing the buffer indefinitely—compare the managed vs unmanaged options if you would rather not operate that pipeline yourself.