Analyzing systemd-coredump Crash Dumps on a VPS: A Practical Debugging Workflow

When a service dies on a small VPS you usually get “status=11/SEGV” and nothing else. The crash dump that would tell you exactly which instruction faulted and on what address is almost always already captured by systemd-coredump — it is just never opened. This walkthrough sets up core capture correctly and then reads the stack trace out of it, turning an opaque restart loop into a specific bug you can fix.

This is written for a memory-constrained VPS where unbounded core files could fill the disk, so every step includes the guard rails. The same debugging discipline applies to any workload you run on the hosting platform.

Why you probably see no core file

Three independent things can suppress a core dump, and missing any one means silence:

ulimit -c                       # must be non-zero for the process
cat /proc/sys/kernel/core_pattern
systemctl show myapp.service -p LimitCORE

On Debian/Ubuntu, core_pattern is typically |/lib/systemd/systemd-coredump %P %u %g %s %t %c %h — the leading pipe means cores are piped to the coredump helper rather than written as a file. That is fine, but only if systemd-coredump is installed and the process’s LimitCORE is not zero. A hardened unit with LimitCORE=0 (a common hardening copy-paste) guarantees you never get a dump.

Setting it up without filling the disk

apt-get install -y systemd-coredump
cat <<'EOF' > /etc/systemd/coredump.conf
[Coredump]
Storage=external
Compress=yes
ProcessSizeMax=2G
ExternalSizeMax=1G
MaxUse=200M
KeepFree=1G
EOF
systemctl restart systemd-coredump.socket

MaxUse=200M and KeepFree=1G are the important guard rails on a small box: cores are compressed and old ones are pruned automatically, so a crash loop cannot exhaust storage. ExternalSizeMax=1G caps any single dump. Verify the effective settings with coredumpctl --help won’t show them — use systemd-analyze cat-config systemd/coredump.conf instead.

Then make sure your unit keeps its dump policy sane. Remove LimitCORE=0 if it is present, or set it explicitly:

[Service]
LimitCORE=infinity

Listing and inspecting dumps

coredumpctl list --since "-2h"
coredumpctl info myapp.service

coredumpctl info prints the signal, fault address, command line, and environment of the crash without extracting anything. The Message field distinguishes “Segmentation fault” (bad memory access) from “Assertion failed” (a deliberate abort()) — assert crashes need no stack inspection, the message already names the failed invariant.

Getting a real backtrace

Symbolised backtraces require the binary and, for interpreted or JIT workloads, the interpreter. Install debug symbols first, otherwise frames are addresses with no names.

coredumpctl gdb myapp.service <<'EOF'
set pagination off
thread apply all bt full
info registers
x/4i $pc-8
info proc mappings
quit
EOF

The three commands that matter most: bt full shows the call stack with local variables, info registers plus x/4i $pc-8 disassembles around the program counter showing exactly which instruction faulted, and info proc mappings tells you whether the fault address was inside a mapped region — a fault on an obvious mapped address means a use-after-free or buffer overflow, while a fault on an unmapped address usually means a wild pointer or stack overflow.

Debug symbols without bloating the box

echo "deb http://deb.debian.org/debian-debug bookworm-debug main" \
  > /etc/apt/sources.list.d/debug.list
apt-get update
apt-get install -y systemd-coredump-dbgsym
# install the specific binary's symbols, not the whole debug repo
debuginfod-find debuginfo /usr/sbin/myapp

Debug packages are large, so install only the symbol package for the crashing binary and remove it afterwards. debuginfod can fetch symbols on demand, but on a metered VPS it is worth controlling the traffic by installing explicitly rather than letting gdb pull them automatically.

Reading the story in a stack trace

A few patterns cover most crash classes:

Frame ends in __memcpy or __strlen with a garbage source pointer: buffer overflow or a freed pointer being read.
Recursion repeated hundreds of frames ending in SIGSEGV: stack overflow — raise LimitSTACK for a stopgap and fix the recursion for real.
Frame in malloc or free with a corrupted heap: double-free, and MALLOC_CHECK_=3 or linking against libasan will localise it faster than stack reading.
Crash only under LimitAS or cgroup memory caps: an allocation failure that the code did not check — grep the source for unchecked malloc.

Automating post-crash capture

A crash that auto-restarts loses its context once the unit is back up if you do nothing. Extract a summary automatically so it survives restarts:

[Service]
OnFailure=core-report@%n.service

# [email protected]
[Service]
Type=oneshot
ExecStart=/usr/local/bin/core-report.sh %i

#!/bin/bash
# /usr/local/bin/core-report.sh
svc="$1"
out="/var/log/crash-$(date +%s)-$svc.txt"
coredumpctl info "$svc" > "$out"
coredumpctl gdb "$svc" <<'EOF' 2>> "$out"
set pagination off
thread apply all bt
quit
EOF
logger -t core-report "captured $out"

With OnFailure= wired up, any crash leaves a text report with a backtrace behind, so you can debug a service that crashes at 3 a.m. without having to reproduce it. Keep the reports pruned with logrotate and you have a self-maintaining crash forensics pipeline on a box with no external tooling.