Finding CPU Hotspots on a Small VPS with perf and Flamegraphs

top shows a process at 100% CPU and tells you nothing about which function is burning it. That is the gap perf and flamegraphs close: instead of sampling at the process level, they sample at the instruction level and aggregate the samples into a single visual tree where the widest frames are the hottest code paths. This walkthrough builds that workflow on a small VPS, from installing perf to reading a flamegraph for the first time.

Installing perf and verifying it works

perf ships with the kernel tools package. On Debian and Ubuntu the package is linux-tools and it must match your running kernel version, which is the first place people get stuck.

# Debian / Ubuntu
apt update
apt install -y linux-tools-common linux-tools-$(uname -r) linux-cloud-tools-$(uname -r)

# RHEL / Alma / Rocky
dnf install -y perf

# sanity check
perf --version
perf stat -e cycles true

If perf stat reports Permission denied or an empty result, check that kernel.perf_event_paranoid is not too restrictive. Values above 2 block non-root sampling; setting it to 1 allows user-space profiling, which is what we need.

sysctl kernel.perf_event_paranoid
# allow user-space sampling (not persistent; add to /etc/sysctl.d/ to keep)
sysctl -w kernel.perf_event_paranoid=1

Sampling the right target on a small VPS

You do not profile the whole system. Profile the process that is busy, and choose the event that matches the symptom. Frequency of samples is a tradeoff: more samples give finer resolution but add overhead you must not ignore on a 1-2 vCPU instance.

Goal	Command	Note
Find CPU hot functions	`perf record -F 99 -p PID -g -- sleep 30`	99 Hz is the standard; 30s window
Whole-system profile	`perf record -F 99 -a -g -- sleep 30`	Higher overhead; use short windows
Count events only	`perf stat -p PID sleep 10`	Cheap; good for IPC and cache misses
Call graph depth	add `--call-graph dwarf`	Better stacks, larger files, more overhead

On a two-vCPU server, an -F 99 profile for thirty seconds costs well under one percent of a core. Avoid profiling for minutes at a time on a small instance; the perf ring buffer itself starts to compete for memory and cache. Short, targeted windows beat long, unfocused ones.

Generating the flamegraph

FlameGraph scripts turn the perf.data file into an SVG. Clone the tooling once, then the pipeline is three commands.

git clone https://github.com/brendangregg/FlameGraph.git

# 1. record
perf record -F 99 -p $(pgrep -n php-fpm) -g -- sleep 30

# 2. dump the stack samples
perf script > out.perf

# 3. fold and render
FlameGraph/stackcollapse-perf.pl out.perf > out.folded
FlameGraph/flamegraph.pl out.folded > flamegraph.svg

Copy the SVG to your laptop and open it in a browser. On a VPS with no desktop you can serve it over an existing nginx vhost or pull it down with scp. The file is self-contained HTML-wrapped SVG, so a single scp is all that is required.

Reading a flamegraph correctly

Width, not height, is cost. A wide frame consumes proportionally more CPU samples. Height is stack depth, not severity.
Look for wide plateaus. A single function occupying 40% of total width is your target; ignore thin slivers.
Follow the widest path first. Trace from the top down through the widest child at each level.
Mind the colors. flamegraph.pl colors are random by default; do not read meaning into hue unless you pass a palette.
Check for missing frames. If the graph is dominated by [unknown], you are missing debug symbols or the right --call-graph mode.

The most common real-world finding on a small web VPS is a wide frame under regex or strstr called from a templating or routing layer — a symptom of unindexed string work being repeated per request. The second most common is time in the kernel under fib_lookup or socket handling, which points at a network configuration problem rather than your code.

A worked example: trimming a hot path

Suppose the flamegraph shows 35% of samples under preg_match called from your framework’s router. That is a routing table being evaluated with regex on every request. The fix is usually to add an exact-match fast path before the regex list, or to compile the routes once at startup. After the change, re-record and diff the two graphs:

perf record -F 99 -p $(pgrep -n php-fpm) -g -- sleep 30
perf script | FlameGraph/stackcollapse-perf.pl | FlameGraph/flamegraph.pl > after.svg
perf diff old.perf after.perf   # function-level delta, if you kept the raw files

Keep the raw perf.data files alongside the SVGs. perf diff gives you a numeric before/after comparison that is far more convincing in a change review than two pictures.

Reducing perf’s own overhead

Lower the sampling rate to -F 49 when profiling a busy production process; the loss of resolution is usually irrelevant.
Prefer --call-graph fp over dwarf when binaries are compiled with frame pointers — stack unwinding becomes far cheaper.
Profile a single thread with -t TID rather than the whole process when only one worker is hot.
Write perf.data to a tmpfs mount to avoid competing with your workload for disk I/O.

If the workload is so tight that even 1% overhead is unacceptable, capture a short profile at the start of a maintenance window instead of continuously. Perf is a diagnostic, not an always-on monitor; use it to find the hotspot, then use lightweight counters to track whether your fix held.

When to reach for perf

Use perf when CPU is the bottleneck, when the process is busy but the profile is unknown, and when you have already ruled out I/O wait and steal. It is a poor tool for diagnosing slow disk, network latency, or memory growth — those need iostat, tcpdump, or heap profiling respectively. Matching the tool to the bottleneck is half the work; if you are sizing new capacity to match a profiling result, review our VPS plans to buy only the cores the profile justifies, and compare managed versus unmanaged options if you want the platform to carry part of the tuning burden.