Node Exporter Cardinality Control: Cutting Prometheus Series Churn on a Single-VPS Stack

A default node_exporter on a modern Linux host exposes well over a thousand series, and the collector set keeps growing. On a single VPS running Prometheus locally, the first real constraint is not ingest rate but the size of the active series set held in memory combined with the head block written every two hours. Reduce cardinality with intent and you can keep a 30-day retention window on a 1 GB instance without losing the signals you actually alert on.

Measure cardinality before removing anything

Start with the top series producers on the /metrics endpoint itself, before Prometheus is even involved. The --collector.disable-defaults flag combined with explicit enables is the controlled way to build a minimal set, but first quantify what each collector costs:

curl -s localhost:9100/metrics | grep -v '^#' | grep -v '_total$' \
  | sed 's/{.*//; s/ [0-9].*//' | sort | uniq -c | sort -rn | head -25

# and per labelled metric, worst offenders
echo 'topk(15, count by (__name__)({job="node"}))' \
  | curl -s --data-urlencode 'query=topk(15, count by (__name__)({job="node"}))' \
      http://localhost:9090/api/v1/query | python3 -m json.tool | head -60

The usual results: node_schedstat_* (one series per CPU per state), node_softnet_*, node_cpu_seconds_total at 8-11 series per core, and node_systemd_unit_state, which produces 8 series per systemd unit and can be the single largest contributor on a host with dozens of units. On a 4-vCPU box, node_cpu_seconds_total alone is roughly 44 series; add the per-core schedstat breakdown and you have doubled that for data you will never graph.

Collector	Series scale	Keep?	Rationale
cpu, memory, filesystem	low, per-core/mount	yes	core alerting input
systemd	8 x units	filtered	whitelist units you own
schedstat	~30 x cores	no	rarely actionable
softnet	~11 x cores	conditional	only for NIC-drop debugging
pressure (PSI)	3 resource x 3 windows	yes	best saturation signal available
mountstats	large on NFS	no unless NFS	combinatorial explosion

Filter labels at scrape time, not in queries

Metric relabeling is cheaper than a query-time filter because the series never enters the TSDB. The distinction matters: a metric_relabel_configs drop rule reduces storage and memory, while a PromQL filter only reduces response size. Here is a targeted drop that keeps CPU accounting intact while removing the schedstat and softnet explosion:

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['localhost:9100']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'node_schedstat_.*|node_softnet_.*|node_mountstats_.*'
        action: drop
      - source_labels: [__name__, unit]
        regex: 'node_systemd_unit_state;.*\.(mount|socket|slice|target|automount)'
        action: drop
      - source_labels: [__name__]
        regex: 'node_cpu_seconds_total'
        action: keep
      - source_labels: [__name__, mode]
        regex: 'node_cpu_seconds_total;(guest|guest_nice|steal|iowait)'
        action: drop

Dropping iowait from node_cpu_seconds_total is a judgment call. On a VPS, iowait is frequently an artifact of the hypervisor’s block layer and steal is the more reliable contention indicator – but if you alert on storage saturation via iowait, keep it and drop guest_nice instead. The point is that the decision is explicit rather than inherited from the default collector set.

Per-mount and per-device labels

node_filesystem_* emits roughly 12 series per mount point across eight metric families. Containerized hosts with overlay and tmpfs mounts included by default can triple this. Filtering on the mountpoint label rather than dropping the whole family preserves the signal you need for the root and data volumes:

      - source_labels: [mountpoint]
        regex: '/run/.*|/sys/.*|/var/lib/docker/.*|/dev/shm'
        action: drop
      - source_labels: [fstype]
        regex: 'tmpfs|overlay|squashfs|nsfs'
        action: drop

# verify post-change series count
curl -s 'http://localhost:9090/api/v1/query?query=count(count+by(__name__)({job="node"}))' \
  | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d["data"]["result"][0]["value"][1],"series names")'

Check the impact on the head block rather than trusting a series count alone. Prometheus exposes prometheus_tsdb_head_series and prometheus_tsdb_head_chunks_storage_size_bytes; the second number divided by the first tells you the average chunk footprint and whether a small number of high-churn series is inflating the head.

Retention arithmetic on a small disk

A typical compressed sample is 1 to 2 bytes. With 2,000 active series at a 15-second interval, that is 11.5 samples per series per minute, or roughly 33 MB per day at a 1.5-byte average – about 1 GB for 30 days before index overhead. Reduce the scrape interval to 30 seconds for node metrics you alert on at a 5-minute threshold and the same retention fits in half the space with no loss of resolution for anything you act on.

• --storage.tsdb.retention.time=30d and --storage.tsdb.retention.size=800MB together, so a cardinality spike degrades retention instead of filling the disk.
• Scrape node metrics at 30s, application metrics at 15s only where a dashboard genuinely needs it.
• Never enable a collector to inspect one value once; use curl against /metrics for that and leave it disabled.
• Alert on prometheus_tsdb_head_series growth rate to catch a label regression within one scrape cycle.

If you need to run Prometheus, Grafana, and the exporter stack alongside the workload itself, the practical constraint is resident memory rather than CPU. Our VPS plans and available RAM allocations list the headroom we reserve for observability daemons on each tier, which is the number to size against when the monitoring stack shares a host with the application it watches.

Remote write versus local storage

If you keep Prometheus on the same instance as the workload, disk pressure from the TSDB competes with the application’s own I/O. Watch prometheus_tsdb_compaction_duration_seconds and the node-level node_disk_io_time_seconds_total together: compaction spikes that coincide with application latency spikes are the clear signal to either move metrics to remote write or cap the retention size more aggressively. Cardinality control is what makes either option affordable. Provisioning for the combined resident set is covered in the memory specifications we publish on our VPS hosting specifications page.