{"id":1072,"date":"2026-09-07T23:13:59","date_gmt":"2026-09-07T23:13:59","guid":{"rendered":"https:\/\/virtualserversvps.com\/blog\/?p=1072"},"modified":"2026-09-28T22:02:33","modified_gmt":"2026-09-28T22:02:33","slug":"node-exporter-cardinality-control-prometheus-vps","status":"publish","type":"post","link":"https:\/\/virtualserversvps.com\/blog\/node-exporter-cardinality-control-prometheus-vps\/","title":{"rendered":"Node Exporter Cardinality Control: Cutting Prometheus Series Churn on a Single-VPS Stack"},"content":{"rendered":"\n

A default node_exporter<\/code> on a modern Linux host exposes well over a thousand series, and the collector set keeps growing. On a single VPS running Prometheus locally, the first real constraint is not ingest rate but the size of the active series set held in memory combined with the head block written every two hours. Reduce cardinality with intent and you can keep a 30-day retention window on a 1 GB instance without losing the signals you actually alert on.<\/p>\n\n\n\n

Measure cardinality before removing anything<\/h2>\n\n\n\n

Start with the top series producers on the \/metrics<\/code> endpoint itself, before Prometheus is even involved. The --collector.disable-defaults<\/code> flag combined with explicit enables is the controlled way to build a minimal set, but first quantify what each collector costs:<\/p>\n\n\n\n

curl -s localhost:9100\/metrics | grep -v '^#' | grep -v '_total$' \\\n  | sed 's\/{.*\/\/; s\/ [0-9].*\/\/' | sort | uniq -c | sort -rn | head -25\n\n# and per labelled metric, worst offenders\necho 'topk(15, count by (__name__)({job=\"node\"}))' \\\n  | curl -s --data-urlencode 'query=topk(15, count by (__name__)({job=\"node\"}))' \\\n      http:\/\/localhost:9090\/api\/v1\/query | python3 -m json.tool | head -60<\/code><\/pre>\n\n\n\n
The usual results: node_schedstat_*<\/code> (one series per CPU per state), node_softnet_*<\/code>, node_cpu_seconds_total<\/code> at 8-11 series per core, and node_systemd_unit_state<\/code>, which produces 8 series per systemd unit and can be the single largest contributor on a host with dozens of units. On a 4-vCPU box, node_cpu_seconds_total<\/code> alone is roughly 44 series; add the per-core schedstat breakdown and you have doubled that for data you will never graph.<\/p>\n\n\n\n
Collector<\/th>Series scale<\/th>Keep?<\/th>Rationale<\/th><\/tr><\/thead>
cpu, memory, filesystem<\/td>low, per-core\/mount<\/td>yes<\/td>core alerting input<\/td><\/tr>
systemd<\/td>8 x units<\/td>filtered<\/td>whitelist units you own<\/td><\/tr>
schedstat<\/td>~30 x cores<\/td>no<\/td>rarely actionable<\/td><\/tr>
softnet<\/td>~11 x cores<\/td>conditional<\/td>only for NIC-drop debugging<\/td><\/tr>
pressure (PSI)<\/td>3 resource x 3 windows<\/td>yes<\/td>best saturation signal available<\/td><\/tr>
mountstats<\/td>large on NFS<\/td>no unless NFS<\/td>combinatorial explosion<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Filter labels at scrape time, not in queries<\/h2>\n\n\n\n
Metric relabeling is cheaper than a query-time filter because the series never enters the TSDB. The distinction matters: a metric_relabel_configs<\/code> drop rule reduces storage and memory, while a PromQL filter only reduces response size. Here is a targeted drop that keeps CPU accounting intact while removing the schedstat and softnet explosion:<\/p>\n\n\n\n
scrape_configs:\n  - job_name: node\n    static_configs:\n      - targets: ['localhost:9100']\n    metric_relabel_configs:\n      - source_labels: [__name__]\n        regex: 'node_schedstat_.*|node_softnet_.*|node_mountstats_.*'\n        action: drop\n      - source_labels: [__name__, unit]\n        regex: 'node_systemd_unit_state;.*\\.(mount|socket|slice|target|automount)'\n        action: drop\n      - source_labels: [__name__]\n        regex: 'node_cpu_seconds_total'\n        action: keep\n      - source_labels: [__name__, mode]\n        regex: 'node_cpu_seconds_total;(guest|guest_nice|steal|iowait)'\n        action: drop<\/code><\/pre>\n\n\n\n
Dropping iowait<\/code> from node_cpu_seconds_total<\/code> is a judgment call. On a VPS, iowait<\/code> is frequently an artifact of the hypervisor’s block layer and steal<\/code> is the more reliable contention indicator – but if you alert on storage saturation via iowait, keep it and drop guest_nice<\/code> instead. The point is that the decision is explicit rather than inherited from the default collector set.<\/p>\n\n\n\n
Per-mount and per-device labels<\/h2>\n\n\n\n
node_filesystem_*<\/code> emits roughly 12 series per mount point across eight metric families. Containerized hosts with overlay and tmpfs mounts included by default can triple this. Filtering on the mountpoint label rather than dropping the whole family preserves the signal you need for the root and data volumes:<\/p>\n\n\n\n
      - source_labels: [mountpoint]\n        regex: '\/run\/.*|\/sys\/.*|\/var\/lib\/docker\/.*|\/dev\/shm'\n        action: drop\n      - source_labels: [fstype]\n        regex: 'tmpfs|overlay|squashfs|nsfs'\n        action: drop\n\n# verify post-change series count\ncurl -s 'http:\/\/localhost:9090\/api\/v1\/query?query=count(count+by(__name__)({job=\"node\"}))' \\\n  | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d[\"data\"][\"result\"][0][\"value\"][1],\"series names\")'<\/code><\/pre>\n\n\n\n
Check the impact on the head block rather than trusting a series count alone. Prometheus exposes prometheus_tsdb_head_series<\/code> and prometheus_tsdb_head_chunks_storage_size_bytes<\/code>; the second number divided by the first tells you the average chunk footprint and whether a small number of high-churn series is inflating the head.<\/p>\n\n\n\n
Retention arithmetic on a small disk<\/h2>\n\n\n\n
A typical compressed sample is 1 to 2 bytes. With 2,000 active series at a 15-second interval, that is 11.5 samples per series per minute, or roughly 33 MB per day at a 1.5-byte average – about 1 GB for 30 days before index overhead. Reduce the scrape interval to 30 seconds for node metrics you alert on at a 5-minute threshold and the same retention fits in half the space with no loss of resolution for anything you act on.<\/p>\n\n\n\n