Intrusion Detection on a VPS: Fail2ban, AIDE and Rootkit Scans That Survive Reboots

A firewall and updated packages stop known attacks; they do nothing about a compromised binary or a modified system file. Intrusion detection on a VPS answers three distinct questions: who is currently attacking, is there a rootkit present, and has anything on disk changed since you last trusted it. Fail2ban, RKHunter and AIDE answer those three questions respectively — and they only work if their output survives a reboot and reaches a human.

Fail2ban: Blocking the Obvious, Logging the Rest

Fail2ban reads logs and adds firewall rules on repeat offenders. The common mistake is tuning it so aggressively that a mistyped password locks you out of your own box. Use findtime and maxretry deliberately, and always whitelist your own management path.

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sudo tee /etc/fail2ban/jail.local <<'EOF'
[DEFAULT]
bantime  = 3600
findtime = 600
maxretry = 5
ignoreip = 127.0.0.1/8 ::1 YOUR.MGMT.IP/32

[sshd]
enabled = true
mode    = aggressive

[nginx-http-auth]
enabled = true
EOF
sudo systemctl enable --now fail2ban

Verify the jails actually load rather than assuming:

sudo fail2ban-client status
sudo fail2ban-client status sshd

The recidive jail matters on internet-facing hosts. It watches Fail2ban’s own ban history and issues long bans to IPs that return repeatedly — a single sshd jail only sees one service.

RKHunter: Scanning for Rootkits and Suspicious State

RKHunter checks for known rootkit signatures, hidden files, and unexpected modifications to system binaries. Its weakness is false positives from legitimate package updates, which is exactly why you baseline it on a clean install.

sudo apt install rkhunter -y
sudo rkhunter --propupd          # record known-good properties NOW
sudo rkhunter --check --sk

--propupd must be re-run after every legitimate system upgrade. Skip that step and every future scan reports the package manager’s own changes as warnings, training you to ignore real alerts. Schedule the scan and route the output somewhere durable:

sudo tee /etc/systemd/system/rkhunter.service <<'EOF'
[Unit]
Description=RKHunter scan
[Service]
Type=oneshot
ExecStart=/usr/bin/rkhunter --cronjob --report-warnings-only --rwo
EOF
sudo systemctl enable --now rkhunter.timer

AIDE: Detecting Change You Did Not Make

AIDE builds a cryptographic database of file hashes, permissions and timestamps, then compares against it. The value is in the baseline: an alert that a file changed is noise unless you know it should not have. Initialize on a freshly hardened system:

sudo apt install aide -y
sudo aideinit
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
sudo aide --check

On a small VPS, a full scan competes for disk I/O with your workloads. Constrain AIDE to the paths that matter instead of the whole filesystem:

• Watch /etc, /bin, /sbin, /usr/bin, /usr/sbin — where persistence lives.
• Exclude /var/log, /var/cache, and application upload directories that change constantly.
• Run nightly with nice and ionice so the scan never spikes user-facing latency.

Alerting: The Detection That Fails Silently Is Useless

All three tools write to logs that nobody reads. Make sure output reaches you, and that the path is tested. A minimal approach is a wrapper that emails or posts to a webhook only when a scan reports findings, keeping noise near zero so real alerts stand out.

Tool	Answers	Cadence	Disk cost
Fail2ban	Who is attacking now?	Continuous	Negligible
RKHunter	Is a known rootkit present?	Nightly	Low
AIDE	Did files change unexpectedly?	Nightly	Moderate

Correlating the Three Signals

The real value of running all three detection layers is correlation. A change detected by AIDE in /usr/sbin, combined with an unexpected process listening on a new port and a burst of outbound connections in the firewall log, is a coherent incident. Any one of those alone is ambiguous; together they tell a story. Keep the timestamps aligned — journald, AIDE and Fail2ban all log in local time, so a drifting clock makes correlation unreliable.

On a compromised host, the attacker’s first move is often to disable or tamper with these very tools. That is why integrity of the detection stack itself matters. AIDE watches /usr/bin and /etc where Fail2ban’s filters live; if someone edits a filter to ignore a specific IP, the next nightly AIDE run should catch the modified file hash. This mutual coverage is the point of stacking tools rather than trusting any single one.

Surviving Reboots and Restores

Detection state is only useful if it persists. Confirm every service is enabled, not just started, and that databases live outside volatile paths:

systemctl is-enabled fail2ban rkhunter.timer aide.timer
ls -l /var/lib/aide/aide.db /var/lib/rkhunter/db

Then test the recovery path: reboot, confirm jails are active, and confirm AIDE can still read its baseline. A detection stack that breaks silently after a reboot is worse than none, because you believe you are covered when you are not. If you are auditing a server you have not touched in months, pair this with a provider-focused security review to check the infrastructure layer too, since host-level protections complement — but do not replace — what runs on the instance.

Putting It Together

1. Baseline RKHunter with --propupd and AIDE with aideinit on a clean build.
2. Configure Fail2ban with a management-IP whitelist and the recidive jail.
3. Schedule nightly scans with I/O priority limits.
4. Route warnings to a channel a human actually monitors.
5. Re-baseline after every legitimate upgrade and test survivability after reboot.

Run on a schedule and re-baselined honestly, these three tools turn “I think the server is clean” into an answer with evidence behind it. Choosing an instance with the disk headroom to run integrity scans without hurting production is part of the design — see how these hosts compare before scaling the workload.