VPS Intrusion Detection: Setting Up Fail2ban, RKHunter, and AIDE on Your Server

Securing a VPS requires more than a strong SSH password and a firewall. Attackers constantly scan for vulnerable services, outdated packages, and backdoor access points. An intrusion detection strategy combines three tools — Fail2ban for active threat blocking, RKHunter for rootkit detection, and AIDE for filesystem integrity checking. This guide walks through installation and configuration for each. For providers with built-in DDoS protection and security features, see our VPS comparison table.

What Youll Need

• A VPS running Ubuntu 22.04, Debian 12, or AlmaLinux 9
• Root or sudo access
• Outbound email access (or an SMTP relay) for alert notifications
• At least 1 GB of free disk space for log files and database snapshots
• Basic familiarity with systemd, cron jobs, and Linux file permissions

Step 1: Install and Configure Fail2ban

Fail2ban scans log files (e.g., /var/log/auth.log) for repeated authentication failures and temporarily bans offending IP addresses using iptables or nftables.

Installation

sudo apt update && sudo apt install fail2ban -y   # Debian/Ubuntu\nsudo dnf install fail2ban -y                      # AlmaLinux/RHEL

Configuration

Create a local jail file to override defaults:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local\nsudo nano /etc/fail2ban/jail.local

Configure SSH protection with a notification action:

[sshd]\nenabled = true\nport = ssh\nfilter = sshd\nlogpath = /var/log/auth.log\nmaxretry = 5\nbantime = 3600\nfindtime = 600\n\n[nginx-http-auth]\nenabled = true\nport = http,https\nfilter = nginx-http-auth\nlogpath = /var/log/nginx/error.log\nmaxretry = 10\nbantime = 1800\n\n[postfix]\nenabled = true\nfilter = postfix\nlogpath = /var/log/mail.log\nmaxretry = 3\nbantime = 3600

Enable the service and verify active jails:

sudo systemctl enable fail2ban && sudo systemctl start fail2ban\nsudo fail2ban-client status

Email Alerts

Configure the destemail and action settings in jail.local to receive email notifications when an IP is banned.

Step 2: Install and Configure RKHunter

RKHunter (Rootkit Hunter) scans for rootkits, backdoors, and local exploits by comparing file properties against known good values and checking for suspicious kernel modules.

Installation

sudo apt install rkhunter -y   # Debian/Ubuntu\nsudo dnf install rkhunter -y   # AlmaLinux/RHEL

Initial Database Setup

Update RKHunters file properties database and run the initial system scan:

sudo rkhunter --propupd\nsudo rkhunter --check --skip-keypress

Scheduled Scans

Set up a daily cron job to check for rootkits and email the report.

Step 3: Install and Configure AIDE

AIDE (Advanced Intrusion Detection Environment) initializes a database of file checksums, permissions, and ownership, then compares the live filesystem against that database to detect changes.

Installation

sudo apt install aide -y   # Debian/Ubuntu\nsudo dnf install aide -y   # AlmaLinux/RHEL

Configuration

Edit /etc/aide/aide.conf to define which directories to monitor. A sensible starting configuration for a typical web server monitors /etc, /bin, /sbin, /lib, /usr/bin, /usr/sbin, and /boot while excluding /var/log, /tmp, /proc, and /sys.

Initialize the AIDE database:

sudo aideinit\nsudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Scheduled Checks

Add a daily cron job for AIDE integrity checks. Store the database on read-only media or transfer it off-server after initialization — an attacker who gains root can modify the database to hide changes.

Step 4: Interpret Scan Results

Fail2ban logs — Check banned IPs with sudo fail2ban-client status sshd. Investigate repeated patterns from the same IP ranges (could be a distributed attack).

RKHunter warnings — Warnings about /dev or /proc are often false positives. Focus on Rootkit checks that flag hidden processes, suspicious kernel modules, or malware signatures.

AIDE changes — Legitimate system updates modify binaries in /usr/bin and /usr/lib. After a package update, regenerate the database. Unexpected changes to /etc/shadow, /etc/cron*, or /etc/ssh warrant immediate investigation.

Automating the Full Security Pipeline

Create a shell script that runs all three checks sequentially and sends a consolidated daily report via cron.

Conclusion

Fail2ban, RKHunter, and AIDE form a layered intrusion detection strategy. Fail2ban stops brute-force attacks in real time, RKHunter identifies active rootkits and malware, and AIDE detects unauthorized file changes that could indicate a breach. No single tool is sufficient — defense in depth is the only approach that works. For VPS providers with advanced security features like DDoS protection and automated patching, check our VPS comparison table.

Related Articles：

Nginx Security Hardening for VPS: Firewall, SSL Config, Rate Limiting, WAF, and 2026 Best Practices How to Secure SSH on Your VPS: Fail2ban, Key Authentication, and Firewall Setup VPS SSH Key Management: Best Practices for Managing Access Across Multiple Servers VPS Firewall Setup: UFW vs iptables vs nftables — A Practical Command-Line Guide for 2026