Custom Fail2ban Filters and Recidive Jails for VPS Bruteforce Defense

The default sshd jail in Fail2ban bans an IP after repeated failures — but the moment an attacker rotates source addresses, a per-IP ban is worthless. This article focuses on what most tutorials skip: writing custom filters for services that log in non-standard formats, using recidive to catch repeat offenders across jails, and testing filters before they go live. Set the base protections up first, then extend with the jails below.

Diagnose which jail is doing the work

Before tuning blind, confirm what is actually being banned. The default sshd jail catches password guesses, but if your box only allows key auth, most failures are pre-auth probes that still count. Look at the distribution of banned addresses over a week.

grep " Ban " /var/log/fail2ban.log | awk '{print $NF}' | sort | uniq -c | sort -rn | head
fail2ban-client status sshd

If the same source subnet keeps reappearing after short bans, the default bantime is too short for that source. That is the exact case the escalating and recidive jails are built for.

Confirm what is actually being banned

Before adding jails, check the numbers. On a default instance the log tells the truth; a grep over the last day shows the real attack surface.

journalctl -u ssh --since yesterday | grep -c "Failed password"
fail2ban-client status sshd

If status sshd shows a healthy ban count but failures keep climbing, the noise is distributed across many single-shot IPs. That is the signal to move to aggregated bans.

Write a custom filter for a non-standard log

Fail2ban ships filters for common services only. For anything else, point a filter at the log and give it a regex with a named <HOST> group. Example for an app that writes JSON lines to its own log:

# /etc/fail2ban/filter.d/myapp.conf
[Definition]
failregex = ^.*"ip":"<HOST>".*"event":"auth_fail".*$
ignoreregex =

Test the filter before enabling it

A wrong regex bans innocent traffic. Always dry-run against real log lines.

fail2ban-regex /var/log/myapp/access.log /etc/fail2ban/filter.d/myapp.conf

The output shows matched and missed lines. Only wire the filter into a jail once the match count is non-zero and the missed lines are genuinely unrelated traffic.

Catch repeat offenders with recidive

Single-IP bans expire. The recidive jail watches Fail2ban’s own log for IPs that trigger multiple bans, then applies a long ban. That is what turns a nuisance filter into a real deterrent.

# /etc/fail2ban/jail.d/recidive.local
[recidive]
enabled  = true
logpath  = /var/log/fail2ban.log
banaction = iptables-allports
bantime  = 1w
findtime = 1d
maxretry = 3

Setting	Nuisance jail	Recidive jail
bantime	1h	1w
findtime	10m	1d
maxretry	5	3
Scope	Single service	All ports

Whitelist before you widen the net

Aggressive jails ban everything they see. Protect your own access and monitoring before enabling them.

# /etc/fail2ban/jail.d/ignoreips.local
[DEFAULT]
ignoreip = 127.0.0.1/8 ::1 203.0.113.7 198.51.100.0/24

• Include your office/static IP and any monitoring or uptime prober that hits protected endpoints.
• Reload, do not restart, to apply without dropping existing bans: fail2ban-client reload.
• Verify the active rule set with iptables -S | grep f2b after each change.

Why per-IP bans are not enough

A single-source ban assumes the attacker uses one address. Distributed credential-stuffing spreads attempts across thousands of addresses, each sending only a handful of failures, so no individual IP ever trips the retry threshold. The defence has two parts: lower your thresholds for the services that matter, and aggregate across jails so that an address banned once is treated as suspect system-wide. The recidive jail does the second; per-service filters with tighter maxretry do the first.

Drop the noisy traffic at the firewall first

Fail2ban reacts to events; a firewall drop rule prevents them. A rule that drops known-bad address ranges before they reach sshd keeps your ban list short and your logs readable.

iptables -I INPUT -m conntrack --ctstate NEW -p tcp --dport 22 -m hashlimit   --hashlimit-name ssh --hashlimit-above 6/min --hashlimit-burst 6   --hashlimit-mode srcip --hashlimit-htable-expire 60000 -j DROP

This limits any single source to six new SSH connections per minute. Legitimate users never notice; credential-stuffing scripts hit the wall immediately.

Tune the ban lifecycle for your threat model

Parameter	Effect	Sensible start
bantime	How long an IP stays blocked	3600 s
findtime	Window over which retries are counted	600 s
maxretry	Failures allowed before ban	4
maxretry (recidive)	Jail triggers before long ban	3
ignoreip	Never ban these	Your admin + monitor IPs

Use incremental banning so repeat offenders escalate: first offence one hour, then one day, then a week. It keeps accidental lockouts short while making persistent attacks expensive.

# /etc/fail2ban/jail.d/escalate.local
[DEFAULT]
bantime.increment = true
bantime.factor = 24
bantime.maxtime = 4w

What to monitor after the fact

• Watch the total banned-IP count over time; a sudden climb usually means a new campaign against your range.
• Alert if the number of currently banned IPs spikes above ten times your daily average.
• Re-check fail2ban-client status after every reload to confirm all jails loaded — a syntax error in one jail file silently drops it.

Handle the base SSH jail with the standard recipe, then let custom filters and recidive carry the load. A hardening pass runs best on a clean system image — see the cloud VPS benefits walkthrough for how the base environment is configured, or browse VPS tutorials for the rest of the hardening series.