The Jail Story: 2,408 Bans That Blocked Nobody

The setup, plainly

One DigitalOcean droplet. No CDN, no Cloudflare proxy in front — the IP is the domain and the domain is the IP, which is the whole point of running it ourselves. SSH is key-only: PasswordAuthentication no, MaxAuthTries 6, no root password login. nginx serves the site; fail2ban watches the logs; the site is a self-hosting blog that costs less per month than most gaming subscriptions.

This is what the numbers looked like on 12 October, straight out of fail2ban-client status:

Two of those three numbers were fiction, and we're going to take them in order, because the way they failed is more instructive than the fact that they did.

127,453 attempts that could never work

The sshd count is real, in the narrow sense that 127,453 pre-authentication attempts hit port 22. That's roughly 466 a day, around the clock: indiscriminate botnets sweeping entire IPv4 space for password logins, the digital equivalent of trying every doorknob in the city.

Not one of them could have succeeded. Password authentication is off; the attempts die at the identity-proof step against a key they do not have. The number measures our attractiveness to scanner noise, not a threat. Every self-hosted box with port 22 open reads the same score. The mistake would be either extreme: ignoring it entirely, or treating 127k as "under attack" when it means "unremarkable."

The lesson here isn't the volume. It's what the volume did to the next number.

2,408 bans that blocked nobody

While wiring the monitoring dashboard we built one rule: never display a claim a tool makes about itself; display what the kernel actually enforces. So we asked nftables — the packet-filter layer itself — what its ban sets contained, and cross-checked against fail2ban's claimed bans.

The fail2ban log had the smoking gun sitting in plain sight, one line repeated for as long as the retention goes back:

ERROR ... 'nft: not found'
Failed to execute ban jail 'sshd' action 'nftables'

Debian 13 ships fail2ban with banaction = nftables as the default. The box did not have the nft binary installed. Every ban the sshd jail recorded — all 2,408 of them, over months — was bookkeeping. fail2ban wrote a proud line in its log, tried to invoke a tool that wasn't there, and the "banned" IP kept knocking. The repeat offenders proved it: one address logged 3 bans inside 20 minutes, because a ban that blocks nothing invites unlimited retries. The only jail that had actually been working was the odd one out — cloudherder-auth, configured with the older iptables action.

apt install nftables, restart, and the three bans the scanner jail claimed materialised in the kernel set within seconds. Enforcement proof, not log proof.

That discovery reframed the whole dashboard: the pre-fix ban counts were measuring how interesting our server looked to botnets, not how well it defended. Same interface. Different reality. The number you can see is only worth what you've verified about the layer beneath it.

The firewall that wasn't

While auditing that week we ran the standard check: ufw status. It printed nothing. We nearly read that as "inactive but installed" — it was actually worse. ufw was not installed at all. The command was erroring, and an old helper script with a 2>/dev/null on it had been swallowing the error for as long as anyone had checked.

There is a specific class of failure hiding here: the absence of a check is not a passing check. The redirect was written months ago to quiet noise, and quiet noise is indistinguishable from silence-with-victory until you go looking. Default-deny, allow 22/80/443, enabled at boot — five minutes of work, and one deliberate exception we'd broken SSL before: port 80 stays open, because the certificate renews over HTTP-01. Close it "for hardening" and the cert quietly walks off a cliff in ninety days.

(One more gotcha from the same week: systemctl is-active ufw reports inactive on a correctly running ufw, because its unit is a oneshot. Trust ufw status, not the service manager. We wrote the card to parse the tool, not the unit.)

What the stack looks like now

Four layers, each verified against the layer beneath, in the order we arrived at them:

1. sshd key-only — the actual defence against the 127,453. Was in place the whole time; nothing else was needed.
2. ufw default-deny — closes every port that isn't the point.
3. fail2ban with escalation — repeat offenders now get exponentially longer bans (bantime.increment, factor 2, capped at a week). The landmine here: increment settings belong in jail.local, not fail2ban.local. Written to the wrong file they load with no error and do nothing. We know, because we proved it with fail2ban-client -d before trusting the config. A silent no-op is the most dangerous failure mode in configuration management: it looks like the feature.
4. A permanent-abuse table — for the botnets that rotate through sequential /24s faster than per-IP bans can catch them: four ranges (2.57.122.0/24, 80.94.92.0/24, 92.118.39.0/24, 195.178.110.0/24) silently blackholed at nftables hook priority -20, ahead of the jail and firewall layers. Seven packets dropped within minutes of enablement — the first honest number this stack ever produced.

What we deliberately didn't block matters as much as what we did. One /24 went quiet the day enforcement was fixed — quiet is not guilt. The single most-banned address on the box (22 lifetime bans) is one IP, not a range; the escalation ladder is allowed to finish its work before the blunt instrument comes out. And a hard guardrail from the very first enforcement day: the scanner jail banned an address inside Cloudflare's space — which is how a privacy-minded reader with WARP looks to a log. CDN and VPN ranges never join a blocklist. The ban-the-world shortcut is a users-removal program wearing a security costume. No country blocks either; a hobby blog that geo-fences the planet has optimised for the wrong variable.

The discipline, not the drama

The stack is now small, boring, and verified at each layer — which is a different claim than "installed," and the difference took us 2,408 fake bans to learn. Every Saturday the tally runs again: zgrep across the retained logs, cross-check the abuse table counters against the kernel, drop the ranges that stay quiet, add the ones that rotate. The dashboard is growing a card for the permanent bans this week, so the check stays one screen instead of a terminal session — the same principle as everything above: make the true number easier to see than the convenient one.

What's in your control: what your tools enforce, as opposed to what they log. The stderr you redirect. The config file the option actually belongs in. The cadence of re-checking, because a defence that was verified once was verified once. What's not: the 466 attempts a day, the botnets' choice of hosting provider, or whether the scanner sweeping your ports has opinions about your framework. What we're testing next: the weekly review as automation instead of ritual — ban tallies and abuse-table counters visible on the dashboard, one glance a Saturday, receipts in git. If the numbers stay honest, we'll publish what they do next month.

Disclosure: the audit lines, log quotes and counters in this post were collected live by CloudHerder's agent during the hardening work of 12–17 October 2026 (verification chain in our deploy notes); the writing, judgement calls and errors of judgement are human-edited and human-signed.