Rotating SSH Keys Across Many Servers: A Practical Access Audit Workflow

Managing SSH keys on one server is easy. Managing them across twenty is where things rot: keys issued to a contractor who left last year, a shared deploy key pasted into six authorized_keys files, and no record of which key maps to which human. This is a runbook for bringing that under control, auditing what is actually installed, and rotating keys without locking yourself out.

This workflow assumes you already have a hardened per-host baseline. If your single-server SSH hardening is still in progress, work through the host-level setup on our VPS platform documentation first, then scale it with the steps below.

Prerequisites

• A list of managed hosts reachable from one admin workstation
• An SSH config with per-host aliases (or a simple inventory file)
• Root or sudo access on every target host
• Python 3 locally for the audit script
• One break-glass key stored offline before you start rotating anything

Step 1: Audit What Is Actually Installed

Do not trust your own records. Read the ground truth from each host and collect fingerprints — never the keys themselves.

cat > /tmp/inventory.txt <<'EOF'
web01
web02
db01
worker01
EOF

while read -r host; do
  echo "=== $host ==="
  ssh -o BatchMode=yes "$host" \
    "getent passwd | awk -F: '\$3>=1000 || \$1==\"root\" {print \$1}' | while read u; do
       h=\$(getent passwd \$u | cut -d: -f6)
       [ -f \"\$h/.ssh/authorized_keys\" ] && \
         ssh-keygen -lf \"\$h/.ssh/authorized_keys\" | sed \"s|^|\$u |\"
     done"
done < /tmp/inventory.txt | tee /tmp/key-audit.txt

This prints every key’s bit size, fingerprint, comment, and owning account. Review the result for three red flags: keys with no comment (impossible to attribute), 2048-bit RSA keys on new systems, and the same fingerprint on accounts that should not share access.

Step 2: Normalize authorized_keys With a Managed Block

Hand-edited authorized_keys files drift. Constrain them with options and a comment that names the owner, so future audits are self-documenting.

cat >> /home/deploy/.ssh/authorized_keys <<'EOF'
# owner:alice expires:2027-01-31 from=10.0.0.0/8
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... alice@workstation-2026
EOF

chmod 700 /home/deploy/.ssh
chmod 600 /home/deploy/.ssh/authorized_keys
chown -R deploy:deploy /home/deploy/.ssh

Use from= to restrict source networks and expiry-time= where your OpenSSH supports it (8.2+), which lets the key die on its own without a coordinated removal:

ssh -V
# If >= OpenSSH_8.2:
#   ssh-ed25519 AAAA... alice@ws expires="2027-01-31" [email protected]
# OpenSSH silently enforces expires="YYYY-MM-DD"

Step 3: Rotate Without Locking Yourself Out

Always add the new key first, prove it works, and only then remove the old one. Never do both in one session.

# 1. Generate the replacement
ssh-keygen -t ed25519 -a 100 -C "alice@workstation-2026" -f ~/.ssh/id_ed25519_2026

# 2. Install alongside the old key on every host
while read -r host; do
  ssh-copy-id -i ~/.ssh/id_ed25519_2026.pub "deploy@$host"
done < /tmp/inventory.txt

# 3. Prove the new key works in a clean, non-interactive session
while read -r host; do
  ssh -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_2026 -o BatchMode=yes \
      "deploy@$host" 'echo OK $(hostname)'
done < /tmp/inventory.txt

Only proceed to removal when every host prints OK. Then strip the retired fingerprint:

OLD_FP="SHA256:abc123OldFingerprint..."
while read -r host; do
  ssh "deploy@$host" "sed -i '/${OLD_FP#SHA256:}/d' ~/.ssh/authorized_keys || \
    ssh-keygen -R placeholder"
done < /tmp/inventory.txt

Deleting by fingerprint string inside a remote sed is fragile. The reliable method is to run the audit from Step 1 after removal and confirm the fingerprint no longer appears — treat that audit as your source of truth rather than the edit command.

Step 4: Enforce and Monitor

Lock down the daemon so removed keys cannot be reintroduced by a forgotten account, and log who authenticated with what.

# /etc/ssh/sshd_config.d/10-hardening.conf
PermitRootLogin prohibit-password
PasswordAuthentication no
PubkeyAuthentication yes
KbdInteractiveAuthentication no
AllowUsers deploy alice

sshd -t && systemctl reload sshd

Turn on fingerprint logging so authentication events are attributable:

# /etc/ssh/sshd_config.d/20-logging.conf
LogLevel VERBOSE

journalctl -u ssh --since "24 hours ago" | grep -i "Accepted publickey"

Verification Step

Re-run the Step 1 audit and confirm the retired fingerprint count is zero across the fleet. Then confirm a negative test — the retired key must be rejected:

ssh -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_OLD -o BatchMode=yes \
    deploy@web01 true
# Expected: Permission denied (publickey). Non-zero exit is the pass condition.

grep -c "owner:" /tmp/key-audit.txt   # every key should be attributable

Troubleshooting

Symptom	Cause	Fix
Permission denied (publickey) after adding key	Home directory permissions too open	chmod 750 ~, 700 ~/.ssh, 600 authorized_keys
New key ignored, old one still works	Client offering too many identities	Use -o IdentitiesOnly=yes
expires= not enforced	OpenSSH older than 8.2	Rotate manually or upgrade OpenSSH
Locked out after hardening	No retained session during reload	Keep a session open; use sshd -t before reload
Audit shows unattributed keys	Legacy keys with no comment	Remove and reissue with owner comments
sshd config edits ignored	Directive overridden earlier	Order files in sshd_config.d/ and verify with sshd -T

The discipline that makes this scale is the order of operations: audit, add, verify, remove, re-audit. Follow that sequence and a fleet-wide key rotation becomes a routine maintenance window instead of an outage risk.