Setting Up a Mail Server on a VPS: Postfix and Dovecot with SPF, DKIM, and DMARC

Sending mail from a VPS without authentication setup is a one-way ticket to the spam folder: receiving servers check your IP’s reverse DNS, your SPF record, and your DKIM signature before they decide whether your message is legitimate. A bare Postfix install fails all three checks. This guide sets up a working mail server on a VPS with Postfix for SMTP, Dovecot for IMAP, and the three DNS records — SPF, DKIM, DMARC — that determine whether your mail is delivered or dropped.

Before installing anything, confirm three prerequisites. First, your provider must let you set a PTR record (reverse DNS) matching your hostname — most do, some require a support ticket. Second, outbound port 25 must not be blocked; several hosts block it by default to fight spam, so verify with telnet aspmx.l.google.com 25 from your VPS. Third, point an A record like mail.yourdomain.com at your server. If the provider blocks port 25, no config will save you — compare VPS plans on our comparison table and pick a host that leaves SMTP open.

Postfix: The MTA

Install Postfix and Dovecot, choosing Internet Site when prompted:

apt update && apt install -y postfix dovecot-imapd dovecot-pop3d
postconf -e "myhostname = mail.yourdomain.com"
postconf -e "mydomain = yourdomain.com"
postconf -e "myorigin = \$mydomain"
postconf -e "inet_interfaces = all"
postconf -e "mydestination = \$myhostname, localhost.\$mydomain, localhost, \$mydomain"
postconf -e "home_mailbox = Maildir/"
systemctl restart postfix

home_mailbox = Maildir/ switches Postfix to Maildir format, which Dovecot reads natively and which does not require a global file lock. Every user on the system now has a mailbox: create one with useradd -m alice && passwd alice. For real deployments you will want Dovecot’s authentication to back SMTP AUTH on port 587 so your mail clients can send without relaying through an open server — enable smtpd_sasl_auth_enable = yes and the smtp submission service in /etc/postfix/master.cf.

Dovecot: IMAP for Your Clients

Dovecot delivers the inbox to mail clients over IMAP on port 143 (and 993 with TLS). The minimal working config for a single domain:

# /etc/dovecot/conf.d/10-mail.conf
mail_location = maildir:~/Maildir

# /etc/dovecot/conf.d/10-auth.conf
disable_plaintext_auth = yes

# /etc/dovecot/conf.d/10-ssl.conf
ssl = required
ssl_cert = </etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.yourdomain.com/privkey.pem

Get the certificate with certbot certonly --standalone -d mail.yourdomain.com — a mail server without TLS gets its passwords sniffed in cleartext and will be refused by modern clients. Restart Dovecot and check systemctl status dovecot, then test with openssl s_client -connect mail.yourdomain.com:993. Configure your mail client with mail.yourdomain.com as the IMAP server on port 993 with SSL/TLS, and the same hostname on port 587 with STARTTLS for SMTP — port 25 is for server-to-server transfer, not for clients, and most ISPs block it from residential connections anyway.

SPF, DKIM, and DMARC: The Deliverability Trio

These three DNS records tell receiving servers who is allowed to send mail for your domain and what to do with failures. Missing any of them guarantees spam-folder placement:

Record	Type	Value	Purpose
SPF	TXT	v=spf1 mx ip4:YOUR_VPS_IP -all	Lists authorized senders
DKIM	TXT	v=DKIM1; k=rsa; p=...public key...	Cryptographically signs mail
DMARC	TXT	v=DMARC1; p=quarantine; rua=mailto:[email protected]	Tells receivers how to handle failures

DKIM needs the opendkim package and a generated key pair. Create a signing key, wire opendkim to Postfix, and publish the public half as a TXT record named default._domainkey.yourdomain.com:

apt install -y opendkim opendkim-tools
opendkim-genkey -s default -d yourdomain.com -b 2048 -D /etc/opendkim/keys
chown opendkim:opendkim /etc/opendkim/keys/default.private
# postconf -e "milter_default_action = accept"
# postconf -e "smtpd_milters = inet:localhost:8891"
# postconf -e "non_smtpd_milters = inet:localhost:8891"

Then copy the content of /etc/opendkim/keys/default.txt (minus the quotes and the hostname label) into the TXT record for default._domainkey at your DNS provider. If you would rather not assemble this stack from parts on a production box, InterServer’s VPS plans include full root access and unblocked SMTP at flat-rate pricing, which removes the two most common reasons mail setup fails.

Verify Everything

Send a test message and watch the logs, then validate the records from outside:

swaks --to [email protected] --from [email protected] --server mail.yourdomain.com
tail -f /var/log/mail.log            # look for "status=sent"
dig TXT yourdomain.com               # SPF
dig TXT default._domainkey.yourdomain.com   # DKIM
dig TXT _dmarc.yourdomain.com        # DMARC

A Gmail test address is the fastest feedback loop — Google is strict and publishes exactly why it rejected a message in its headers. Work through any SPF alignment or DKIM signature errors, then check your score at a service like mail-tester.com. Finally, lock the server down: restrict relaying to authenticated users only, enable the postfix-sasl jail in fail2ban to catch SMTP brute force, and monitor mail.log for unusual queue growth — an open relay or a compromised account is how VPS mail servers become spam zombies. With reverse DNS, the three records, and TLS in place, your VPS mail server is as deliverable as any hosted service. For the hardware to run it around the clock, see the full specs and pricing on our comparison table.