← Back to Blog

How to Set Up a Mail Server on a VPS (2026)

Published May 4, 2026 · 9 min read · Galaxy Cloud Solutions

I am going to be honest with you before we get into the commands: running your own mail server is genuinely difficult compared to almost anything else you can self-host. Email has decades of anti-spam infrastructure layered on top of it and getting deliverability right takes real effort. That said, it is absolutely doable and for the right use case it is worth every minute. Here is the real guide.

⚡ VPS from $5/mo — Use code LAUNCH2026 for 50% off

Should You Actually Run Your Own Mail Server?

Before we start: if you just need business email, use Google Workspace or Zoho Mail. They cost $6-12/month, handle deliverability automatically, and give you apps that work on every device. The time investment of self-hosting email is not worth it for most people.

Self-hosted email makes sense if you need full control over your email data, you are building a transactional email system for your app, you want to learn how email actually works, or you are handling sensitive communications you do not want on third-party servers.

The Stack: Postfix + Dovecot + Rspamd

• Postfix — handles sending and receiving (SMTP)
• Dovecot — handles IMAP/POP3 so your email client can connect
• Rspamd — spam filtering
• Certbot — SSL for your mail server

Before You Start: Deliverability Checklist

This is the part most guides skip and then wonder why their emails land in spam:

• Reverse DNS (PTR record) — your VPS IP must have a PTR record matching your mail server hostname. Contact your VPS provider to set this up. Without it, major providers will reject your mail.
• Port 25 open — many VPS providers block outbound port 25 by default. You need to request it to be opened.
• Clean IP — check your IP at mxtoolbox.com/blacklists.aspx before starting. If it is blacklisted from a previous tenant, request a new IP or get it delisted first.
• SPF, DKIM, DMARC records — mandatory. We set these up during installation.

Step 1: Set Your Hostname

sudo hostnamectl set-hostname mail.yourdomain.com
echo "YOUR_SERVER_IP mail.yourdomain.com" | sudo tee -a /etc/hosts

Set an A record in your DNS: mail.yourdomain.com pointing to your VPS IP.

Step 2: Install Postfix

sudo apt update
sudo apt install -y postfix
# Choose "Internet Site" when prompted
# Enter your domain name (yourdomain.com) when asked

Step 3: Install Dovecot

sudo apt install -y dovecot-core dovecot-imapd dovecot-pop3d dovecot-lmtpd

Step 4: Get SSL Certificates

sudo apt install -y certbot
sudo certbot certonly --standalone -d mail.yourdomain.com

Step 5: Configure Postfix

sudo tee /etc/postfix/main.cf << 'EOF'
myhostname = mail.yourdomain.com
mydomain = yourdomain.com
myorigin = $mydomain
inet_interfaces = all
mydestination = $myhostname, $mydomain, localhost
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.yourdomain.com/privkey.pem
smtpd_tls_security_level = may
smtp_tls_security_level = may
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
home_mailbox = Maildir/
EOF
sudo systemctl restart postfix

Step 6: Set Up DNS Records

These are non-negotiable for deliverability:

# MX record
yourdomain.com.  MX  10  mail.yourdomain.com.

# SPF record
yourdomain.com.  TXT  "v=spf1 mx ~all"

# DMARC record
_dmarc.yourdomain.com.  TXT  "v=DMARC1; p=none; rua=mailto:[email protected]"

For DKIM, install opendkim and generate keys — this is the step that makes the biggest difference for deliverability.

Step 7: Install Rspamd for Spam Filtering

curl https://rspamd.com/apt-stable/gpg.key | sudo apt-key add -
echo "deb https://rspamd.com/apt-stable/ $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/rspamd.list
sudo apt update && sudo apt install -y rspamd
sudo systemctl enable --now rspamd

Testing Deliverability

After setup, send a test email to mail-tester.com — it gives you a score out of 10 and tells you exactly what is wrong. Aim for 8+ before you consider the server production-ready.