Self-Hosted Email Server: Setting Up Mail with Docker and Mailcow

Email is arguably the most critical communication infrastructure on the internet, yet most organizations hand control of it to third-party providers like Google Workspace or Microsoft 365. Self-hosting your email server gives you complete ownership of your data, eliminates per-user licensing fees, and removes the risk of a provider suddenly changing terms or scanning your messages for advertising purposes.

That said, running your own mail server is one of the more demanding self-hosting projects. Email deliverability depends on correct DNS records, proper TLS configuration, active spam filtering, and maintaining a clean IP reputation. This guide walks through the entire process using Mailcow, a production-ready Docker-based mail server suite.

Why Self-Host Email?

Before committing to this path, weigh the benefits against the very real operational burden:

Reality check: If you are running a business where email downtime directly costs revenue, consider a hybrid approach: self-host for internal and sensitive communications, and use a transactional email provider (like Postmark or Amazon SES) for outbound bulk/notification email.

Mailcow vs Mail-in-a-Box vs Mailu

Three dominant open-source mail server solutions exist for self-hosters. Each takes a different approach:

This guide focuses on Mailcow because it offers the best balance of features, customization, and Docker-native deployment. If you want a simpler setup and are willing to dedicate a full machine, Mail-in-a-Box is a solid alternative. Mailu is worth considering if you need a lighter Docker-based option.

Prerequisites

Before you begin, ensure you have:

• A VPS or dedicated server with a clean IP address (check against DNSBL lists at mxtoolbox.com)
• A static IP with proper reverse DNS (PTR record) pointing to your mail hostname
• Port 25 open (many cloud providers block this by default; you may need to request unblocking)
• Ports 80, 443, 587, 465, 993, 995, 4190 available
• At least 2 GB RAM (4 GB recommended with ClamAV enabled)
• Docker Engine 20.10+ and Docker Compose v2
• A domain name with full DNS control

DNS Configuration

DNS is the foundation of email deliverability. Incorrect records mean your messages end up in spam folders or get rejected outright. For a mail server at mail.example.com handling email for example.com, configure the following:

MX Record

The MX record tells other mail servers where to deliver email for your domain:

# MX record for example.com
example.com.    IN    MX    10    mail.example.com.

# A record for the mail server
mail.example.com.    IN    A    203.0.113.50

# AAAA record if you have IPv6
mail.example.com.    IN    AAAA    2001:db8::50

SPF (Sender Policy Framework)

SPF tells receiving servers which IP addresses are authorized to send email for your domain:

# SPF record - only your mail server can send
example.com.    IN    TXT    "v=spf1 mx a:mail.example.com -all"

# If you also use a third-party for transactional email:
example.com.    IN    TXT    "v=spf1 mx a:mail.example.com include:_spf.google.com -all"

The -all suffix is a hard fail policy, meaning any server not listed is unauthorized. Use ~all (soft fail) during initial testing, then switch to -all once confirmed working.

DKIM (DomainKeys Identified Mail)

DKIM cryptographically signs outgoing messages so recipients can verify they were not tampered with. Mailcow generates DKIM keys automatically. After setup, add the provided public key as a TXT record:

# DKIM public key (Mailcow generates this)
dkim._domainkey.example.com.    IN    TXT    "v=DKIM1; k=rsa; p=MIIBIjANBgkqhki..."

DMARC (Domain-based Message Authentication)

DMARC ties SPF and DKIM together and tells receivers what to do with messages that fail authentication:

# DMARC record - start with monitoring, then enforce
_dmarc.example.com.    IN    TXT    "v=DMARC1; p=none; rua=mailto:[email protected]; pct=100"

# After monitoring looks clean, switch to quarantine or reject:
_dmarc.example.com.    IN    TXT    "v=DMARC1; p=reject; rua=mailto:[email protected]; pct=100"

Reverse DNS (PTR Record)

This is configured at your hosting provider, not in your domain DNS. The PTR record for your server IP must resolve to your mail hostname:

# PTR record (set at hosting provider)
50.113.0.203.in-addr.arpa.    IN    PTR    mail.example.com.

# Verify DNS records
dig MX example.com +short
dig TXT example.com +short
dig TXT dkim._domainkey.example.com +short
dig TXT _dmarc.example.com +short
dig -x 203.0.113.50 +short

Installing Mailcow with Docker

Mailcow uses a well-structured Docker Compose deployment with over a dozen containers working together:

# Clone the Mailcow repository
cd /opt
git clone https://github.com/mailcow/mailcow-dockerized.git
cd mailcow-dockerized

# Run the configuration generator
./generate_config.sh

# This will prompt you for:
# - Mail server hostname (FQDN): mail.example.com
# - Timezone: America/New_York (or your timezone)
# - ClamAV: yes (recommended) or no (saves ~1GB RAM)
# - Solr: yes (full-text search) or no

The generator creates a mailcow.conf file. Review and adjust key settings:

# mailcow.conf - key settings
MAILCOW_HOSTNAME=mail.example.com
MAILCOW_PASS_SCHEME=BLF-CRYPT
DBPASS=CHANGE_THIS_STRONG_PASSWORD
DBROOT=CHANGE_THIS_STRONG_ROOT_PASSWORD

# TLS configuration
SKIP_LETS_ENCRYPT=n
ADDITIONAL_SAN=autodiscover.example.com,autoconfig.example.com

# Resource limits
SKIP_CLAMD=n
SKIP_SOLR=y  # Enable only if you have 4GB+ RAM

# Logging
LOG_LINES=9999
COMPOSE_PROJECT_NAME=mailcowdockerized

Start the stack:

# Pull images and start all containers
docker compose pull
docker compose up -d

# Watch the startup logs
docker compose logs -f --tail=100

# Check all containers are healthy
docker compose ps

Mailcow spins up approximately 15 containers including Postfix (SMTP), Dovecot (IMAP), Rspamd (spam filter), SOGo (webmail), Nginx, MySQL, Redis, ClamAV, and others. Initial startup takes 2-5 minutes as ClamAV downloads virus definitions.

Post-Installation Configuration

Access the admin UI at https://mail.example.com with default credentials admin / moohoo. Change the password immediately.

Adding Your Domain

1. Navigate to Configuration > Mail Setup > Domains
2. Add your domain (example.com)
3. Set the default mailbox quota
4. Enable DKIM signing and note the generated key
5. Add the DKIM TXT record to your DNS

Creating Mailboxes

1. Go to Configuration > Mail Setup > Mailboxes
2. Create mailboxes for your users
3. Set individual quotas as needed
4. Configure aliases under Aliases

TLS Certificate Configuration

Mailcow handles Let's Encrypt automatically if ports 80 and 443 are accessible. Verify the certificate:

# Check TLS certificate
echo | openssl s_client -connect mail.example.com:443 -servername mail.example.com 2>/dev/null | \
  openssl x509 -noout -dates -subject

# Test SMTP STARTTLS
echo | openssl s_client -connect mail.example.com:587 -starttls smtp 2>/dev/null | \
  openssl x509 -noout -dates

Spam Filtering with Rspamd

Mailcow uses Rspamd, a high-performance spam filter. Access the Rspamd UI at https://mail.example.com/rspamd.

Key Rspamd tuning settings:

# Custom Rspamd local configuration
# Place in data/conf/rspamd/local.d/

# actions.conf - adjust scoring thresholds
reject = 15;
add_header = 6;
greylist = 4;
rewrite_subject = 8;

# greylist.conf - greylisting for unknown senders
enabled = true;
timeout = 300s;
expire = 1d;

# antivirus.conf - ClamAV integration
clamav {
  action = "reject";
  type = "clamav";
  servers = "clamd:3310";
}

Rspamd learns from your spam and ham classifications. Train it regularly:

# Train Rspamd on spam (move messages to Junk folder in your mail client)
# Mailcow auto-trains when users move messages to/from Junk

# Manual training via CLI
docker compose exec rspamd-mailcow rspamc learn_spam /path/to/spam/message
docker compose exec rspamd-mailcow rspamc learn_ham /path/to/ham/message

# Check Rspamd statistics
docker compose exec rspamd-mailcow rspamc stat

Backup Strategy

Email data is critical and often irreplaceable. Implement a comprehensive backup strategy:

#!/bin/bash
# mailcow-backup.sh
set -euo pipefail

BACKUP_DIR="/backups/mailcow/$(date +%Y%m%d_%H%M%S)"
MAILCOW_DIR="/opt/mailcow-dockerized"
RETENTION_DAYS=30

mkdir -p "$BACKUP_DIR"

# 1. Backup the MySQL database
docker compose -f "$MAILCOW_DIR/docker-compose.yml" exec -T mysql-mailcow \
  mysqldump --all-databases --single-transaction -u root -p"${DBROOT}" | \
  gzip > "$BACKUP_DIR/mysql_dump.sql.gz"

# 2. Backup mail data (vmail volume)
docker run --rm \
  -v mailcowdockerized_vmail-vol-1:/source:ro \
  -v "$BACKUP_DIR":/backup \
  alpine tar czf /backup/vmail.tar.gz -C /source .

# 3. Backup Rspamd data
docker run --rm \
  -v mailcowdockerized_rspamd-vol-1:/source:ro \
  -v "$BACKUP_DIR":/backup \
  alpine tar czf /backup/rspamd.tar.gz -C /source .

# 4. Backup configuration
tar czf "$BACKUP_DIR/mailcow_conf.tar.gz" \
  -C "$MAILCOW_DIR" mailcow.conf data/conf

# 5. Backup DKIM keys
tar czf "$BACKUP_DIR/dkim_keys.tar.gz" \
  -C "$MAILCOW_DIR" data/dkim

# 6. Clean old backups
find /backups/mailcow -maxdepth 1 -type d -mtime +$RETENTION_DAYS -exec rm -rf {} \;

echo "Backup complete: $BACKUP_DIR"
echo "Total size: $(du -sh $BACKUP_DIR | cut -f1)"

Alternatively, Mailcow includes a built-in backup script:

# Use Mailcow's built-in backup tool
cd /opt/mailcow-dockerized
./helper-scripts/backup_and_restore.sh backup all

# Restore from backup
./helper-scripts/backup_and_restore.sh restore

Monitoring and Maintenance

A mail server requires more active monitoring than most self-hosted services. Set up alerting for:

• Mail queue length: A growing queue indicates delivery problems
• Disk space: Mail storage can grow rapidly
• TLS certificate expiry: Expired certs break SMTP between servers
• DNSBL listing: Check regularly that your IP is not blacklisted
• Authentication failures: Brute force attempts against your SMTP/IMAP

# Check mail queue
docker compose exec postfix-mailcow postqueue -p

# Check for DNSBL listing
docker compose exec postfix-mailcow postconf -d | grep smtpd_client_restrictions

# Monitor container health
docker compose ps --format "table {{.Name}}\t{{.Status}}\t{{.Health}}"

# Check Rspamd throughput
docker compose exec rspamd-mailcow rspamc stat

# Test email deliverability (use external tools)
# - mail-tester.com (send a test email, get a score)
# - mxtoolbox.com/deliverability (check DNS and IP reputation)
# - learndmarc.com (verify DMARC alignment)

Automated Health Checks

#!/bin/bash
# mail-health-check.sh - Run via cron every 15 minutes
MAIL_HOST="mail.example.com"
ALERT_EMAIL="[email protected]"

# Check SMTP is responding
if ! echo "QUIT" | timeout 10 nc -w5 "$MAIL_HOST" 587 | grep -q "220"; then
  echo "SMTP not responding on $MAIL_HOST:587" | mail -s "ALERT: Mail server down" "$ALERT_EMAIL"
fi

# Check IMAP is responding
if ! echo "A001 LOGOUT" | timeout 10 openssl s_client -connect "$MAIL_HOST":993 -quiet 2>/dev/null | grep -q "OK"; then
  echo "IMAP not responding on $MAIL_HOST:993" | mail -s "ALERT: IMAP down" "$ALERT_EMAIL"
fi

# Check mail queue size
QUEUE_SIZE=$(docker compose -f /opt/mailcow-dockerized/docker-compose.yml exec -T postfix-mailcow mailq | tail -1 | grep -oP '\d+' | head -1)
if [ "${QUEUE_SIZE:-0}" -gt 100 ]; then
  echo "Mail queue has $QUEUE_SIZE messages" | mail -s "ALERT: Mail queue growing" "$ALERT_EMAIL"
fi

# Check disk usage
DISK_USAGE=$(df /var/lib/docker | tail -1 | awk '{print $5}' | tr -d '%')
if [ "$DISK_USAGE" -gt 85 ]; then
  echo "Disk usage at ${DISK_USAGE}%" | mail -s "ALERT: Disk space low" "$ALERT_EMAIL"
fi

Challenges and Caveats

Self-hosting email is not for everyone. These are the ongoing challenges you must be prepared for:

1. IP reputation is fragile: If a single account is compromised and sends spam, your IP gets blacklisted within hours. Delisting can take days or weeks.
2. Major providers are hostile to small mail servers: Gmail, Outlook, and Yahoo increasingly treat mail from small, unknown servers as suspicious, even with perfect DNS and DKIM. Warming up a new IP takes time.
3. Uptime expectations are high: Email is expected to work 24/7. Unlike a wiki or file server, downtime means missed messages and failed business communications.
4. Security is critical: An exploited mail server can be used for phishing, spam relay, or data exfiltration. Keep everything patched and monitored.
5. Storage grows indefinitely: Users rarely delete email. Plan for storage growth and implement quota policies.

Hardening Your Mail Server

Beyond the basic setup, apply these security measures:

# Fail2ban integration for brute force protection
# Mailcow includes Netfilter container that handles this automatically

# Check banned IPs
docker compose exec netfilter-mailcow fail2ban-client status postfix-sasl
docker compose exec netfilter-mailcow fail2ban-client status dovecot

# Configure rate limiting in mailcow.conf
MAILCOW_RATE_LIMIT=20      # Messages per hour per user
MAILCOW_RATE_LIMIT_INTERVAL=3600

# Enable two-factor authentication for admin panel
# Configure in Configuration > Access > Two-Factor Authentication

# Restrict admin panel access by IP (optional)
# Edit data/conf/nginx/site.custom.conf

Client Configuration

Mailcow supports autodiscovery, making client configuration straightforward. Add these DNS records for automatic configuration:

# Autodiscover (Outlook)
autodiscover.example.com.    IN    CNAME    mail.example.com.

# Autoconfig (Thunderbird)
autoconfig.example.com.      IN    CNAME    mail.example.com.

# SRV records for generic clients
_submission._tcp.example.com.    IN    SRV    0 1 587 mail.example.com.
_imaps._tcp.example.com.         IN    SRV    0 1 993 mail.example.com.
_pop3s._tcp.example.com.         IN    SRV    0 1 995 mail.example.com.

For manual configuration:

Summary

Running your own email server is one of the most rewarding but demanding self-hosting projects. Mailcow makes the Docker deployment side straightforward, but the real work is in DNS configuration, IP reputation management, and ongoing monitoring. Start with a test domain, send to various providers, monitor DMARC reports, and only migrate your primary email once you are confident in the setup.

For those managing multiple self-hosted services alongside email, platforms like usulnet help keep track of container health, resource usage, and backup status across your entire infrastructure, including the dozen-plus containers that make up a Mailcow deployment.