Writing an SMTP + IMAP server from scratch in Go: the decisions I regret and the ones I don't

The question I got most often when I told people I was building a custom email server was: why? There are established open-source mail servers. Postfix for SMTP, Dovecot for IMAP, SpamAssassin for filtering. You could wire them together in a weekend. The answer is: because KaryoSpace's entire value proposition is data sovereignty. An enterprise that deploys KaryoSpace owns its data completely. Shipping a product that secretly depends on Gmail or Exchange would contradict that claim. So I built the email server myself.

This is the story of what that cost, what broke, and which decisions I'd make the same way again.

Architecture overview

The email stack in KaryoSpace is split across four Go files:

Everything persists to MongoDB. There is no local filesystem storage of messages.

The five-folder decision

Email clients expect a specific folder hierarchy. I settled on five: INBOX, Sent, Drafts, Trash, Spam. This is the minimum set that works correctly with Thunderbird, Apple Mail, and the KaryoSpace web client simultaneously. I considered supporting custom folders (Labels in Gmail terminology) but deferred it: custom folders require IMAP LIST to return dynamic content, which requires more session state management. Deferred means not yet, not never.

The folders are stored as a folder field on each message document in MongoDB. Moving a message means updating that field. EXPUNGE means soft-deleting (setting a deleted flag) and removing from the folder view. Physical deletion happens when the Trash folder is emptied.

DKIM signing

Every outbound message from KaryoSpace is DKIM-signed. The implementation is in backend/go/email/dkim_signer.go. The signing key is an RSA private key stored encrypted (AES-256) in MongoDB, decrypted at startup using the DKIM_ENCRYPTION_KEY environment variable.

The bug I spent the longest on: the key was being stored as raw bytes in MongoDB, but the signing code was calling base64.StdEncoding.DecodeString() expecting a base64-encoded string. The result was a signing failure every time. The fix was to store the key as a base64-encoded string at creation time and decode it at use time.

// Wrong: stored raw bytes, tried to decode as base64
privKeyBytes, err := base64.StdEncoding.DecodeString(string(storedKey))

// Correct: store as base64 string, decode correctly
privKeyBytes, err := base64.StdEncoding.DecodeString(storedKeyBase64String)

This caused DKIM=fail on every outbound message until I traced it through the Gmail postmaster tools. DKIM failures are silent from the sender's perspective, which made this particularly painful to debug.

SPF verification and the loopback bug

When an inbound SMTP connection arrives, the server checks the sending IP against the domain's SPF record. This works correctly for external senders. It did not work correctly for internal email at first.

The bug: when a KaryoSpace user sends an email to another user in the same org, the message routes through the local SMTP submission server and then loops back into the same server for delivery. The sending IP is 127.0.0.1. The SPF check for 127.0.0.1 fails (it's not in any domain's SPF record), which triggered the spam scorer and deposited internal messages into Spam instead of INBOX.

The fix is an isLoopback() check in smtp_handler.go that skips the SPF check and spam scoring for connections from 127.0.0.1:

func isLoopback(addr string) bool {
    host, _, _ := net.SplitHostPort(addr)
    ip := net.ParseIP(host)
    return ip != nil && ip.IsLoopback()
}

// In the delivery path:
if !isLoopback(remoteAddr) {
    spamScore = computeSpamScore(message, spfResult)
}

The outbound queue and backoff

Outbound delivery to external domains uses a queue-and-retry pattern. When a message fails delivery (temporary SMTP error, network timeout, TLS negotiation failure), it goes into email_outbound_queue in MongoDB with a next_attempt timestamp. The retry schedule uses exponential backoff:

Attempt	Delay	Total elapsed
1	5 min	5 min
2	30 min	35 min
3	2 hr	~2.5 hr
4	6 hr	~8.5 hr
5	24 hr	~32.5 hr
6	bounce	~32.5 hr

After 6 failed attempts, the queue worker generates a bounce NDR (non-delivery report) and sends it back to the original sender. The NDR is formatted as a standard DSN (Delivery Status Notification) MIME message so that email clients recognize it correctly.

Brevo (a transactional email relay service) is currently configured as the outbound relay via SMTP_RELAY_HOST. This adds DKIM alignment for Brevo's sending infrastructure on top of our own DKIM signature. Direct MX delivery (no relay) is the eventual target, but relaying through a reputable provider improves deliverability during the early months when the domain's reputation is being established.

The goroutine race

Early in development, outbound relay was handled inline in the SMTP transaction: the relayExternal() function was called directly during message acceptance. This meant the SMTP client was waiting while we tried to connect to the relay. If the relay was slow, SMTP clients would timeout. If the relay was down, the SMTP transaction would fail.

The fix: move relay to a standalone relayExternalAsync() goroutine. The SMTP server accepts the message, queues it, returns 250 OK immediately, and the goroutine handles delivery independently. The queue worker is then responsible for retry on failure. This decoupled SMTP acceptance latency (sub-second) from relay latency (variable, relay-dependent).

// Before: blocking, caused SMTP client timeouts
func (s *SMTPServer) deliverMessage(msg *Message) error {
    if isExternal(msg.To) {
        return s.relayExternal(msg) // blocks here
    }
    return s.storeMessage(msg)
}

// After: non-blocking
func (s *SMTPServer) deliverMessage(msg *Message) error {
    if isExternal(msg.To) {
        go relayExternalAsync(msg) // fire and forget; queue handles retry
        return nil
    }
    return s.storeMessage(msg)
}

The goroutine model in Go maps cleanly onto this pattern. Each inbound SMTP connection already runs in its own goroutine. Spawning a goroutine for outbound relay adds almost no overhead and eliminates the entire class of relay-induced timeout failures on the inbound side.

IMAP submission port and iptables

IMAP operates on port 993 (TLS) in production. SMTP submission operates on port 587 (STARTTLS). In development, these are mapped to higher ports (1993 and 2587) via iptables PREROUTING rules, because non-root processes can't bind to ports below 1024.

# Development port mapping (persisted via netfilter-persistent)
iptables -t nat -A PREROUTING -p tcp --dport 25 -j REDIRECT --to-port 2525
iptables -t nat -A PREROUTING -p tcp --dport 587 -j REDIRECT --to-port 2587
iptables -t nat -A PREROUTING -p tcp --dport 993 -j REDIRECT --to-port 1993

The Go process binds to the high ports; the kernel translates inbound connections transparently. This is the same pattern used by most production Go web servers (which bind to 8080 and let nginx or iptables handle 443).

Testing

The test suite has 239 PASS, 0 FAIL, 112 SKIP. The 112 SKIPs are integration tests that require a live MongoDB connection. They skip instantly when MongoDB is not available rather than hanging for 3 seconds each, thanks to a shared sync.Once availability check in testmain_test.go.

The hardest test to get right was TestIMAPServer_GracefulShutdown. The IMAP server needs to close all active client connections when it receives a stop signal. The race condition: if you call Stop() and then close the test client connection, the server might not have had time to process the close. The fix: close the client connection concurrently with Stop() in a separate goroutine, then wait for both to complete.

// Flaky: sequential close causes race with server shutdown
server.Stop()
clientConn.Close()

// Correct: concurrent, both wait on a WaitGroup
var wg sync.WaitGroup
wg.Add(2)
go func() { defer wg.Done(); server.Stop() }()
go func() { defer wg.Done(); clientConn.Close() }()
wg.Wait()

Real email clients (Thunderbird, Apple Mail) exposed quirks in the IMAP FETCH response format that no unit test had caught. Testing against real clients from day one would have surfaced those issues weeks earlier.

What I'd do differently

Three things:

1. Start with the outbound queue. I built inline delivery first and added the queue later. Every time I changed the delivery path, I had to refactor both. Queue-first would have been cleaner and the refactoring cost would have been zero.
2. Use a proper MIME parser library. I wrote custom MIME parsing for attachments. It works, but MIME is a specification that has been interpreted in many creative ways by email clients over 30 years. A library that handles the edge cases is worth the dependency. The time I spent on MIME edge cases could have gone elsewhere.
3. Test with real email clients from day one. I wrote unit tests first and only tested with Thunderbird and Apple Mail late in the process. Real email clients exposed quirks in my IMAP FETCH response format that all the unit tests had missed. Closing that feedback loop early would have saved significant debugging time.

What I wouldn't change: building it in Go was the right call. The goroutine model maps cleanly onto the SMTP and IMAP server architectures, which are inherently concurrent (one goroutine per connection). The performance is good. And the entire email stack is under 3,000 lines of Go, which is small enough to hold in your head.

---

KaryoSpace is live at karyospace.com. Questions about SMTP/IMAP implementation or enterprise email: sumanakkisetty@gmail.com