Building incident management with SLA enforcement in Go: priority routing, breach detection, and notification fanout

SLA management sounds like a spreadsheet problem until you have 50 open incidents, four priority levels, and two engineers on call. At that point, the question "which of these is about to breach?" cannot be answered by looking at a list. You need the system to tell you, before it happens, with context attached.

KaryoSpace's incident management module lives in backend/go/incident/. The SLA worker is the part I am most proud of: a background goroutine that runs every five minutes, finds breaches, marks them, and fires notifications with zero manual intervention.

The SLA worker

func StartSLAWorker(ctx context.Context, store IncidentStore, notifier *Notifier) {
    ticker := time.NewTicker(5 * time.Minute)
    log.Println("✅ SLA worker started (5-minute tick)")
    go func() {
        defer ticker.Stop()
        checkSLABreaches(store, notifier)  // run immediately on start
        for {
            select {
            case <-ticker.C:
                checkSLABreaches(store, notifier)
            case <-ctx.Done():
                log.Println("SLA worker stopped")
                return
            }
        }
    }()
}

The design choices here are deliberate:

• Run immediately on start: if the server was down during a breach window, the worker catches it the moment it comes back up rather than waiting up to five minutes.
• 5-minute tick: SLA targets are in hours. A 5-minute detection window is well within acceptable granularity and keeps MongoDB query load low.
• Cancellable context: the worker exits cleanly when the server shuts down. No goroutine leaks.

Finding breaches

func checkSLABreaches(store IncidentStore, notifier *Notifier) {
    now := time.Now().UTC()
    incidents, err := store.ListSLABreaching(now)
    if err != nil {
        log.Printf("SLA worker query error: %v", err)
        return
    }
    for _, inc := range incidents {
        if err := store.SetSLABreached(inc.ID, now); err != nil {
            log.Printf("SLA worker: SetSLABreached failed for %s: %v", inc.TicketID, err)
            continue
        }
        inc.SLABreached = true
        inc.SLABreachedAt = &now
        _ = store.AddUpdate(&IncidentUpdate{
            IncidentID: inc.ID,
            TicketID:   inc.TicketID,
            AuthorID:   "system",
            AuthorName: "System",
            Type:       UpdateTypeStatusChange,
            Content:    fmt.Sprintf("⚠ SLA breached — %s priority target exceeded", string(inc.Priority)),
            CreatedAt:  now,
        })
        if notifier != nil {
            go notifier.NotifySLABreach(inc)
        }
        log.Printf("SLA breach: %s (%s, %s)", inc.TicketID, inc.Priority, inc.Department)
    }
}

ListSLABreaching(now) runs a MongoDB query that finds incidents where the calculated breach time has passed but the incident is not yet marked as breached. The breach time is created_at + sla_target_for_priority. This is computed in the store layer, not in Go, so the query is index-efficient.

Each breach gets an immutable audit record added via AddUpdate. This is critical for compliance: every SLA breach is permanently recorded with the exact timestamp and the system actor that detected it.

Priority tiers and SLA targets

Priority	SLA Target	First Notified	ServiceNow Sync
Critical	1 hour	Assignee + Reporter + Team Lead	Yes, immediate
High	4 hours	Assignee + Reporter	Yes, on create
Medium	8 hours	Assignee	Optional
Low	24 hours	Assignee	No

Priority is set at incident creation and affects every subsequent system behavior: which MongoDB index the breach query hits, how many people are notified, and whether a ServiceNow ticket is created automatically.

The incident model

type Incident struct {
    ID            primitive.ObjectID `bson:"_id"`
    TicketID      string             `bson:"ticket_id"`      // e.g. INC-2026001
    OrgID         string             `bson:"org_id"`
    Department    string             `bson:"department"`
    Priority      Priority           `bson:"priority"`       // Critical/High/Medium/Low
    Status        Status             `bson:"status"`         // Open/InProgress/Resolved/Closed
    AssigneeID    string             `bson:"assignee_id"`
    ReporterID    string             `bson:"reporter_id"`
    SLABreached   bool               `bson:"sla_breached"`
    SLABreachedAt *time.Time         `bson:"sla_breached_at,omitempty"`
    CreatedAt     time.Time          `bson:"created_at"`
}

The TicketID is generated as INC-{YYYY}{sequential_number}. It is human-readable and sortable, making it useful in email subjects, Slack messages, and ServiceNow cross-references without requiring an ID lookup.

Notification fanout

The first version of NotifySLABreach notified only the assignee. This turned out to be wrong in practice: the reporter, who raised the incident, also needs to know it has breached. They are the ones who made the commitment to the customer and need to communicate status.

func (n *Notifier) NotifySLABreach(inc Incident) {
    // Notify assignee
    n.sendBell(inc.AssigneeID, BellNotification{
        Title: fmt.Sprintf("SLA breached: %s", inc.TicketID),
        Body:  fmt.Sprintf("%s priority — target exceeded", inc.Priority),
        URL:   "/incidents/" + inc.TicketID,
    })
    // Also notify reporter if different from assignee
    if inc.ReporterID != "" && inc.ReporterID != inc.AssigneeID {
        n.sendBell(inc.ReporterID, BellNotification{
            Title: fmt.Sprintf("SLA breached: %s", inc.TicketID),
            Body:  "An incident you reported has exceeded its SLA target.",
            URL:   "/incidents/" + inc.TicketID,
        })
    }
}

Department routing

The department router (department_router.go) maps incident types to handling teams. Critical incidents surface in the home feed for all team leads in the relevant department, not just the assignee. This uses the same insightChip system as the rest of the home dashboard:

sources := []insightChip{
    {Label: "Incidents", Icon: "incident", URL: "/incidents?status=open"},
    {Label: "ServiceNow", Icon: "servicenow", URL: serviceNowURL},
}
actions := []insightChip{
    {Label: "Email team", Icon: "email",
     URL: "/email?compose=1&subject=" + urlQuoteSafe("Re: open incidents")},
    {Label: "War room chat", Icon: "chat", URL: "/messaging"},
}

These chips appear on the home dashboard when there are open incidents. The "Email team" action pre-populates the compose subject line with "Re: open incidents" so the message is immediately identifiable in thread views.

Integration with ServiceNow

Critical and High priority incidents can be mirrored to ServiceNow via ServiceNowCreateIncident() in backend/go/integrations/servicenow_handlers.go. The internal ticket ID (INC-2026001) and the ServiceNow sys_id are both stored on the incident so either system can cross-reference the other.

What I got wrong first

The first SLA check ran on every incident list request. With 50+ open incidents and multiple engineers refreshing the dashboard during an outage, this meant the breach detection query was running on every page load, under exactly the conditions when the database is already under stress. The fix was obvious in retrospect: move breach detection to a background worker, never to a request handler.

The second mistake was notification-only-to-assignee, which I described earlier. The third was ticket ID format: the first version generated INC-{ObjectID.Hex()}, which is a 24-character hex string. Nobody wants to say "INC-507f1f77bcf86cd799439011" in a Slack message. The sequential human-readable format came from the first real incident war room.

---

The SLA worker is 60 lines of Go. The entire incident management package is under 1000 lines including tests. Keeping scope narrow and making each component testable independently (IncidentStore interface, Notifier struct, isolated department router) made it possible to iterate quickly on the parts that turned out to be wrong without touching the rest.