Every integration library makes the same promise: connect to Slack in three lines of code, pull Jira issues without reading the API docs, sync Gmail without touching OAuth. I did not use any of them. KaryoSpace connects to six enterprise tools using raw HTTP clients and a thin interface layer I own completely. This article explains what that architecture looks like, why I made that call, and what the tradeoffs cost me in practice.
The six integrations are Gmail, Microsoft Outlook, Slack, Jira, Confluence, and ServiceNow. Each one is a different company, a different auth model, a different data format, and a different failure mode. The thing they share is a single Go interface: DataSource.
All integration logic lives in backend/go/integrations/. The package is built around four files that handle the coordination layer, and then individual files for each connected service.
The core abstraction is datasource.go, which defines the DataSource interface:
type DataSource interface {
Fetch(ctx context.Context, orgID string) ([]RawDocument, error)
}
Every integration implements exactly this. The Registry in registry.go maps string source IDs to their implementations:
// registry.go
var sources = map[string]DataSource{
"gmail": &GmailSource{},
"outlook": &OutlookSource{},
"slack": &SlackSource{},
"jira": &JiraSource{},
"confluence": &ConfluenceSource{},
"servicenow": &ServiceNowSource{},
"github_mcp": &GitHubMCPSource{},
"slack_mcp": &SlackMCPSource{},
}
Adding a new integration means implementing Fetch() and registering a key. The sync infrastructure, scheduling, error handling, vectorization, picks it up automatically.
The application layer never calls an integration directly. It only calls Registry.Get(sourceID) and then Fetch(). This means every integration is substitutable, testable in isolation, and can fail without affecting the others.
sync_worker.go is where the scheduling happens. For each registered source, the SyncWorker starts a goroutine with a ticker set to a configurable poll interval. The default is 15 minutes:
const defaultPollInterval = 15 * time.Minute
func (w *SyncWorker) startSourceLoop(sourceID string, ds DataSource) {
interval := w.GetSourceInterval(sourceID)
ticker := time.NewTicker(interval)
for {
select {
case <-ticker.C:
docs, err := ds.Fetch(w.ctx, w.orgID)
if err != nil {
log.Printf("sync error [%s]: %v", sourceID, err)
continue
}
w.vectorize(sourceID, docs)
case <-w.ctx.Done():
ticker.Stop()
return
}
}
}
Two things worth noting here. First, sync errors are logged and the loop continues. One integration returning a 503 does not stop Slack from syncing on the next tick. Second, the ctx.Done() path provides clean shutdown, which matters during deploys: the worker drains gracefully before the process exits.
Per-source intervals can be overridden at runtime. SetSourceInterval() persists the new value to the integration_sync_config MongoDB collection and restarts the goroutine with the updated ticker. The admin panel exposes GetLastTick() and GetSourceInterval() for visibility into what each source is doing and when it last ran.
vectorizer.go takes the []RawDocument output of each Fetch() call and converts it into chunks suitable for the RAG retriever. The core struct is IntegrationChunk:
type IntegrationChunk struct {
SourceID string
DocID string
OrgID string
ChunkIndex int
Text string
Embedding []float64
}
Chunking uses a sliding window of 512 words with a 50-word overlap. The stride is 462 words (integrationChunkWords = 512, integrationChunkStride = 462), which means consecutive chunks share about 10% of their content. This overlap prevents relevant sentences from being split across chunk boundaries and lost during retrieval.
One feature specific to ticket-based sources (Jira, ServiceNow): a regex called ticketIDRe pins chunks that mention a ticket reference. The pattern is (?i)\b([A-Z]{2,10})[-\s](\d+)\b, which matches "SCRUM-7", "INC-123", and even the looser "scrum 7". When a chunk is pinned, it gets a higher priority score in the retriever. This means asking "what's the status of SCRUM-7?" reliably surfaces the right chunk even when SCRUM-7 was mentioned only once in passing.
All chunks are stored in the rag_integration_chunks MongoDB collection and feed directly into the RAG retriever alongside email chunks from the main pipeline.
A Confluence page can be 8,000 words. A ServiceNow incident description can be 50. If you store the full document as one retrieval unit, a 50-word query returns the entire Confluence page as context, overwhelming the LLM with noise. Chunking with overlap gives the retriever surgical precision: return only the 512 words closest to what the user asked about.
| Integration | Auth method | MongoDB collection | Poll interval |
|---|---|---|---|
| Gmail | OAuth2 (Google) | rag_integration_chunks | 15 min (default) |
| Microsoft Outlook | OAuth2 (Microsoft) | rag_integration_chunks | 15 min (default) |
| Slack | Bot token | rag_integration_chunks | 15 min (default) |
| Jira | API key (REST v3) | rag_integration_chunks | 15 min (default) |
| Confluence | API key (REST) | rag_integration_chunks | 15 min (default) |
| ServiceNow | Basic auth (REST) | rag_integration_chunks | 15 min (default) |
Gmail uses golang.org/x/oauth2/google and Outlook uses golang.org/x/oauth2/microsoft. These are the standard Go OAuth2 helpers, not vendor SDKs. The credential vars are GMAIL_CLIENT_ID, GMAIL_CLIENT_SECRET, OUTLOOK_CLIENT_ID, and OUTLOOK_CLIENT_SECRET.
One decision that took me longer than expected to get right: credentials are read at call time, not at struct initialization. This sounds like a minor implementation detail but it matters in practice. KaryoSpace loads integration configs from MongoDB at startup via applyIntegrationConfigFromDB(), which calls os.Setenv() to populate the environment. If credentials were read once during struct creation (the obvious approach), they'd be empty because the env vars aren't set yet at init time. Reading them in Fetch() means the integration always sees the current state of the environment, regardless of when the struct was created.
Token refresh uses a callback pattern. GetTokensForUser(userID, provider) retrieves a stored OAuth2 token from MongoDB. The oauth2.TokenSource wraps it with automatic refresh, and a SaveToken() callback persists the refreshed token back to MongoDB when the access token rotates. Users never re-authenticate unless the refresh token itself expires or is revoked.
Calendar sync is handled by ExternalCalendarSync.PushEvent(), which fires async goroutines for pushGoogle and pushOutlook in parallel. Errors from these goroutines are logged but never propagated. Calendar push is best-effort: if Google Calendar returns a 503, the event is logged as failed and the application continues. A failed calendar push should never surface as an error to the user who just created a meeting in KaryoSpace.
Slack has two integration paths in the codebase: slack_direct.go and slack_mcp.go. The direct path uses a bot token to fetch channel message history via the Slack Web API. Channel IDs are resolved per-org and messages are fetched in time-ordered batches, deduped by timestamp before vectorization.
slack_handlers.go handles the webhook receiver for real-time events (the Slack Events API posts to a registered endpoint). The MCP bridge in slack_mcp.go wraps Slack as an MCP tool, which means AI queries that need to send a Slack message, look up a channel, or post a status update can call Slack as a tool call rather than through a pre-scheduled sync. The two paths are complementary: the sync worker keeps the RAG index current, and the MCP bridge handles interactive, action-oriented queries.
The Jira integration talks to the Jira REST API v3. jira.go handles the core sync: fetching issues from configured projects, extracting summary, description, status, assignee, and labels into RawDocument fields. jira_actions.go provides the action handlers that the home slash command uses.
Two endpoints are wired up for interactive use: GET /integrations/jira/projects returns the list of Jira projects visible to the configured API key, used to populate dropdowns in the UI. POST /integrations/jira/create calls JiraCreateIssue() to create a new issue from a home command. The flow is: user types a natural-language request in the AI assistant, the assistant extracts the issue summary, priority, and project key, and posts to /integrations/jira/create. Jira confirmation (issue key and URL) comes back in the AI response.
A user asks the AI: "Create a high-priority Jira ticket in SCRUM for the login timeout bug." The assistant parses the project key, summary, and priority, calls JiraCreateIssue(), and returns "Created SCRUM-412: Login timeout bug (High)." No browser tab, no Jira UI, no copy-paste.
confluence.go fetches pages from configured Confluence spaces using the Confluence REST API. Each page's full body HTML is extracted and converted to plain text before being passed to the vectorizer. The extraction strips macro output, navigation elements, and metadata tables that Confluence embeds in page HTML but that carry no semantic value for retrieval. What remains is the authored content: the actual text a human wrote.
Confluence pages tend to be long. The 512-word chunk window was sized partly with Confluence in mind. A typical Confluence runbook is 2,000 to 4,000 words. Without chunking, the RAG retriever would return the entire runbook as context for a query about one specific step. With 512-word chunks and ticket ID pinning disabled (Confluence doesn't have ticket IDs), the retriever surfaces the chunk that contains the relevant procedure, not the whole document.
servicenow.go syncs two record types: incidents and change requests. Both are fetched via the ServiceNow Table API with basic auth. Fields mapped to RawDocument include number, short description, description, state, priority, and assignment group. These fields are what users are most likely to reference in natural language: "What's the status of INC-4501?" or "Who's handling the P1 incidents this week?"
ServiceNowCreateIncident() is the action handler for home slash commands, parallel to Jira's create handler. It accepts structured input from the AI assistant and posts a new incident to ServiceNow via REST. Return value includes the incident number and URL, which the assistant surfaces to the user.
Every integration uses raw net/http clients with no vendor SDK. The decision is partly pragmatic (fewer dependencies to update, no abstraction layers to fight) and partly security-driven. All outbound HTTP calls go through ssrfguard, a wrapper that checks the target URL against a denylist of private IP ranges before making the request.
SSRF (Server-Side Request Forgery) is a real risk in an integration-heavy application. If user-supplied config can influence where the integration makes HTTP calls, a malicious config could redirect a request to an internal service at 169.254.x.x or 10.x.x.x. The ssrfguard wrapper blocks this at the HTTP client layer, before any request leaves the process. Using raw HTTP clients makes it straightforward to apply this guard uniformly. SDK-wrapped clients are harder to intercept.
// All integration HTTP calls go through this client client := ssrfguard.NewClient() resp, err := client.Get(apiURL)
All integration credentials, API keys, OAuth client IDs, bot tokens, are stored in the integration_configs MongoDB collection. At startup, applyIntegrationConfigFromDB() reads this collection and calls os.Setenv() for each key-value pair. This means integrations read credentials from the environment, but the source of truth is MongoDB, not a static config file.
The pattern lets admins update credentials through the KaryoSpace admin panel without touching environment files or redeploying. The admin sets a new Jira API key in the UI, it's written to integration_configs, and on the next restart (or explicit reload), the new key is live. There's no need to SSH into the server and edit a .env file.
Each sync goroutine is independent. If the ServiceNow REST API is unreachable, that goroutine logs the error and waits for the next tick. The Jira goroutine, the Confluence goroutine, and the Slack goroutine continue on their own schedules. There's no shared error state, no circuit breaker at the registry level, and no retry queue (though per-source retry with backoff is on the roadmap).
This isolation extends to the vectorization step. If vectorization fails for a specific document (malformed text, unexpected encoding), that document is skipped and logged. The other documents from the same Fetch() call proceed to storage. A single bad Confluence page does not block the rest of the space from being indexed.
DataSource.Fetch(ctx, orgID)Each integration's Fetch() implementation is called on the ticker interval. Returns []RawDocument with source-specific fields mapped to a common struct. Errors logged, loop continues.
vectorizer.goEach RawDocument is split into 512-word chunks with 50-word overlap. Ticket IDs matched by ticketIDRe receive a priority boost. Each chunk becomes an IntegrationChunk with SourceID, DocID, OrgID, ChunkIndex, and Text.
rag_integration_chunksChunks are upserted to MongoDB by (SourceID, DocID, ChunkIndex) composite key. Existing chunks for a document are replaced on each sync, not appended. This keeps the index current without unbounded growth.
The RAG retriever in rag/retriever.go queries rag_integration_chunks alongside email chunks. Results are ranked by MongoDB $text score, deduped by doc ID, and assembled into the context block. Source attribution badges in the UI show which integrations contributed.
Three things, in order of how much they'd have changed my workload:
About two weeks of initial implementation time, spread across learning each API's quirks directly rather than via a library that had already solved them. What I got in return: I own every HTTP call, I can add ssrfguard uniformly, I understand every auth flow from first principles, and when an API changes (Jira REST v3 introduced breaking changes to field names in early 2026), I know exactly which file to update and why.
More practically: vendor SDKs for enterprise tools like ServiceNow and Jira often trail the API by months. The Go ecosystem's coverage is thinner than Python's for these services. Using raw clients meant I was never blocked waiting for an SDK update that might never come.
The DataSource interface is thin enough that any future integration can be added without touching existing code. That's the real payoff: the architecture stays clean as the integration list grows.
KaryoSpace is live at karyospace.com. Questions about integration architecture or enterprise data pipelines: sumanakkisetty@gmail.com