Vector embeddings are the foundation of semantic search. Every email chunk, knowledge document, incident, and query in KaryoSpace gets embedded into a 768-dimensional float64 vector. At query time, the system embeds the question and finds documents by cosine similarity rather than keyword match. Here is how the embedding pipeline works, why we chose nomic-embed-text, and how cosine re-ranking improves result quality.
Why local embeddings
The first version of KaryoSpace used Gemini's embedding API. It worked well for low volume. At higher volume, two problems emerged: the free tier rate limit (1 request per second) created a backpressure problem during ingestion, and every embedding request sent content to an external API, which creates a data sovereignty concern for enterprise customers.
The move to local embeddings with Ollama and nomic-embed-text solved both problems. The model runs on the same Oracle VM as the app. No rate limit. No external API call. Content never leaves the server.
nomic-embed-text
nomic-embed-text is an Apache 2.0 licensed embedding model that produces 768-dimensional vectors. It is fast enough to run on CPU (the Oracle A1 Flex VM has no GPU), accurate enough for document retrieval, and small enough (~274 MB) to load alongside the Go app without memory pressure.
// backend/go/rag/embedder.go
type OllamaEmbedder struct {
BaseURL string // http://localhost:11434
Model string // nomic-embed-text
Client *http.Client
}
func (e *OllamaEmbedder) Embed(ctx context.Context, text string) ([]float64, error) {
truncated := truncateToWords(text, 380) // 512-token limit
body, _ := json.Marshal(map[string]string{
"model": e.Model,
"prompt": truncated,
})
req, _ := http.NewRequestWithContext(ctx, "POST",
e.BaseURL+"/api/embeddings", bytes.NewReader(body))
resp, err := e.Client.Do(req)
if err != nil { return nil, err }
var result struct{ Embedding []float64 `json:"embedding"` }
json.NewDecoder(resp.Body).Decode(&result)
return result.Embedding, nil
}
The 380-word truncation
nomic-embed-text has a 512-token context limit. Tokens and words are not the same thing, but 380 words is a conservative bound that stays safely under 512 tokens for English text. The original implementation truncated by character count (2,000 characters), which was too long for texts with many short words and occasionally too short for texts with long words. Word-count truncation is more predictable.
func truncateToWords(text string, maxWords int) string {
words := strings.Fields(text)
if len(words) <= maxWords { return text }
return strings.Join(words[:maxWords], " ")
}
Storing embeddings in MongoDB
Each embedded document stores the vector as a []float64 field in the same document as the content. MongoDB does not have native vector index support in the version KaryoSpace uses, so retrieval uses the text search index as a first pass and cosine similarity as a re-ranking second pass.
type KnowledgeChunk struct {
ID primitive.ObjectID `bson:"_id"`
DocID string `bson:"doc_id"`
OrgID string `bson:"org_id"`
ChunkIndex int `bson:"chunk_index"`
Text string `bson:"text"`
Embedding []float64 `bson:"embedding"` // 768 dimensions
CreatedAt time.Time `bson:"created_at"`
}
Cosine similarity re-ranking
The retrieval flow uses MongoDB's $text search to fetch a candidate pool (3x the desired result count, capped at 60), then re-ranks by cosine similarity between the query embedding and each candidate embedding. The top N results after re-ranking are returned to the caller.
func cosineSim(a, b []float64) float64 {
var dot, normA, normB float64
for i := range a {
dot += a[i] * b[i]
normA += a[i] * a[i]
normB += b[i] * b[i]
}
if normA == 0 || normB == 0 { return 0 }
return dot / (math.Sqrt(normA) * math.Sqrt(normB))
}
func SearchKnowledgeHybrid(ctx context.Context, query string, queryEmb []float64,
orgID string, limit int) ([]KnowledgeChunk, error) {
// Pass 1: text search for candidate pool
candidates, _ := textSearchKnowledge(ctx, query, orgID, limit*3)
// Pass 2: cosine re-rank
for i := range candidates {
candidates[i].Score = cosineSim(queryEmb, candidates[i].Embedding)
}
sort.Slice(candidates, func(i, j int) bool {
return candidates[i].Score > candidates[j].Score
})
if len(candidates) > limit { candidates = candidates[:limit] }
return candidates, nil
}
Model warm-up
The first embedding request after Ollama starts takes 2-4 seconds as the model loads into memory. Subsequent requests take 50-150ms. To eliminate this cold-start penalty on the first user query, the app embeds a dummy string 10 seconds after startup:
// In main.go, after MongoDB connects
go func() {
time.Sleep(10 * time.Second)
_, err := embedder.Embed(context.Background(), "warmup")
if err != nil {
slog.Warn("embedding warmup failed", "err", err)
}
}()
Embedding pipeline coverage
Five collections are embedded in KaryoSpace:
rag_email_chunks: email content, 256-word chunks, 50-word overlapknowledge_chunks: knowledge base docs, 500-word chunks, 50-word overlaprag_integration_chunks: Confluence, Jira, ServiceNow contentrag_query_cache: query embeddings for the KRE semantic cacherag_refined_knowledge: distilled KnowledgeAtom fact embeddings
Email and integration chunks are embedded at ingest time. Query cache entries are embedded at query time (the query embedding is already computed for retrieval, so storing it is free). KnowledgeAtom embeddings are computed by the background distillation worker at 2-second intervals to avoid saturating the Ollama process during peak hours.