When I first built the knowledge base, I split documents into 3000-character chunks. The chunks crossed sentence boundaries, broke mid-paragraph, and produced retrieval results that confused the LLM because critical context was always just outside the returned window. The fix was to switch to word-count chunking with overlap.
This article covers the knowledge base in backend/go/knowledge/: the chunking algorithm, PDF text extraction, store design, and the visibility model that separates personal notes from org-wide documents.
const (
ChunkWords = 500 // target words per chunk
OverlapWords = 50 // overlap between consecutive chunks
MaxDocChars = 50000 // hard cap on input text
)
func SplitIntoChunks(text string) []string {
words := strings.Fields(text)
if len(words) == 0 { return []string{} }
if len(words) <= ChunkWords { return []string{strings.Join(words, " ")} }
stride := ChunkWords - OverlapWords // = 450
var result []string
for i := 0; i < len(words); i += stride {
end := i + ChunkWords
if end > len(words) { end = len(words) }
result = append(result, strings.Join(words[i:end], " "))
if end == len(words) { break }
}
return result
}Why word counts instead of character counts: words map cleanly to tokens. A 500-word chunk is roughly 650-700 tokens, well within context window limits and MongoDB text index efficiency. Character-based chunking breaks mid-word at the boundary, corrupting the last token of every chunk.
Why 50-word overlap: a sentence that spans a chunk boundary would be split in half without overlap. The 50-word overlap ensures both chunks contain the complete sentence, so a search for a concept at the boundary can hit either chunk. Retrieval quality improved measurably after adding this.
Why MaxDocChars = 50000: without a cap, a user uploading a 500-page PDF would block the chunker goroutine for several seconds. The 50k char limit corresponds to roughly 35-40 pages, which covers virtually all realistic knowledge documents.
func ExtractText(data []byte, mimeType string) (string, error) {
switch mimeType {
case "text/plain", "text/markdown", "text/x-markdown":
return cleanText(string(data)), nil
case "application/pdf":
return extractPDFText(data)
default:
return "", fmt.Errorf("unsupported file type: %s", mimeType)
}
}
func extractPDFText(data []byte) (string, error) {
r, err := pdf.NewReader(bytes.NewReader(data), int64(len(data)))
// iterate: r.Page(i).GetPlainText(nil)
// fallback: raw-byte scan if text layer is empty
// error if < 100 chars extracted (image-only PDF)
}The library used is github.com/ledongthuc/pdf. Page iteration includes a nil guard: if page.V.IsNull() { continue }. Without this, scanned PDFs with blank pages panic the reader. The 100-character minimum check surfaces a clear error to the uploader: "This PDF has no extractable text layer."
func (s *KnowledgeStore) ensureIndexes() error {
_, err := s.coll.Indexes().CreateMany(ctx, []mongo.IndexModel{
// Fast sort: user's documents, pinned first, newest last
{Keys: bson.D{
{Key: "user_id", Value: 1},
{Key: "org_id", Value: 1},
{Key: "visibility", Value: 1},
{Key: "created_at", Value: -1},
}},
// Tag filter: user_id + tags array
{Keys: bson.D{{Key: "user_id", Value: 1}, {Key: "tags", Value: 1}}},
// Full-text search scoped to user
{Keys: bson.D{{Key: "user_id", Value: 1}, {Key: "text", Value: "text"}}},
})
return err
}The text index is scoped to user_id. This means full-text search only returns results for the querying user. Organization-wide documents require a separate search with org_id scope.
Every knowledge document has a visibility field with three values:
| Visibility | Who can read | RAG scope |
|---|---|---|
| personal | Owner only | User-scoped queries only |
| org | All org members | All org members' queries |
| department | Same department | Department-filtered queries |
The RAG retriever enforces visibility at query time, not at upload time. When a user asks the AI a question, the retriever builds a query with the appropriate visibility filter before searching. Personal documents never appear in another user's search results, even if they search for the exact title.
Each edit to a knowledge document creates a new version record:
func (s *KnowledgeStore) Update(id, userID, text string, tags []string) error {
// Save current version to knowledge_versions before overwriting
_, _ = s.versionColl.InsertOne(ctx, VersionRecord{
DocID: id,
Text: currentText,
Version: currentVersion,
SavedAt: time.Now(),
})
// Then update the main document
_, err = s.coll.UpdateOne(ctx, filter, bson.M{"$set": bson.M{
"text": text,
"tags": tags,
"version": currentVersion + 1,
}})
return err
}Users can restore any previous version from the document history view. The version trail also allows the AI to answer questions like "what did this document say before the last edit?"
The first implementation stored tags as nil when the user submitted an empty tag list. MongoDB stored this as null, which means array operators like $in: [] matched nothing and tag-filter queries broke silently. The fix was a one-liner: initialize tags to []string{} before inserting, so MongoDB always stores an empty array, never null.
The first PDF extractor did not handle image-only PDFs. When a scanned document produced zero extracted text, the chunker stored empty strings. Those empty chunks matched every query and polluted retrieval results. Adding the 100-character minimum check and returning a clear error to the user fixed both the symptom and the root cause.
The knowledge base is the least glamorous part of the platform and one of the most used. Documents upload, chunk, and index in under two seconds for typical files. The chunking algorithm is 30 lines of Go with no external dependencies. Getting the fundamentals right here is what makes the RAG pipeline useful.