Building a RAG pipeline without frameworks: what I learned

The first question I had to answer when building the AI layer of KaryoSpace was: LangChain or not? The framework promised to handle chunking, embedding, retrieval, prompt construction, chain orchestration, and output parsing. It would have saved me weeks of initial implementation time.

I chose not to use it. This article explains why, what I built instead, what broke along the way, and what I think now, six months into running it in production.

Why I skipped the frameworks

My decision wasn't ideological. It was practical: KaryoSpace is an enterprise platform where data lives in multiple collections with different schemas, emails, chat threads, knowledge base documents, incident records, Jira issues, ServiceNow tickets. Each source has different relevance signals, different recency profiles, and different importance in the context of a workplace query.

A generalised RAG framework makes sensible default assumptions about what retrieval means. I needed to override almost every default: how chunking works differs by source type, how text scoring is weighted differs by collection, and which collections are searched depends on the classified intent of the query, not just keyword overlap. When I estimated how much I'd be fighting the framework versus building with it, I decided to build directly.

The other reason: I wanted to understand every part of the pipeline well enough to debug it. Frameworks abstract away failure modes. I needed to see them.

The pipeline architecture

The KaryoSpace RAG pipeline has three stages, implemented as three distinct packages in Go:

The retrieval approach: MongoDB $text search

I did not use vector embeddings. KaryoSpace uses MongoDB's built-in $text search, a full-text index with TF-IDF scoring, for all retrieval. This is the decision I get the most questions about, so let me explain it clearly.

Vector search (semantic similarity via embeddings) is better at finding conceptually related content when the query doesn't share vocabulary with the documents. For a general-purpose knowledge base, that matters. For an enterprise workplace assistant, it matters less than you might expect. When someone asks "what did Sarah say about the AWS migration deadline?", the relevant emails almost certainly contain the words "AWS", "migration", and "deadline". The vocabulary overlap is high by nature of how workplace communication works.

The tradeoff I accepted:

Capability	$text search	Vector search
Exact and near-exact keyword match	Excellent	Good
Conceptual / semantic similarity	Poor	Excellent
Deployment complexity	None (MongoDB built-in)	Embedding model, vector index, additional infra
Query latency	~30–80ms	~100–300ms (model + search)
Cost per query	Zero	Embedding API calls

For the current scope of KaryoSpace, workplace queries against a user's connected data, the tradeoff works. I will revisit vector search if query patterns start revealing semantic gaps that $text can't bridge.

The bug that took me longest to find

The MongoDB $meta textScore sort syntax has a non-obvious structure. My first implementation wrote the sort as:

bson.D{{Key: "$meta", Value: "textScore"}}

This causes a server-side error. MongoDB expects the sort value to be a document, not a string. The correct form is:

bson.D{{Key: "score", Value: bson.D{{Key: "$meta", Value: "textScore"}}}}

The error was silent in my initial implementation. It returned results sorted by insertion order rather than relevance, which looked correct for recent data but broke for older documents. AIO surfaced it as poor source attribution on older queries before I traced it to the sort bug. Three hours I won't get back.

Multi-LLM provider abstraction

I needed the ability to swap LLM providers without changing the application layer. The solution is a clean Go interface:

type LLMProvider interface {
    Complete(ctx context.Context, msgs []Message, cfg Config) (string, error)
}

Three implementations: Groq (llama-3.3-70b-versatile), Google Gemini (gemini-2.0-flash), and Ollama for local inference. Each handles its own message format differences internally . Gemini doesn't accept a system role and needs the system prompt prepended to the first user message; Ollama streams newline-delimited JSON that needs a custom parser. The application layer sees only LLMProvider.Complete().

Provider is set at startup via environment variable. Switching from Groq to Ollama in production is a config change and a restart.

Multi-turn conversation history

The first version of the pipeline was stateless: every query was answered in isolation. Follow-up questions like "what did they say about the deadline?", where "they" requires context from the previous exchange, returned useless results.

The fix: a per-user in-memory conversation history, a rolling window of 20 messages (10 exchanges). The history is prepended to every LLM call, giving the model the context needed for coherent multi-turn responses. History is reset on session end and not persisted. This is a deliberate privacy decision, not a limitation.

Source attribution: the feature that changed everything

Every response in KaryoSpace surfaces which data sources contributed to the answer: a row of source badges (Email, Chat, Knowledge Base, Jira, ServiceNow) visible below the response text. This was conceived as a UI feature, but it changed how I thought about the pipeline.

Once source attribution is a first-class output, you have to think about provenance at every stage. Which collection did this chunk come from? Was it the most relevant chunk, or just the highest text-scored one? Is the model using all the sources, or anchoring to the first one?

These questions led directly to the observability work in AIO. Source attribution is the user-visible output of retrieval quality. AIO is the developer-visible version of the same data. They're the same signal at different levels of the stack.

What I'd do differently

Three things:

1. Add query rewriting earlier. The retrieval quality for vague queries, "what's the status of that project?", is poor because $text search can't infer what "that project" refers to. A query rewriter that expands vague references using conversation history would improve retrieval significantly for follow-up queries.
2. Add hybrid search sooner. Pure $text works well enough for explicit keyword queries but struggles with paraphrased questions. A hybrid approach, $text for recall with a small local embedding model for reranking, would improve precision without the cost of full vector search.
3. Write the evaluation harness first. I built retrieval quality evaluation (via AIO) after the pipeline. It should have been first. You can't improve what you can't measure, and the measurement lag cost me weeks of debugging by intuition.

What the no-framework decision actually cost

Honesty: it cost me about three weeks of initial implementation time compared to using LangChain. What I got in return: I understand every part of the pipeline, I can instrument every stage independently, I have no dependency on a rapidly-changing third-party abstraction, and when something breaks I know exactly where to look.

For a solo-built production system where I'm also the debugger, the support team, and the on-call engineer. The latter set of properties is worth more than the three weeks.

---

KaryoSpace is live at karyospace.com. Questions about RAG pipeline design or enterprise AI systems: sumanakkisetty@gmail.com