Most AI products are black boxes twice over. The model is a black box to the developer, and the product is a black box to the user. When I started building KaryoSpace, a real-time enterprise workspace where an AI can answer questions across email, messages, documents, and project history. I assumed the hard part would be the retrieval pipeline. I was wrong. The hard part was knowing whether the answers were any good, and knowing it before a user did.
This is about AIO, the AI Intelligence Observatory I built into KaryoSpace, and the specific things it taught me about how LLM systems fail in production. I built KaryoSpace solo, alongside a full-time job leading delivery for a 30+ person engineering organisation, using Claude Code as my primary development partner. That combination: one person, limited hours, a system with an LLM in the critical path. It forced a discipline I now think every AI product needs from day one: if you can't observe it, you can't trust it, and if you can't trust it, you can't ship it.
Standard application logging answers "did the request succeed?" For LLM-backed features, that question is almost useless. The request succeeds every time. The model always returns something. The failure is in the content, and content failures are invisible to conventional monitoring.
So AIO tracks a different set of signals across every AI interaction in the platform:
Signal capture happens at a single chokepoint: storeUserQuery() in the Go backend, called by every AI path regardless of outcome: guardrail intercept, RAG answer, cached response, Jira write intent. Each MongoDB document includes the query text, full response, UTC timestamp, a type field (guardrail / rag / insight / cta), scope (user-level or org-wide), an array of source collection names that contributed, the raw retrieved text chunks for grounding verification, a from_cache boolean, and a UUID trace ID. The AIO admin grid queries the user_queries collection filtered by org_id, with time-window filters (24h / 7d / 30d) and an "attention" flag that surfaces users with elevated suppression rates. No separate analytics pipeline. The same collection that serves the home page chat history is the source of truth for AIO.
These are the categories of failure that observability surfaced, roughly in order of how much they changed my thinking.
The most dangerous failure mode I encountered was not hallucination in the classic sense. It was the model producing a fluent, plausible answer grounded in retrieved but irrelevant content. The retrieval step returned something, the model dutifully used it, and the answer was wrong in a way that was very hard to spot without checking the sources.
The clearest example: a query about the Q2 cloud migration budget came back citing a specific figure and a specific approver. Both real names, wrong project. The retrieval step matched on "budget" and "cloud migration" and returned two email chunks from a decommissioned initiative 11 months prior. The model read those chunks, synthesised correctly from that context, and produced a structurally sound, factually wrong answer.
AIO showed the source as email_messages, two chunks, high text scores. Pulling the chunk content from the trace record immediately revealed the problem. The documents were genuine, the query matched, but there was no recency signal anywhere in the retrieval path. The fix was two-pronged: a default 90-day recency filter on email retrieval, and the current date injected into the system prompt so the model can flag when retrieved content seems temporally mismatched. Stale-source incidents dropped immediately after.
Retrieval quality failures masquerade as model failures. Without per-answer source attribution in AIO, I would have blamed the model every time, fixing the wrong component instead.
LLM systems rarely fall over. They erode. A prompt that worked well against one model version behaves subtly differently after an update; a retrieval strategy that worked at 1,000 documents behaves differently at 10,000.
In May, after a Groq model update, AIO's type-field distribution started shifting over three days: the ratio of guardrail records to rag_response records for queries containing "write me" or "can you generate" dropped by roughly 30%. The new model variant was handling those phrasings more "helpfully", attempting partial answers rather than declining cleanly. No user had reported a bypass. AIO surfaced it as a trend in the type-field counts over the 7-day window.
The fix was architectural: I moved codegen classification fully into the pre-LLM intent classifier, so enforcement happens before the model is invoked regardless of how the model would respond. Model-update drift can no longer affect guardrail enforcement.
Point-in-time evaluation tells you almost nothing. The valuable signal is the trend line, meaning you have to capture quality signals continuously, not just when you're debugging.
Running local (Ollama) and hosted (Groq) inference side by side taught me that model choice is not a one-time architecture decision. It's a per-query routing decision, and you can only make it well if you're measuring both sides honestly.
On the same Oracle Cloud ARM64 instance (4 OCPU, 24GB), Ollama running gemma3:1b shows median inference latency of 600–900ms for short responses. Groq's API with llama-3.3-70b-versatile runs at 1.8–2.5s end-to-end. But quality diverges sharply: gemma3:1b reliably anchors to the first retrieved document and ignores the rest when the context exceeds roughly 1,200 tokens. Groq maintains consistent cross-source synthesis even at 3,000 tokens of retrieved context.
| Scenario | Route to | Reason |
|---|---|---|
| Single-source lookup, query <15 words | Ollama (gemma3:1b) | Fast; one document; quality sufficient |
| Guardrail-declined response | Ollama (any) | Model not invoked; inference quality irrelevant |
| Multi-source synthesis | Groq (llama-3.3-70b) | Cross-document reasoning requires larger model |
| Multi-turn conversation with history | Groq (llama-3.3-70b) | Context window and coherence requirements |
AIO's per-trace latency breakdown, recording retrieval time and inference time separately, made this routing decision data-driven. Without it, I'd have been guessing at where the seconds were going.
During the adversarial testing phase, firing 9 categories of test queries before going live, query volume in AIO's admin grid spiked in recognisable bursts: clusters of guardrail-type traces, then silence, then another cluster. That pattern made testing sessions immediately visible in the dashboard.
The same visibility works in reverse. A week of unusually low trace volume from a previously active org is the first signal that something is wrong with retrieval quality, often before anyone thinks to file a report. The query frequency graph in AIO became one of the most reliable leading indicators I have for system health, precisely because users don't complain about bad AI answers. They just quietly stop trusting the system and route around it.
Three practices I now treat as non-negotiable for any LLM-backed feature:
Building AIO left me with more questions than answers, and they're the questions I most want to work on next: How do you evaluate answer quality automatically when there's no ground truth? How do you detect when users are adapting their queries to probe around a constraint, not just querying, but learning the system's shape? What does drift detection look like when the corpus, the model, and the users are all changing at once?
I don't have rigorous answers yet. What I have is a production system, real signals, and a strong conviction that the interesting problems in applied AI right now are less about making models more capable and more about making systems built on them legible.
KaryoSpace is live at karyospace.com. The platform is built in Go with MongoDB, Docker, and Okta SSO, developed end-to-end with Claude Code in the loop. If you're working on LLM observability or evaluation and want to compare notes: sumanakkisetty@gmail.com