KaryoSpace's homepage makes a specific promise. One AI that knows your entire company. I've spent months building the machinery behind that promise, an engine we call AIME, and most of that time the measurements kept telling me something I didn't want to hear. This post is the honest version of what they said, including the part where nearly everything I built measured out at zero.
The first golden set had 270 questions, generated automatically from the seeded mailbox. It felt rigorous. We pre-registered success gates before running experiments, froze the thresholds in writing, and refused to move them after seeing results. Then four workstreams in a row came back with nothing.
Hybrid retrieval fusion regressed the mean reciprocal rank from 0.768 to 0.744. Contextual chunk-prefixing regressed on every cell we measured, at both an 8B and a 70B model. LLM-based reranking failed its stability gate outright, scoring a Spearman rho of 0.6982 against a required 0.80, because asking a language model to score twenty candidates in one prompt gives different answers when you shuffle the order. We rebuilt that as a proper cross-encoder and it passed stability perfectly, then turned out to be unmeasurable anyway.
The reason was embarrassing once we finally looked. In 85.7% of the generated questions, the question text quoted the subject line of the email containing the answer. Retrieval had a 0% failure rate before we improved anything. Worst-case rank across all 270 items was 4. We'd spent months optimising a system that had no headroom to improve, and the discipline of the gates meant we at least didn't ship any of it. Four negative results on a set with no headroom isn't four failed experiments. It's one measurement problem, reported four times.
Auto-generating an eval means optimising for what's generatable, not what's real. The corpus was easy because easy questions are the ones a script can produce. Every hour of tuning after that was spent polishing a solved problem.
So we threw it away and wrote a new one by hand. Thirty email threads across five fictional projects, 82 messages, every body written individually. The threads do the things real project mail does. Dates slip three times, so the confident answer from April is wrong by July. A payment provider switches from Stripe to Adyen and then back, so a system that grabs the most recent-sounding change lands on the wrong answer. There's a ticket called ALF-87 and a closed one called ALF-870. There's a causal chain spanning three projects that no single thread contains.
Delivering that corpus through our own SMTP pipeline instead of inserting it into the database found two real production bugs on day one. The mail store was throwing away Message-ID, In-Reply-To and References headers and generating fresh ones, which silently collapsed threading to subject-line matching. It was also stamping every message with receipt time instead of the Date header, so any backfilled mailbox looked like it all arrived today. Both fixes shipped before the eval ever ran.
Then came the test that reframed everything. We gave a model the entire corpus in its context window, no retrieval at all, and asked it our twenty hardest questions. It scored 19 out of 20. Every superseded date, every trap, every multi-hop chain. At that corpus size, retrieval wasn't the bottleneck. The bottleneck was that nobody had asked what happens when the corpus doesn't fit.
To make retrieval fail honestly, we flooded the mailbox with a thousand hard negatives, junk that names the same projects and people and uses the same vocabulary without containing any answers. Retrieval recall at ten results collapsed from 87% to 56%. Finally, a number that could move.
What moved it wasn't clever. Our vector ranking scanned only the 500 most recent embedded messages, a constant chosen early and never revisited. Once a mailbox grows past 500 messages, older signal becomes invisible to semantic search entirely. Raising that cap to 2,000 lifted hybrid recall from 60% to 65% and multi-hop recall from 58% to 75%. That fix is live in production today, and it's worth being plain about what it is. It's an infrastructure bug fix. Not a ranking innovation, not a new architecture, just a limit that was too small.
The innovations, measured on the same corpus, did much less. The cross-encoder reranker improved mean reciprocal rank by 0.061 but cost two points of recall, because promoting the single best result pushes supporting material out of the window. A relationship graph over entities, the thing every retrieval roadmap says you need, changed nothing at the cutoff that matters and only helped at depths you could reach by just asking for more results. We built it to check, rather than assuming either way.
Every ranking improvement we measured helped most on the questions that were already nearly solved, and least on the ones that were actually failing. That pattern held across fusion, reranking and the graph. It was trying to tell us the remaining problems weren't ranking problems.
Here's the framing that finally made it click. Everything above is librarian work. You ask for a document, the system finds the document, and ours does that genuinely well now. But the questions that kept failing don't have a document containing the answer.
Ask "what's going on with Project Alpha" against twelve threads of real mail. The March email says launch is 15 June. April says 6 July. June says 20 July. The true answer, that it shipped on 22 July after two slips and a weekend hold, is written nowhere. A librarian hands you all three dates and wishes you luck. An analyst has been reading the mail as it arrived and already knows. Our synthesis questions scored 31% on retrieval recall and the temporal ones, questions like "what was the originally announced date", scored exactly 0%, because the original date is precisely the thing three newer emails contradict. No ranker can fix that. Relevance ranking actively prefers the confident stale answer.
The product promise was never "find me an email". It was "know my company". Those are different machines, and we'd only built the first one.
The analyst layer is almost offensively simple. As each message arrives, an observer updates a small living state document for every project the message mentions. Status, key dates kept as supersession chains rather than single values, decisions with their history, people and their moves, open questions. The Alpha document records its launch date as one line:
Launch: 15 June -> 6 July (OAuth scope) -> 20 July (security review) -> shipped 22 JulyThat one line answers the current-date question, the original-date question, and the why-did-it-slip question, all of which the retrieval stack fails. Answering "what's going on with Alpha" reads a 280-word document instead of searching 5,400 words of corpus, and links back to the source threads for verification.
Email, chat, knowledge docs and synced tickets all flow through one observation contract. If it enters the system, the analyst sees it.
One small model call per relevant message updates the project's state document. Old values are never deleted, they become chains, so the past stays answerable.
Fact lookups still go to search, which is good at them. Status, history and why questions read the state document instead.
We prototyped it in an evening and measured it the same way as everything else. On the ten questions the librarian architecture fails, the state documents scored 9 correct, 1 partial, 0 wrong, identical to giving the model the entire corpus, from about a quarter of the tokens. The scaling argument is the part I care about. State grows with the number of projects. The corpus grows with every message forever. Full context stops being possible; a 300-word state document doesn't.
The stress test that matters ran next. Production ingest doesn't see clean data, so we rebuilt the state documents from a stream where junk outnumbered signal five to one, 415 noise updates against 83 real ones, junk that names the same projects and talks about launch dates and rate limiters. The documents came out 52% fatter and visibly cluttered, carrying placeholder phase dates from a fake planning email and a pile of administrative reminders nobody needs. They also scored 8 out of 10 against the clean run's 9, with nothing marked wrong.
The split in that result is the useful part. Every supersession chain survived untouched, five out of five, and so did the temporal question, because a date chain is a structured field and junk mail doesn't produce competing chains. What degraded was the free-text status line, where a fake weekly update saying engineering is steady and QA is in progress has exactly the shape of a real one, so it blends in and pushes out the sentence that mattered. Noise dilutes the prose and leaves the structure alone, which says the fix is more structure rather than better filtering.
If you're building retrieval over workplace data, the honest sequence I'd recommend from this experience looks like this:
Write your eval corpus by hand, because generated ones test what generators produce. Run the no-retrieval baseline first, because if full context answers everything, your corpus can't measure retrieval at all. Check your infrastructure constants before your architecture, because our biggest shipped win was a scan limit. And when a class of question keeps failing every ranking improvement you throw at it, stop ranking. The answer probably isn't in any document. You need the thing that was reading all along.
KaryoSpace is live at karyospace.com. Questions: sumanakkisetty@gmail.com
Thanks for reaching out. We'll get back to you shortly.