Multi-LLM routing in production: when local inference is good enough and when it isn't

When I started building KaryoSpace's AI layer, I made a deliberate choice to not commit to a single model provider. The AI landscape changes fast enough that what's best today may not be best in six months. What I needed was the ability to swap providers without rewriting the application. What I ended up with is a routing layer that makes per-query decisions about which model to use based on query type, latency requirements, and cost constraints.

The interface

The entire LLM abstraction is a single Go interface in backend/go/llm/provider.go:

type LLMProvider interface {
    Complete(ctx context.Context, msgs []Message, cfg Config) (string, error)
}

type Message struct {
    Role    string // "system", "user", "assistant"
    Content string
}

type Config struct {
    MaxTokens   int
    Temperature float64
    Model       string // overrides provider default if set
}

The application layer only ever calls LLMProvider.Complete(). It doesn't know which provider is behind it. Provider selection happens in backend/go/llm/registry.go via NewProvider(name string) LLMProvider, which reads the LLM_PROVIDER environment variable at startup.

Three implementations exist:

• llm/groq.go: Groq API, default model llama-3.3-70b-versatile
• llm/gemini.go: Google Gemini API, default model gemini-2.0-flash
• llm/ollama.go: Local Ollama inference, default model qwen2.5:7b

What each provider does differently

The interface hides three very different APIs.

Groq is the straightforward one. It speaks the OpenAI-compatible chat completions format. System messages work as expected. JSON parsing is clean. The only edge case: Groq's rate limits are per-minute token limits, not per-request limits, so a burst of long-context queries can hit the limit unexpectedly.

Gemini has a quirk: it does not accept a role: "system" message. The Gemini API expects all messages to be either user or model. The gemini.go implementation handles this by prepending the system prompt to the content of the first user message:

// Gemini doesn't support system role
// Prepend system content to first user message instead
if msgs[0].Role == "system" {
    systemContent := msgs[0].Content
    msgs = msgs[1:]
    if len(msgs) > 0 && msgs[0].Role == "user" {
        msgs[0].Content = systemContent + "\n\n" + msgs[0].Content
    }
}

This is invisible to the application layer. callAIService() in main.go always passes a system message first. Gemini's provider implementation handles the transformation.

Ollama is the most different. It runs locally on the karyospace-prod server, an Oracle ARM64 instance with 24GB RAM. The API is HTTP but the response format for streaming is newline-delimited JSON, not the SSE format that Groq and Gemini use:

{"model":"qwen2.5:7b","created_at":"...","message":{"role":"assistant","content":"The "},"done":false}
{"model":"qwen2.5:7b","created_at":"...","message":{"role":"assistant","content":"answer "},"done":false}
{"model":"qwen2.5:7b","created_at":"...","message":{"role":"assistant","content":"is..."},"done":true}

The ollama.go parser reads the response body line by line, decodes each JSON object, and concatenates the message.content fields. The done: true line signals completion.

The routing logic

KaryoSpace's routing does not send every query to the LLM. The callAIService() function in main.go applies three filters before invoking the provider:

The full routing table:

Query type	Retrieval	Model call	Response path
Statement / acknowledgment	No	No	Canned response
Guardrail violation	No	No	Rejection message
RAG query (mentions people, projects, emails, deadlines)	Yes	Yes (with context)	RAG response
General question	No	Yes	Direct LLM response

Latency numbers

Real production numbers from karyospace-prod:

Provider	Typical latency	Token limit	Notes
Groq (llama-3.3-70b)	1.8 to 2.5 seconds	3000 tokens	Consistent at scale
Gemini Flash	1.5 to 2.2 seconds	2000 tokens	Occasionally spiky
Ollama (qwen2.5:7b, local)	600 to 900ms	1200 tokens	Breaks on longer context
Canned response (no LLM)	Under 5ms	N/A	Statements, guardrail hits

The Ollama latency is deceptively attractive. 600ms vs 2 seconds sounds like a clear win. But the context limit is the problem. Ollama on qwen2.5:7b starts producing incoherent output when the context exceeds roughly 1200 tokens. A RAG-augmented query with conversation history easily exceeds 1200 tokens. For RAG queries specifically, Ollama is not a viable option on the current hardware. For short, context-free queries, it's fast and capable.

The current production configuration uses Groq as the primary provider. Ollama is available as a fallback when Groq is unavailable, but only for non-RAG queries (query type 4 in the routing table above).

Multi-turn conversation history

One implementation detail that affects all providers: KaryoSpace maintains per-user conversation history, a rolling window of 20 messages (10 exchanges), stored in memory in main.go. This history is prepended to every model call.

var conversationHistory = map[string][]llm.Message{}

func appendHistory(userID string, role, content string) {
    h := conversationHistory[userID]
    h = append(h, llm.Message{Role: role, Content: content})
    if len(h) > 20 {
        h = h[len(h)-20:] // rolling window
    }
    conversationHistory[userID] = h
}

This means the effective token count per request is: system prompt + conversation history + (optional RAG context) + current query. For a user who has been in a long session, the history alone can be 800 or more tokens. This is why Ollama's 1200-token limit is a harder constraint than it first appears.

Provider swap in production

Switching providers requires one config change and one restart:

# In start.sh
export LLM_PROVIDER=groq    # or: gemini, ollama

The application reads LLM_PROVIDER at startup in llm/registry.go and constructs the appropriate implementation. No code changes, no recompilation. This has been useful twice: once when Groq had a brief outage (switched to Gemini in 30 seconds), once when testing whether Ollama could handle a specific query pattern (switched, tested, switched back).

That 30-second incident recovery is the concrete payoff of the abstraction. Without the interface, a provider swap would have required hunting down every http.Post call and adapting the request format. With it, the change is one environment variable.

What I would change

Two things stand out after running this in production.

Per-query provider selection at the application layer. Currently all queries go to the same provider, set at startup. A better design: short queries without RAG context go to Ollama (fast, cheap), long RAG-augmented queries go to Groq (handles context length), creative or summarization queries go to Gemini (strong at synthesis). This requires the routing table to include model selection, not just RAG vs. no-RAG. The interface already supports it since Config.Model lets the caller override the provider's default model. The routing logic just needs to be wired up to use it.

Token counting before model calls. Right now, the system discovers it has exceeded a context limit when the model returns a truncated or incoherent response. Counting tokens before the call would let the system trim history, reduce context, or switch to a larger-context model proactively. The fix requires a tokenizer or a conservative heuristic (characters divided by 4 approximates tokens well enough for a guard check). Either would be better than the current situation where the model is the first to notice the problem.

When local inference is good enough

The honest answer: for non-RAG, short-context queries, local inference with a 7B-parameter model is good enough for an enterprise assistant. Questions like "summarize my tasks for today" or "write a short reply declining this meeting" don't need a 70B model running in a data center. A well-tuned 7B model running locally answers them correctly, and does it in under a second.

Local inference is not good enough when: the query requires reasoning over a long context (RAG augmentation, long email threads), the query requires nuanced language generation (formal communications, executive summaries), or the context window approaches the model's limit. In those cases, the quality drop is noticeable and the latency advantage disappears because the model has to process more tokens regardless of where it runs.

The routing decision should not be "local or cloud." It should be "what does this specific query need?" A classification layer that scores query complexity, estimates token budget, and selects the minimum-viable provider is the right architecture. That is what I am building toward.

The interface was the right call

The LLMProvider interface was the best architectural decision I made in the AI layer. It cost me a few hours to design and implement correctly. It has saved me from being locked into any single provider's pricing changes, availability incidents, or model deprecations. In a field where the best model changes every few months, the ability to swap without touching application code is not a nice-to-have.

The abstraction also forced me to think clearly about what the application layer actually needs from a language model. It needs one thing: given a list of messages and some config, produce a string. Everything else is implementation detail. When you strip the interface down to that, it becomes obvious that three wildly different APIs (Groq, Gemini, Ollama) can all satisfy it. The interface made the providers interchangeable because it correctly identified what "interchangeable" actually means in this context.

---

KaryoSpace is live at karyospace.com. Questions about LLM routing or multi-provider AI systems: sumanakkisetty@gmail.com