All postsLLM Engineering

Why every production LLM app needs a two-stage intent router

August 20, 2026 7 min read

Most teams building their first LLM product make the same mistake: they ask a 70B-parameter model to do everything in one prompt. Classify the user's intent, decide which tools to call, generate the response. It's a 3-second response time and $0.05 per turn.

Here's what we learned building production LLM apps for Artume OS and other clients: the routing decision is the easiest part to offload to a tiny model, and the results get dramatically better.

The architecture

Stage 1: a 1B–4B parameter classifier. Takes the user's utterance and a list of ~20 possible intents. Returns one intent + confidence score in under 50ms. Runs on CPU. Cost is negligible.

Stage 2: a 7B–70B parameter reasoning model. Only invoked for intents that need real language generation (conversation, summarization, open-ended Q&A). Everything else (system commands, file lookups, calendar operations) goes to a deterministic handler.

Why it works

Speed: the classifier runs in 50ms; only the reasoning calls take seconds. Most turns don't need reasoning at all.

Cost: the classifier costs $0.0001/turn. Routing 80% of turns away from the big model drops your LLM bill by ~5x.

Reliability: when a turn fails, you know exactly which stage failed. With a monolithic model, "the assistant got confused" is your only debugging signal.

Safety: the classifier can veto dangerous intents before the LLM ever sees them. A monolithic model can be tricked into bypassing your system prompt; a separate classifier cannot.

When NOT to do this

If your product is a single-purpose chatbot (one domain, one task), the routing layer adds overhead you don't need. The two-stage architecture pays off when you have ≥3 distinct intents with very different downstream behaviors.

Our numbers

Artume OS routes 78% of turns to deterministic handlers, 18% to a fine-tuned 8B Llama, and 4% to the 70B model for the rare cases that need real reasoning. End-to-end latency p95: 380ms. Cost per conversation: $0.003.

If you're building something LLM-shaped, start with the router. You can always bolt on bigger reasoning later.

#llm#architecture#rust#ollama

Product

Designing an LLM UX for users who cannot see the screen

Building conversational AI for blind users forced us to rethink every UX assumption we had. Here are the four biggest lessons that apply to any LLM product.

Infrastructure

We replaced Pinecone with Qdrant + SQLite. Here is the playbook.

For most production AI applications, self-hosting your vector store is 10x cheaper and faster than a managed service. Here is how we did it for AetherFS.