Who this is for, and why it matters
Not a hypothetical feature: Abacus.AI already ships RouteLLM, its own auto-routing layer. This demonstrates the product judgment behind that category.
Platform team
Who: Abacus.AI's own infra team, serving 2M+ customers at $10 to $20/month on a credit system.
Why it helps: every request unnecessarily sent to a premium model erodes margin at that price point.
How they'd use it: as shared infrastructure every product surface calls automatically, with no manual model selection and no per-team routing decisions.
Enterprise customers
Who: SOC 2 / HIPAA-tier customers running compliance-sensitive workloads.
Why it helps: proof that sensitive work isn't quietly routed to a cheaper, riskier model to save cost.
How they'd use it: via the audit trail and override policy, where they'd configure which categories always escalate, regardless of confidence score.
Builders
Who: individual developers on AppLLM and Code AI.
Why it helps: faster iteration, with no waiting on premium-model latency for a trivial formatting fix.
How they'd use it: invisibly: this is the "Coding Assistant" tab in this prototype, routing happens behind the request they already type.
At scale: if even half of request volume across ChatLLM, Code AI, and AppLLM is low-complexity
(classification, extraction, formatting, simple lookups), routing that half to a fast tier at the ~65% lower cost
and ~40% lower latency this prototype's benchmarks show is a meaningful infrastructure cost reduction, without
touching accuracy on the complex reasoning work that actually needs a premium model.
PRD sketch: RELAY routing layer
A discussion draft, not a finished spec
Goal
Reduce serving cost on low-complexity requests without reducing accuracy on high-stakes ones, as a shared layer every product surface calls automatically.
Non-goals
Not a replacement for RouteLLM's existing model catalogue or pricing, but a proposed decision layer on top. Not a new model. Not a change to how any individual model is called once selected.
Primary users
Platform team (cost), enterprise customers (trust/audit), builders (speed), detailed above.
Key user stories
As a platform team, I want low-complexity requests routed automatically to cheaper models, so serving cost doesn't scale linearly with request volume.
As an enterprise admin, I want to configure categories that always escalate, so compliance-sensitive work never depends on classifier confidence alone.
As a builder, I want a visible reason for every routing decision, so I trust the system enough to not manually override it.
Success metrics
| Metric | Illustrative target |
|---|
| Fast-tier traffic share | 40 to 60% of eligible requests |
| Serving cost reduction | Directional, validate against real traffic mix |
| Misroute rate on high-stakes categories | 0%, enforced by override layer, not classifier confidence |
Open questions to validate
What's the real complexity distribution of Abacus.AI's actual traffic? What categories does Trust & Safety require hard-coded escalation for, independent of any classifier? What's an acceptable classification latency budget before it erodes the fast tier's own speed advantage?
The problem
AI-native platforms face a structural tension: premium models are accurate but expensive and slow at scale;
cheap models are fast but unreliable for high-stakes reasoning. Most products pick one tier and live with the
tradeoff company-wide. That's the wrong frame. The real problem isn't which model to standardize on,
it's deciding, per request, which tradeoff is actually worth making, and doing it automatically enough
that no one has to think about it.
System architecture
How this would extend from prototype to production
Step 1
Client request
(chat, code, search)
→
Step 2
Classification layer
(this prototype: rule-based → production: trained model)
→
Step 3
Routing decision
fast /
escalated
→
Step 4
Model tier endpoints
+ telemetry, confidence, cost logging
→
Step 5
Applied surface
coding assistant shown here
This is the request-level flow: one request, five steps. Zoomed out, here's where this layer sits relative
to the rest of an Abacus.AI-style platform.
Where this sits in the platform
Broad-level view: one shared layer, not a feature bolted onto one product
Product surfaces
ChatLLM · Code AI · AppLLM · DeepAgent ·
Enterprise Search (future)
↓ ↓ ↓ ↓ ↓
Routing & dispatch layer (this prototype)
Classification → Override/policy check → Tier decision → Telemetry log
↙ ↘
Fast tier
Llama 3, Mistral-class
Escalated tier
GPT-4, Claude Opus-class
The override/policy check is the piece flagged as missing from this prototype further down. In the broad-level
view it sits inside the routing layer itself, not bolted on afterward.
How this prototype itself is built
Separate from the production architecture above. This is what's actually running when you click around
A single static HTML file, with no server, no database, and no external API calls. The classifier you're interacting
with is real, computed JavaScript logic running entirely in your browser (open dev tools and you can read every
line), not a lookup table and not a call to any model provider. Built with AI as a development partner: I
directed the architecture, the classification signals, the routing thresholds, and the honesty callouts
throughout; AI tooling helped generate and iterate the implementation. Hosted as a static site. The entire
"backend" for this prototype is the logic in this file.
Why intelligent routing, not a single tier
The three approaches a platform can actually choose between
Always premium
Safe on accuracy. Expensive and slow at scale, since every simple classification task pays GPT-4 prices.
Always fast/cheap
Cheap and quick. Unacceptable hallucination risk the moment a request needs real reasoning.
Intelligent routing
Pays premium prices only for requests that need it. Everything else runs fast and cheap, automatically.
Roadmap
Where this goes from a weekend prototype to a production surface
Phase 1
This prototype: rule-based classifier, two model tiers, manual apply/rollback.
Phase 2
Replace the heuristic classifier with a lightweight trained model on real production traffic and outcomes.
Phase 3
Expand beyond two tiers; let teams self-serve their own confidence thresholds per use case.
Phase 4
Extend the same routing layer under Enterprise Search and App Builder: one engine, every surface.
Path to production
This prototype demonstrates the decision logic. Here's what real deployment would actually require.
In production this isn't something end users see or click into. It's infrastructure that sits between
"a request comes in" and "a model executes it," called by every product surface (ChatLLM, Code AI, AppLLM)
before any model runs. Surfaces don't reimplement routing logic themselves; they all lean on one shared service.
Technical & data
Real training data pairing past requests with outcomes · a feature pipeline beyond keyword matching ·
a hard latency budget on classification itself · a feedback loop to catch wrong routes
Safety & governance
A hard override layer above the classifier, where compliance-sensitive categories escalate regardless of
confidence score · full audit trail of every routing decision · staged rollout on a small traffic
percentage before full launch
Organizational
Buy-in from teams giving up direct model-calling for a shared service · an explicit product decision on
acceptable accuracy loss per task and customer segment · an incident process for high-stakes misroutes
A basic version of the override layer described under Safety & Governance is demonstrated in the Routing
Engine tab above, try the "extract-employee-data" example to see policy override a classifier decision in
real time. Production would extend this from a handful of hardcoded terms to a configurable policy engine,
per-tenant rules, a management interface, and full audit logging, but the core mechanism, policy beats
probability, is the same one shown here.