Runtime routing

Runtime Routing

Use this when the question is API vs self-hosted vs hybrid.

Do not start with GPU pricing. Start with the workload.

Stay API-First When

• Volume is low or spiky.
• The main leak is prompt bloat, retries, duplicate RAG, or model overuse.
• The team lacks serving operations capacity.
• The provider model quality is the product advantage.
• Data residency allows it.
• The workload is still changing quickly.

Consider Hybrid When

• Some tasks are cheap and predictable.
• Some tasks need premium frontier models.
• Privacy or residency applies to only part of the flow.
• Cacheable or batch work can move to cheaper paths.
• Latency-critical and quality-critical paths differ.

Consider Self-Hosted When

• Steady-state volume is high enough.
• Provider latency or rate limits are the bottleneck.
• Data residency requires it.
• KV reuse, prefix reuse, batching, or custom serving behavior matters.
• Model choice is stable.
• The team can operate the runtime.

Engine Routing

Only discuss engines after workload fit:

• vLLM: general open-model serving, paged attention, broad ecosystem.
• SGLang: structured generation, agentic serving, prefix/cache-heavy patterns.
• TensorRT-LLM: NVIDIA-optimized serving when the workload matches the path.
• Dynamo: disaggregated and rack-scale serving patterns.
• Modular MAX: portability and compiler/runtime direction across hardware.

Do not call any engine universal.

Output

Produce:

• recommended path: API / self-hosted / hybrid / not enough evidence;
• why;
• required proof before migration;
• first benchmark or replay to run;
• rollback plan.