Research paperKoi: Self-Calibrating Planner for LLM Inference Fleets
papers
Double Your Effective GPU Capacity
Get more from the GPUs you already have. Tandemn’s open-source planner works with your existing orchestration to optimize self-hosted AI inference across hardware, clouds, and regions—without expanding your fleet.
A planner for your stack.
Tandemn is an open-source planner that configures and places inference workloads to unlock more GPU capacity. It works with NVIDIA Dynamo, Ray, Kubernetes, and custom systems, so you can make better use of the stack you already have.
ClustersCoordinate workloads across your clusters.
RegionsPlace workloads across clouds and regions.
HardwarePut heterogeneous GPUs to work together.
EnginesKeep the serving engines you already use.
Serving frameworkKeep your existing serving framework.
Improve All Your Key Metrics.
Simultaneously
GPU optimization usually forces tradeoffs between latency, throughput, utilization, SLOs, and cost. Tandemn continuously adapts GPU selection, parallelism strategies, replica counts, and workload placement to actual demand. Across heterogeneous fleet simulations, it admitted more online and batch jobs while improving SLO-honoring goodput, TTFT attainment, and cost per token.
More Jobs admitted
Higher Goodput
Lower TTFT
Lower Cost / token
All your workloads. All your hardware.
One shared pool.
All your workloads compete for the same shared pool of GPU resources. Optimizing each in isolation leaves capacity on the table. Tandemn jointly plans workload placement and resource allocation, dynamically shifting capacity across hardware, clusters, clouds, and regions as demand changes.
Online
Batch
Embeddings
Vision
TandemnCentral planning layer
GPU types
Clusters
Clouds
Regions
One fleet, allocated in real time.
Tandemn treats fragmented, heterogeneous GPUs as one adaptive capacity pool. As inference demand shifts, it reallocates workloads so more jobs fit on the fleet you already run.
ChatbotEmbeddingsRerankingVision
Shared capacity map99/128 GPU in use
Streamline Model Deployment
Tandemn coordinates model architecture, serving engines, GPU hardware, quantization, and parallelism to find configurations that work together. It handles configuration, placement, scaling, and ongoing optimization, so you can deploy from day one without stitching every layer together by hand.
Backed and Proven by Research.
Tested against alternative scheduling approaches across diverse workloads, GPU configurations, and performance objectives, Tandemn delivers more usable GPU capacity while simultaneously improving latency, goodput, SLO attainment, and cost efficiency.
Up to 2×GPU utilization
Up to 45%Lower cost per token
Research & Writing
Ideas, research, and notes from the Tandemn team.
