Throttle PRE-LAUNCH

          drop-in proxy · no code changes
        

Serve more tokens on the GPUs you already pay for.

Throttle sits in front of your self-hosted LLM inference stack and cuts cost three ways: dynamic request batching, right-sized and spot-aware instance allocation, and semantic caching. Point your traffic at it and nothing else changes.

Get early access View source

          $ export OPENAI_BASE_URL=https://your-proxy.throttle.dev/v1
        

THROUGHPUT · BATCHING ON/OFF

            last run 2026-08-18 14:02 UTC
            n=1,206
          

              max_num_seqs = 1
              312 tok/s · 1.00× baseline
            

              max_num_seqs = 8
              991 tok/s · 3.18×
            

+217.8%

312 tok/s → 991 tok/s (3.18×)
95% CI +189.5% … +246.2%

            +189.5%+246.2%
          

ENGINE

vLLM 0.16.0

HARDWARE

A100 80GB

MODEL

Qwen2.5-0.5B-Instruct

ERRORS

0 / 1,206

01 — INTEGRATION

Three steps, zero code changes.

Throttle speaks the OpenAI-compatible API your clients already use. Swap the base URL and you are done.

STEP 01

Point traffic at the proxy

Change one environment variable. Throttle forwards every request to your existing inference endpoints unchanged.

SHELL

$ export OPENAI_BASE_URL=https://your-proxy.throttle.dev/v1

STEP 02

It profiles your load

Arrival rate, sequence lengths, latency budget and cache hit potential are measured against live traffic.

STEP 03

Savings apply automatically

Batch windows, instance mix and cache policy adjust continuously. No redeploys, no tuning tickets.

02 — COST LEVERS

Where the money comes back.

LEVER 01

Dynamic batching

Single-request serving leaves the GPU idle between forward passes. Throttle holds requests in a short adaptive window and hands the engine full batches, sized against your latency budget rather than a fixed config value.

Measured: 1 → 8 concurrent sequences, +217.8% throughput on the benchmark above.

LEVER 02

Instance right-sizing

Most self-hosted fleets are provisioned for peak and billed for idle. Throttle matches instance class and count to observed load, and shifts eligible traffic onto spot capacity with on-demand fallback when the interruption signal arrives.

Spot-aware placement · graceful drain on interruption notice

LEVER 03

Semantic caching

Production traffic repeats itself. Requests that are semantically equivalent to a recent completion are served from cache above a configurable similarity threshold, so the cheapest inference is the one that never reaches the GPU.

Configurable similarity threshold · per-route TTL · bypass header

03 — GET IN TOUCH

Tell us what you are running.

Throttle is pre-launch and we are onboarding a small number of teams by hand. Engine, model and rough traffic shape are enough to start.

github.com/KushagraKanaujia/throttle
hello@throttle.dev