For teams paying OpenAI & Anthropic

Stop renting
your AI.

Throttle lives in your code and tests every AI job against other models, closed or open, on your own traffic. When one does the job as well for less, you move it with one approval. If not, nothing moves.

Measured by Throttle

32%

cheaper per token from a serving change (pre-quantized FP8 vs BF16), answers still correct, on our own MI300X run · Qwen2.5-72B

Cheaper only counts if it still works.

Most “savings” are noise, or broken answers that cost less. Throttle checks both before it calls anything a win. These are real runs on rented GPUs, with the raw data on GitHub.

See the raw data
  1. 47%

    Looked cheaper · rejected

    vLLM’s on-the-fly FP8 made Qwen2.5-72B look 47% cheaper per token. The answers had broken: on the 32B model every one came back as !!!!. Throttle called it OUTPUT CHANGED, not a saving.

  2. 32%

    Cheaper · verified

    A pre-quantized FP8 checkpoint of the same 72B model, against BF16: $1.67 to $1.13 per million tokens, with correct answers.

  3. 27%

    Swing with nothing changed

    The same untouched server, checked four times, moved 27% one way and 11% the other. Throttle only calls a winner when a change beats that noise. Read the post

Every AI job, tested for the way home.

  1. 01

    Set up once.

    Run the Throttle proxy in your cloud and point your API base URL at it. Add the GitHub Action. Throttle sees every call your app makes to OpenAI or Anthropic. Nothing else changes.

  2. 02

    It watches every change.

    When your team adds or changes an AI feature, Throttle notices in the pull request and picks the job up. No ticket, no project.

  3. 03

    Tests before it moves. Falls back if not.

    Other models, closed or open, answer the same requests in the background, your own model among them. Grading uses a judge model you pick, with your own key. If one does the job as well for less, you approve and the job moves. If it errors or gives a broken answer, that request goes straight back to the API.

  4. 04

    Keeps every miss.

    Every fallback and every lost test is saved in your cloud as a lesson for your model. Training your model on them is coming

The problems, in engineers’ own words.

We read what teams running AI in production write on Hacker News and GitHub. The same problems kept coming up. Here they are in the words people used, and what Throttle does about each.

52%
of teams run offline evals on a test set. Just over half.LangChain, 1,340 teams
45%
average gross margin on AI products in 2025.ICONIQ State of AI 2026
32%
of model pairs in an outside study where the model with the lower list price cost more in real use, by up to 28x. Not the same 32% as our FP8 run above.Chen, Zaharia, Stoica et al. 2026
16%
of Sonnet 4 requests hit in the worst hour of a bug that Anthropic’s own evals “simply didn’t capture”.Anthropic postmortem
  1. 01Re-testing every release costs too much, so it gets skipped.

    “Pin to a specific model version, run evals on every new release, and only upgrade when your test suite passes. But that’s expensive engineering overhead that shouldn’t be necessary.”

    ThrottleKeeps every job’s tests on your own traffic. A changed prompt sends the job back to testing, and trying a new model is one command. Watching prices for you is coming

  2. 02Nobody wants to find out later that the cheaper model was worse.

    “The hard part with these routers is deciding the cheap model is ‘good enough’ without already knowing the answer.”

    ThrottleNothing moves until it passes. Each job shows better, same or worse on real requests, with error bars one click away.

  3. 03The model you rent changes under you.

    “Tuesdays behaviour changes by Thursday, 10AMs production isn’t possible at 11:30AM.”

    ThrottleA model you own only changes when you say so, and its tests run again in one command. Spotting changes on the vendor’s side for you is coming

  4. 04The AI bill grows faster than revenue.

    “Our competitors are giving half of their margin over to these large AI service providers”

    ThrottleEvery job that moves to your own model comes off the API bill. The monthly savings statement shows what you kept after GPU costs.

  5. 05One cheap model does most jobs well, and fails a few.

    “Gemini flash was pretty good for my needs (and significantly cheaper), but failed to follow the XML schema in the prompt that Claude could follow.”

    ThrottleDecides job by job, never all at once. Extraction can move while the agent stays where it works.

  6. 06The fallback was there. It didn’t fire.

    “Today gemini was having issues and kept returning 500 errors, but the fallbacks didn’t seem to kick in at all.”

    ThrottleFallback comes with every move. Errors, timeouts and broken answers go back to the API, and every fallback is logged with its reason, so you can see it work.

  7. 07Owning GPUs brings its own bills.

    “We’re paying for servers that sit idle at night, you don’t find enough sysadmins”

    ThrottleOwning is not always the answer. Throttle counts GPU costs before it calls anything a saving, and when the API is still cheaper, it says keep it there.

Why teams pick Throttle.

Good tools already exist for parts of this. Here is what each one is great at, and the one question none of them answers for you: which model should do each job, this week, on your traffic.

API gateways LiteLLM, Portkey, OpenRouter

Great atOne API for hundreds of models, with retries and fallback. LiteLLM is a great free gateway you can run yourself.

What mattersGatewaysThrottle
Tests each AI job on your traffic before moving itNot their job. You pick the model. Portkey can A/B test live traffic.Yes. Every job, before anything moves.
Catches prompt and code changes in pull requestsNo.Yes. Flags every new or changed AI job in the pull request. A moved job whose prompt changed goes back to testing.
Falls back automaticallyYes. This is what they do best.Yes, on errors and broken answers. Every fallback is logged.
Learns from misses, trains a model you ownNo.Saves every miss as a lesson for your model. Training on them is coming
Runs in your cloudVaries. LiteLLM and Portkey’s gateway can. OpenRouter is hosted.Yes. Prompts and answers never reach us.
Closed and open modelsYes, hundreds.Yes. Any OpenAI-compatible model, closed or open.
PricingFree and open source (LiteLLM), per-log plans (Portkey, from $49/mo), or a 5.5% fee on credits (OpenRouter’s standard plan).From $49.99 a month. No cut of your savings.

Smart routers Not Diamond, Martian

Great atPicking a model for each request across many models, without you writing the rules.

What mattersSmart routersThrottle
Tests each AI job on your traffic before moving itA trained router guesses per request. Not Diamond can train it on your eval data.Yes. Proof per job first, so there is nothing to guess.
Catches prompt and code changes in pull requestsNot that we could find.Yes. Flags every new or changed AI job in the pull request. A moved job whose prompt changed goes back to testing.
Falls back automaticallyVaries.Yes, on errors and broken answers. Every fallback is logged.
Learns from misses, trains a model you ownThe router learns. The models stay someone else’s.Saves every miss as a lesson for your model. Training on them is coming
Runs in your cloudVaries. Not Diamond offers a private deployment on its enterprise plan.Yes. Prompts and answers never reach us.
Closed and open modelsYes.Yes. Any OpenAI-compatible model, closed or open.
PricingPer token routed (Not Diamond).From $49.99 a month. No cut of your savings.

Eval tools promptfoo, Langfuse

Great atWriting tests for prompts and models, and tracing what your app does. Both are open source and excellent.

What mattersEval toolsThrottle
Tests each AI job on your traffic before moving itYes, on tests you write. Langfuse can also score live traces. Moving the traffic is still up to you.Yes, and once a job passes, it moves with your approval.
Catches prompt and code changes in pull requestsYes, with a GitHub Action you set up.Yes. Finds new and changed AI jobs in the pull request, with no tests to write.
Falls back automaticallyNo. They don’t sit in the request path.Yes, on errors and broken answers. Every fallback is logged.
Learns from misses, trains a model you ownLangfuse can turn bad traces into test sets. No training.Saves every miss as a lesson for your model. Training on them is coming
Runs in your cloudYes. Both can be self-hosted.Yes. Prompts and answers never reach us.
Closed and open modelsYes.Yes. Any OpenAI-compatible model, closed or open.
PricingFree and open source, with paid cloud and enterprise plans.From $49.99 a month. No cut of your savings.

Fine-tuning platforms Together AI and others

Great atTraining an open model on your data when you already know which job to train and have the examples.

What mattersFine-tuningThrottle
Tests each AI job on your traffic before moving itNo. You bring the data and decide what to train.Yes. Finds which jobs are worth it first.
Catches prompt and code changes in pull requestsNo.Yes. Flags every new or changed AI job in the pull request. A moved job whose prompt changed goes back to testing.
Falls back automaticallyNo.Yes, on errors and broken answers. Every fallback is logged.
Learns from misses, trains a model you ownYes. Together AI lets you download the weights. Each run is a project you start.Saves every miss as a lesson, with no project to start. Training on them is coming
Runs in your cloudTraining and hosting run on their cloud.Yes. Prompts and answers never reach us.
Closed and open modelsOpen models.Yes. Any OpenAI-compatible model, closed or open.
PricingPer training token, plus hosting.From $49.99 a month. No cut of your savings.

Build it yourself A gateway, a test suite, a script

Great atFull control. Plenty of strong teams do it, and the first version is not hard.

What mattersBuild itThrottle
Tests each AI job on your traffic before moving itYes, once. Then again for every new model, prompt and price.Yes. Keeps every job’s tests, and flags each change that needs them run again.
Catches prompt and code changes in pull requestsIf you wire it into CI and keep it up.Yes. Flags every new or changed AI job in the pull request. A moved job whose prompt changed goes back to testing.
Falls back automaticallyHand-wired. Worth testing often.Yes, on errors and broken answers. Every fallback is logged.
Learns from misses, trains a model you ownA training project per model.Saves every miss as a lesson for your model. Training on them is coming
Runs in your cloudYes.Yes. Prompts and answers never reach us.
Closed and open modelsWhatever you wire up.Yes. Any OpenAI-compatible model, closed or open.
PricingEngineer time, every week.From $49.99 a month. No cut of your savings.

From each company’s own site, checked October 2026. Products change. If we got one wrong, tell us and we’ll fix it.

Nothing moves without proof. Nothing leaves your cloud.

  1. Your data stays in your cloud.Your model runs on GPUs you control. Prompts and answers never come to us, and keys are never stored.
  2. You approve the moves.Each move needs a passing test and your approval. Pin any job to the API. Undo any move.
  3. Fallback is always on.If your model errors, times out or sends a broken answer, that request goes to OpenAI or Anthropic. Your users still get an answer.
  4. Every decision is logged.With the tests behind it, in plain words.

Verdictsthrottle check

  1. FP8 on the flyoutput changedlooked 47% cheaper · answers broke
  2. FP8 on the flyoutput changedevery answer was “!!!!”
  3. FP8 checkpointcheaper$1.67 → $1.13 per 1M tokens · answers correct
  4. 2 GPUs, not 1more expensive1.56× faster · 28% more per token
  5. nothing changedno winner+27%, then −11% · inside the noise
Real runs: AMD MI300X on Hot Aisle, and a MacBook

A note from the founder

Throttle started as my idea. I built the first version alone.

For six weeks I measured what it really costs to run AI on your own machines. Every team I showed it to asked the same next question: should we even keep paying the API?

So I built the loop that answers it on a team’s own traffic. Watch, test, move. Then do it again next week.

Now Throttle is growing past one person. I’m looking for the first teams to run it with, and for people who want to build it with me. If your AI bill keeps growing, write to me. I read every email.

One flat fee. No cut of your savings.

Starter is $49.99 a month for one app. Team is $99.99 a month for every app you run. Large companies get a custom plan. Every dollar Throttle saves you is yours.

See the plans
Starter$49.99 / month
Team$99.99 / month
Large companiesCustom
Cut of your savingsNone

Start with Throttle.

Pick a plan and leave your work email. Kush sets you up himself, usually within a day. Nothing moves until it passes its tests and you approve it.

Plan

Prefer email? kush@throttle-pro.com. What happens to these details: Privacy · Security

Straight answers.

Something else? Ask Kush.

Does it need access to my code?

It uses a GitHub Action that reads where your code calls an AI API. That is how it finds your AI jobs and comments on pull requests. It runs in your own CI, and your prompts and data stay in your cloud.

Where does my model run?

In your own cloud, on GPUs you control. Your prompts and answers never come to us, and Throttle never stores your API keys. See what leaves your cloud.

What if my model gets something wrong?

If your model errors, times out or sends a broken answer (cut off, empty, bad JSON or a bad tool call), that request goes to OpenAI or Anthropic automatically, so your users still get an answer. The miss is saved as a lesson. Training your model on lessons is coming

Do I have to approve every move?

Yes. A job moves only after it passes its tests and you approve it. You can pin any job to the API or undo a move, and every decision is logged. Letting passing jobs move on their own is coming

What does it cost?

Starter is $49.99 a month for one app. Team is $99.99 a month for all your apps. Large companies get a custom plan. No cut of your savings. See the plans.

Which models does it test?

Any model behind an OpenAI-compatible endpoint, closed or open: a hosted API model with your key, or an open model you serve with vLLM, SGLang, Ollama or LMDeploy. Every one runs the same tests on the same requests your app sent to OpenAI or Anthropic. The savings math is built for models you own; for cheaper API models it is coming

I already use a gateway or an eval tool. Do I still need this?

Keep them. A gateway carries your requests and an eval tool runs the tests you write. Throttle answers a different question: which model should do each job this week, on your traffic, and it moves the job when the answer changes. See how it compares.

Is Throttle running at real companies yet?

We are setting up our first teams now. Every Throttle result on this page comes from our own runs. Outside stats are linked to their source. Our runs used rented GPUs and our own machines, and the raw data is on GitHub.

Is it open source?

Yes. The Throttle code is on GitHub under the MIT license.

Bring your AI home.

From $49.99 a month. Nothing moves until it passes its tests on your traffic and you approve it.

Start now
  • Your data stays in your cloud
  • Falls back on errors and broken answers
  • Nothing moves without proof