Routing · Observability · Evals
Every model,
one baseURL
Your OpenAI client already works. And because every request runs through us, you can see what each one cost and catch a bad prompt before it reaches your users.
01Model RoutingIn beta
Switch models, not code
One OpenAI-compatible endpoint in front of every provider. Change the baseURL and keep the SDK, the types and the code you already wrote. Dialects are translated at the edge, byte by byte, while the stream is still moving.
One line to switch
Every OpenAI client on earth already works. There is no SDK of ours to install and none of yours to replace.
Streaming passthrough
Nothing is buffered on the way to you. Time to first token stays the provider’s, not ours.
A catalog that routes
Prices, context limits and dialects live in one dataset — the same one that feeds the model pages and the billing math.
Keys, credits and spend
Prepaid balance, hash-stored keys, per-key limits. Multi-tenant from the first request.
POST /v1/chat/completions
model: "anthropic/claude-opus-5"
stream: true
catalog lookup → adapter spec
- anthropicmessages spec
- openaichat spec
- googlegenerate spec
data: {…}data: {…}data: {…}[DONE]
never buffered · usage captured at stream end
02ObservabilityComing next
See what actually happened
You changed a baseURL for routing, so capture was already on. Every request becomes one immutable event — model, tokens, cost, latency, region, your labels — with the whole conversation kept beside it. Nothing to instrument, no spans to wrap, no SDK in your hot path.
Traces
The full conversation, grouped into agent threads by trace ID and into sessions by session ID.
Cost
Per model, per key, per feature, per customer. Your own tags are the axis you slice on.
Performance
Latency, error rate and geography — the last one free, because we already run at the edge.
Privacy mode
Keep the numbers, drop the bodies. Set it per key or per org and the prompts are never written down.
| Time | Model | Tokens | Cost | Latency | Status |
|---|---|---|---|---|---|
| 12:04:31 | claude-sonnet-5 | 1,284 | $0.019 | 412ms | 200 |
| 12:04:29 | claude-haiku-4.5 | 318 | $0.001 | 188ms | 200 |
| 12:04:24 | claude-opus-5 | 6,902 | $0.412 | 1,240ms | 200 |
| 12:04:18 | claude-sonnet-5 | 944 | $0.014 | 397ms | 200 |
| 12:04:11 | claude-opus-5 | 0 | $0.000 | 92ms | 429 |
03EvalsComing next
Stop shipping prompts on a hunch
Nobody writes prompt tests, because rebuilding the failing situation is miserable — the system prompt, the history, the tool output, the exact phrasing. We captured all of it when it happened. Pinning a real trace as a test case is a pointer, not a reconstruction.
Protect a case
One click from the request you are already staring at. The trace you were debugging becomes the test.
Replay a suite
Run every pinned case through the edited prompt and diff it against known-good, case by case.
Safe model upgrades
A new model drops. Run your real cases at it and read the regressions and the cost delta on one screen.
Scores are events
Results write back into the same stream, so quality becomes another column next to cost and latency.
18passed2regressed
20 pinned cases · replayed from real traces
- refund tonepass
- multi-item orderpass
- missing addressregressed
- partial shipmentpass
- currency mismatchregressed
Under the hood
One decision holds it together
All three products exist because of the same choice: every request that passes through becomes a permanent, queryable event.
- 01
Your request hits the edge
A stateless worker authorises your key from cache, looks the model up in the catalog, and opens the upstream connection. No app server, no cold start.
- 02
Dialects are translated in flight
Anthropic events become OpenAI chunks byte by byte as they pass. The stream is never buffered, so time to first token stays the provider’s — not ours.
- 03
One immutable event is emitted
After your response completes, off the hot path, the request becomes a single append-only event — tokens, cost, latency, region, labels. Traces, dashboards and test cases all drink from that stream.
Reproducing a bad AI response is normally the expensive part — the system prompt, the history, the tool output, the exact phrasing. We captured all of it when it happened, so pinning it as a test case is a pointer, not a reconstruction.
The catalog
Curated, measured, honest
Prices, context windows and model IDs are the provider’s published figures, maintained by hand. Latency and uptime are sample data until the probes are live — we would rather show you the shape of the page than invent a measurement.
| Model | Provider | Context | In / 1M | Out / 1M | TTFT | Uptime |
|---|---|---|---|---|---|---|
| Claude Opus 5anthropic/claude-opus-5 | Anthropic | 1M | $5 | $25 | 780ms | 99.94% |
| Claude Sonnet 5anthropic/claude-sonnet-5 | Anthropic | 1M | $3 | $15 | 430ms | 99.96% |
| Claude Haiku 4.5anthropic/claude-haiku-4-5 | Anthropic | 200K | $1 | $5 | 210ms | 99.97% |
| Claude Fable 5anthropic/claude-fable-5 | Anthropic | 1M | $10 | $50 | 1450ms | 99.9% |
Pricing
You pay for tokens
That’s the whole pricing page.
Inference
List price +5%
The provider’s published rate plus a flat routing fee, drawn from prepaid credits. No minimum, no monthly platform charge, no negotiation.
Observability & evals
Included
Not a seat, not an event, not a retention tier. You already paid for the request; looking at what happened is part of it.
- Unlimited keys and workspace members
- Full trace bodies, with a retention window you set
- Privacy mode — metadata only, per key or per org
- No egress charge to take your events with you
Questions
The obvious ones
Do I have to install an SDK?
No. AI Gate speaks the OpenAI API, so you point your existing client at our baseURL and keep the rest of your code. Observability is a side effect of the request already going through us — there is nothing to instrument.
What happens to my prompts and responses?
Bodies are stored so traces and test cases are possible, with a retention window you set per workspace. Turn on privacy mode for a key or an org and we keep the metadata and skip the body entirely.
Does the proxy slow my requests down?
The gateway is a stateless edge worker: it authorises from cache and pipes bytes through, translating the dialect as they pass. Nothing is buffered, and the event is emitted after your response has finished, off the hot path.
What does it cost?
Provider list price plus a flat 5% routing fee, paid from prepaid credits. Observability and evals are included — no per-seat pricing, no per-event pricing.
What if I want to leave?
Change the baseURL back. You were writing standard OpenAI calls the whole time, so there is nothing to unwind.
Change one line.
Keep the receipts.
AI Gate is in private beta. Leave an email and we’ll send a key with credits on it.
No newsletter. One email when your key is ready.