Most engineers running open-source LLMs in production reach the same point eventually: they have accounts at multiple inference providers, they have a rough sense that one is faster than another, but they cannot say with confidence which provider they should be on for their specific model and traffic pattern. The public benchmarks do not reflect their environment. Their hand-rolled timing scripts are stale. And switching providers without data feels like a coin flip.

This guide walks through a structured approach to benchmarking LLM inference providers using your own API keys, with concrete metrics to collect and a workflow that stays current as provider performance changes.

The three metrics that drive routing decisions

Before running any benchmark, it helps to be clear about which numbers actually matter for your use case.

Time to first token (TTFT) measures the delay between sending a request and receiving the first token in the response stream. This is the metric that controls perceived latency for streaming applications and chat interfaces. If your application streams completions to end users, TTFT is the number you optimize first. A model that streams at 80 tokens per second but takes 1.2 seconds to start will feel slower than one that starts in 200 milliseconds and streams at 60 tokens per second.

Tokens per second measures throughput across the full completion. For batch workloads, background summarization, or any case where you care about total wall-clock time rather than first-response feel, tokens per second is the primary metric.

Cost per million tokens is the product of your input and output token prices at a given provider. It scales directly with usage volume. At low traffic the differences between providers are small enough to ignore. Above roughly 500,000 tokens per day, a 20-30% cost difference between providers becomes a meaningful monthly expense.

What to benchmark

A benchmark is only useful if it is representative. Three decisions shape whether your results will generalize to production behavior.

Use your actual API keys. Provider performance can vary between accounts based on capacity tier, regional allocation, and account age. A benchmark run from a different account may show materially different numbers than what your production application experiences. This is the main reason public leaderboards are unreliable for individual routing decisions.

Use a representative prompt. A 10-token prompt and a 500-token prompt will produce different throughput numbers because the model processes prefill and generation differently. Use a prompt that is similar in length and structure to your real production inputs. If your application uses system prompts, include them.

Set a realistic token limit. The token limit you set in the benchmark should reflect your typical completion length. TTFT is independent of token limit, but tokens per second will vary with output length, and cost depends on total tokens generated.

Running a benchmark manually

The core of a manual benchmark is sending the same request to multiple providers simultaneously, timing each independently, and recording the results. Here is the basic structure in Python:

import asyncio
import time
import httpx

async def benchmark_provider(provider_url, model, api_key, messages, max_tokens):
    headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
    payload = {
        "model": model,
        "messages": messages,
        "max_tokens": max_tokens,
        "stream": True
    }
    start = time.perf_counter()
    ttft = None
    token_count = 0
    async with httpx.AsyncClient(timeout=60.0) as client:
        async with client.stream("POST", f"{provider_url}/chat/completions",
                                  headers=headers, json=payload) as resp:
            async for line in resp.aiter_lines():
                if line.startswith("data: ") and line != "data: [DONE]":
                    if ttft is None:
                        ttft = time.perf_counter() - start
                    token_count += 1
    total_latency = time.perf_counter() - start
    tokens_per_sec = token_count / (total_latency - ttft) if total_latency > ttft else 0
    return {"ttft": ttft, "tokens_per_sec": tokens_per_sec, "total_latency": total_latency}

You run this concurrently across providers using asyncio.gather, collect the results, and compare. The pattern works, and if you need a one-time answer it is good enough.

The problem surfaces when you need this answer repeatedly. Provider performance changes week to week. Pricing updates happen without announcement. A provider that was 40% cheaper three months ago may not be today. Keeping a script like this current, scheduling it to run on a cadence, storing the results, and building trend visibility into it is a significant ongoing maintenance burden on top of whatever else you are shipping.

Using Tesseract instead of maintaining your own script

Tesseract handles the benchmark infrastructure so you do not have to. You connect your API keys for Fireworks, Together AI, Groq, Baseten, and OpenRouter through the provider setup page. Keys are encrypted at rest and never returned in full after they are saved.

To run a benchmark, you go to the benchmark page, select a model from the catalog, enter your test prompt, set a token limit, and click Run. Tesseract fans out the request to every connected provider serving that model in parallel, times TTFT and tokens per second from the stream, computes cost per million tokens from each provider's current price metadata, and writes the results to a table ranked by latency. The run is saved at a permalink so you can share it or return to it later.

The results table shows a green "Fastest" badge on the lowest-latency provider and a blue "Cheapest" badge on the lowest-cost provider. If a provider errors or times out, its row shows "Failed" without affecting the other results.

Building a scoreboard over time

A single benchmark run tells you which provider is winning right now. A scoreboard tells you which provider has been winning consistently, and whether a provider's performance is trending better or worse.

Tesseract's dashboard aggregates your benchmark history into a per-model scoreboard with 24-hour and 7-day windows and trend arrows. You can click into any model to see a latency and cost line chart for each provider over the past week.

On the Pro tier, you can schedule automated benchmarks at hourly, 6-hour, or daily intervals. Tesseract runs the benchmark on your schedule using your stored keys, writes the results to the scoreboard, and keeps the "who is winning right now" answer current without manual intervention.

Setting up alerts for latency and cost changes

Once your scoreboard is running, the final piece is knowing when something changes without having to check manually. Tesseract's alert system lets you define a condition per model — for example, latency over a threshold in milliseconds, cost over a per-token ceiling, or a better provider becoming available — and sends you an email when the condition is met.

The alert fires once when triggered and stays suppressed until the metric recovers and then degrades again. This prevents repeated notifications for the same ongoing degradation and means the emails you receive are actionable: a new condition has been met, here is the model, the condition, and the better provider currently available.

From benchmark to routing config

Knowing which provider is fastest is useful. Having the configuration to switch to it immediately is more useful. Tesseract's routing rules generate a copy-paste config for your chosen model and objective — the provider base URL, the model identifier, a curl command, and a Python snippet — pointed at the current best provider. You update your inference client, deploy, and your application is on the optimal provider without manually checking documentation.

You can start running benchmarks against your own API keys at tesseract.click. The free tier supports two providers and five benchmark runs per day with no payment information required. The pricing page covers the Pro tier if you need automated benchmarks, unlimited providers, and alerts.

Frequently asked questions

How many benchmark runs should I do before trusting the results? A single run can be noisy, particularly for TTFT, which is sensitive to cold starts and request queuing at the provider. Running three to five benchmarks over different times of day and averaging the results gives a more reliable picture of typical performance.

Should I benchmark at different times of day? Yes, if time-of-day performance matters for your application. Some providers have higher utilization during business hours in certain regions, which can affect both TTFT and throughput. Scheduled benchmarks that run across different times of day will surface this pattern in the trend charts.

What token limit should I use in my benchmark? Use a token limit that matches your typical production output length. If most of your completions are 200-400 tokens, benchmark at that range. Benchmarking at 2,000 tokens when your production use is 300 tokens will give you accurate throughput numbers for the wrong scenario.

How do I account for input token costs? Cost per million tokens in Tesseract's results reflects output token pricing from the provider's current rate card. Input token pricing is typically lower and differs by provider. For a full cost estimate, factor in your typical input-to-output token ratio alongside the benchmark cost numbers.