What LLM Gateway Load Testing Actually Measures
LLM gateway load testing measures how an intermediary service behaves when it routes, authenticates, limits, caches, logs, and observes requests across one or more language models. It is more complicated than sending a high request rate at a URL because model providers impose quotas, generation times vary dramatically, and token consumption differs by prompt and response. A valid test therefore measures both gateway capacity and the upstream dependency chain, rather than treating a slow model response as proof that gateway hardware is inadequate.
Also worth reading: Which LLM Gateway Performance Metrics Matter Most for Production Workloads? · How Does AI Router Shadow Testing Reduce Production Risk for Agent Teams? · How Do Engineering Teams Design Production Customer Feedback Vector Clustering Pipelines?
A useful test separates four stages: client-to-gateway acceptance, gateway policy execution, gateway-to-provider connectivity, and model inference. For each stage, record request arrival time, queue time, time to first token, total generation time, status code, token counts, and retry count. As a practical starting point, run at least three steady-state levels—perhaps 5, 20, and 50 requests per second—before adding a short stress test at 1.5 to 2 times the expected peak. Those figures are test recommendations, not universal provider limits.
The central question is whether the gateway preserves service objectives as concurrency rises. For a customer-facing product, that may mean 95% of accepted requests reach a provider in under 100 milliseconds, first-token latency stays below three seconds for a defined model, and error rate remains under 1%. These thresholds must reflect the actual product: an asynchronous support-classification system has different requirements from a real-time coding assistant. Measuring capacity without stating a service objective produces raw traffic data but no decision about deployment readiness.
Why a Synthetic LLM Traffic Test Can Mislead
Most load generators produce unrealistic requests because language-model workloads are not represented well by identical small JSON payloads. Prompts can range from 50 to 100,000 tokens, while outputs may be stopped after 20 tokens or allowed to stream thousands. Provider throughput is often governed by input tokens per minute, output tokens per minute, request concurrency, or an organizational spending limit. Repeating “Hello” 1,000 times may validate a proxy but will not predict the queue created by 1,000 long document-analysis jobs.
A defensible test corpus mixes short commands, long-context requests, streaming and non-streaming calls, structured JSON output, tool invocations, and application error responses. Use a production-like token distribution if one exists. If not, build three workload profiles: a median case, a high-token case, and a worst supported case, and assign them realistic proportions. Keep those profiles stable across runs so that a latency change can be attributed to a configuration or code change.
Rate control also needs care. An open-loop generator maintains the offered rate even when responses slow down, which exposes queues and overload behavior. A closed-loop generator waits for each response before issuing more work and can accidentally hide capacity exhaustion. Run both: closed-loop tests support throughput and latency measurements, while open-loop bursts reveal whether timeouts, backpressure, retry storms, and memory growth appear under sustained pressure. For an important release, test at the expected peak plus 20% headroom rather than relying solely on a theoretical maximum.
The Recommended Practical Test Procedure
Begin by documenting the route under test, including the gateway implementation, deployment size, model providers, regional endpoints, authentication method, timeout, retry policy, and concurrency limits. Freeze or version the application payload because changing prompts during a run makes comparisons noisy. Define the success thresholds before generating traffic; examples include less than 1% non-model errors, no duplicate billable operations after retry, stable memory below 80% of the container limit, and no sustained queue growth after a 15-minute burst.
Execute a smoke test first, using approximately 10 to 20 requests with unique identifiers and a small selection of payload sizes. Confirm that tracing data, cost metadata, token usage, user or tenant identity, and error classification are correct. A load test should never use production prompts containing confidential customer information. Generate synthetic but realistic content, and route test traffic through non-production credentials or clearly marked sandbox endpoints whenever the provider supports them.
Next, run a five-minute warm-up followed by 15 to 30 minutes at the expected peak. A short warm-up can initialize connections, populate caches, and reveal cold-start effects, but a test lasting only 60 seconds will usually miss token-rate throttling and slow resource leaks. Capture gateway-side latency separately from model time to first token and total model duration. If Cloudflare AI Gateway, LiteLLM, Kong, or another layer proxies the request, instrument every hop so the team can distinguish gateway processing from upstream queueing.
After the steady-state run, apply a 60-second burst at 1.5 times peak and, if justified, a five-minute stress test at twice peak. Stop increasing load when error rate exceeds 2%, p95 gateway overhead exceeds 50 milliseconds, CPU remains above 85% for five minutes, memory exceeds 80%, or the upstream provider returns repeated rate-limit responses. Those are guardrails for the test program, not universal industry standards. A controlled overload test can continue after the normal success threshold is crossed to observe recovery, but it should not expose real users to the condition.
Finally, run a soak test lasting at least two hours for a major gateway deployment and preferably eight to 24 hours when testing queues, connection pools, autoscaling, or retry behavior. Compare the beginning and end of the run for memory, active handles, error distribution, and latency percentiles. Traffic generators, gateway instances, observability systems, and the model provider can all throttle, so validate that the load client itself maintained the intended rate.
Comparing Load-Testing Approaches and Gateway Alternatives
There is no single best LLM gateway architecture or load-testing method. The correct comparison depends on whether the objective is proxy performance, provider failover, cost control, multi-model routing, or production resilience. A lightweight open-source gateway may be sufficient for a small internal application, while a managed service can reduce operational work but introduces vendor-specific features and another dependency.
| Feature | Open-source gateway such as LiteLLM | Managed gateway such as Cloudflare AI Gateway | Direct model access |
|---|---|---|---|
| Initial setup | Usually free software plus infrastructure | Typically no separate gateway license for basic use; usage and plan charges may apply | No gateway layer; provider account required |
| Control | High configuration and deployment control | Less infrastructure ownership, with platform constraints | Maximum provider-specific control |
| Load-test flexibility | Easy to instrument and deploy in a test environment | Must account for shared infrastructure and service limits | Simplest path, but no gateway policies to test |
| Model portability | Designed for routing across providers and models | Supports provider abstraction, subject to supported integrations | Locked to one provider unless the app builds its own routing |
| Main trade-off | Engineering and maintenance effort | Dependence on managed features and pricing | Weak portability, governance, and cross-provider controls |
Gateway alternatives include building directly on a provider SDK, deploying an in-process routing library, using a general API gateway with custom model adapters, or purchasing a specialized LLM gateway. Direct access is often sensible during prototyping, but it postpones decisions about credentials, centralized policies, provider failover, budgets, and usage attribution. A general API gateway can cover authentication and traffic control but may require custom work for token accounting, model-specific errors, streaming behavior, and semantic routing.
Architecture should follow the workload rather than framework popularity. NVIDIA NeMo Switchyard addresses multi-model agent routing, while LiteLLM is commonly used as an open-source proxy. Cloudflare AI Gateway emphasizes controls around AI traffic. These technologies are not exact substitutes in every deployment, and a benchmark should test the specific integration, region, authentication mode, and fallback sequence that production will use.
Metrics That Matter More Than Raw Requests Per Second
Requests per second is incomplete for LLM systems because request cost and service time vary. Report requests per second alongside input tokens per second, output tokens per second, concurrent generations, time to first token, total latency, and provider rate-limit responses. Segment results by model, route, tenant, payload size, streaming mode, cache status, and fallback path. Without those dimensions, a gateway may appear fast because it rejected difficult requests or sent them to a different provider.
Use percentile latency rather than averages. Track p50, p95, p99, and maximum values for gateway processing, provider queueing, time to first token, and completion. A p95 first-token target below three seconds is reasonable for many interactive applications, but code generation, long-context analysis, and reasoning models can be slower. Set separate objectives by route instead of forcing every model into one global threshold.
Reliability metrics need equal attention. The non-model error rate should exclude upstream model failures when evaluating gateway health, while a separate end-to-end error metric should include everything the customer experiences. Track timeout rate, retry amplification, circuit-breaker openings, fallback success, queue depth, connection reuse, CPU, memory, event-loop delay, and autoscaling frequency. A retry that successfully returns an answer may still double cost and consume rate-limit capacity.
Cost metrics complete the performance picture. Calculate cost per 1,000 successful requests and cost per million input and output tokens, but retain the actual token distribution rather than assuming published list prices. As of 2026, pricing changes frequently and may vary by model, region, caching, batch mode, or provider tier, so confirm current pricing during procurement. A test is economic evidence only when it includes cache-hit ratio, retries, abandoned streams, and fallback-model premiums.
Common Load-Testing Mistakes
The most common mistake is testing the gateway while ignoring the model provider’s quota. Sending 100 requests per second to a provider limited to 10 concurrent generations does not measure local gateway capacity. Read the provider’s current rate limits, account for organization-wide traffic, and ask whether sandbox quotas differ from production. If upstream constraints cannot be removed, generate controlled fake provider responses for a separate proxy-only test and use a small, explicitly capped live test for end-to-end validation.
Another mistake is using identical requests that hit a cache. This can make routing, authentication, and policy layers appear faster than they will for uncached workloads. Include cache hits, misses, stale entries, and cache-disabled traffic, while never caching sensitive data unless the product’s policy allows it. Streaming also requires dedicated tests because clients, proxies, and observability tools may buffer chunks, hide time-to-first-token behavior, or mishandle disconnects.
Retries are frequently misconfigured. A gateway that retries every timeout three times can turn a 10% failure rate into 27% attempted load after the first layer, depending on timing. Restrict retries by error class, honor Retry-After, use exponential backoff with jitter, cap attempts, and exclude retriable authentication and validation failures. During load testing, deliberately return 429, 500, 503, and connection-reset responses in a controlled stub to verify that behavior.
Finally, teams often run one enormous burst and call it production validation. That approach can find a narrow failure but misses memory leaks, quota exhaustion, autoscaling lag, and gradual connection-pool degradation. It can also create real charges or disrupt other users. Use multiple durations and intensities, cap spend, monitor the run in real time, and establish an immediate stop mechanism.
When to Load-Test, and What It Costs
Load-test whenever the gateway is a new production dependency, a major provider or model is added, streaming behavior changes, or a release modifies routing, caching, retries, timeouts, or authentication. Also test before expected traffic events, but allow enough time to fix findings; a same-day test cannot responsibly validate a new distributed architecture. For lower-risk internal tools with low concurrency, a limited provider-aware benchmark may be enough. Customer-facing systems, long-context workloads, and multi-provider failover deserve deeper testing.
The principal cost of a managed gateway may be usage-based rather than a conventional seat license, while an open-source gateway generally charges no license fee but still requires compute, databases, monitoring, security updates, and staff time. Testing itself generates billable model tokens unless a sandbox or local stub is available. Large prompt corpora are expensive: repeating a 20,000-token input across 10,000 requests creates 200 million input tokens, even if every output is short, so compute the budget before execution.
Use a small paid canary first, then expand only after cost controls are verified. Set provider and organization spend limits, gateway budgets, tenant quotas, and alerts—for example at 50%, 80%, and 100% of the approved test budget. The test should tag traffic clearly so charges can be reconciled and accidental production usage is detected. Report both infrastructure expense and labor rather than claiming that “free” software makes performance validation costless.
A practical decision point is saturation of the stated service objective, not the first 429 response. If the gateway meets its latency and error targets through the expected peak plus 20% headroom, scales predictably, and recovers within two minutes after a burst, it is ready for the tested workload. Continue monitoring in production because real prompt distributions, provider capacity, and regional demand will change. For a B2B customer-signal inbox, feed gateway and model telemetry into the same product feedback process used for customer behavior: compare latency and failure routes by account, workflow, and issue category, then turn recurring complaints or regressions into prioritized engineering work.
A Production-Ready Acceptance Standard
A production-ready load test is reproducible, provider-aware, and tied to business impact. Archive the test version, configuration, workload mix, region, model version, thresholds, raw metrics, and cost report. At least one run should repeat the expected peak after changes, and a soak test should establish that the gateway does not degrade over time. The final report should state what passed, what failed, the saturation point, known limits, and which conclusions remain uncertain because the provider or managed platform could not be fully controlled.
Do not claim universal capacity from one test. “50 requests per second supported” is incomplete without prompt length, output length, concurrency, gateway replicas, provider, region, caching, latency distribution, and error rate. A more defensible statement is: “With two gateway replicas, the selected workload and 20% peak headroom, p95 gateway overhead remained below 50 milliseconds, end-to-end errors remained below 1%, and cost per successful request was $X over a two-hour test.” This gives engineers and decision-makers evidence they can compare with later runs.
Load testing does not prove that every future request will behave the same way, but it reduces uncertainty at a controlled cost. The gateway should be judged on end-to-end customer outcomes, not on a fashionable benchmark or raw throughput number. For customer-facing teams, combine infrastructure telemetry with support and product-signal records so degraded model routes become visible as failed classifications, delayed inbox actions, or repeated customer complaints rather than as harmless fluctuations in a dashboard.