FlashAPI
← Model selection guide

Developer method · 24 September 2026

How to benchmark streaming AI API latency from Southeast Asia

Use a small, repeatable client-side test to compare text chat routes from the country where your users actually are. The downloadable script measures visible streaming response time; it does not depend on FlashAPI or send measurements to us.

The short answer

Run the same OpenAI-compatible stream: true text request from a real Southeast Asian network. Record time from request start to the first non-empty text delta and to the final SSE [DONE] marker. Report p50 and p95 with success count, failures, model, prompt, output limit, client location and test time.

What TTFT and full response mean here

TTFT: first visible text

Elapsed client time from just before the HTTP request until the first SSE chat chunk with non-empty choices[0].delta.content. Role-only, empty and heartbeat events do not count. This is an end-to-end proxy for time to first token, not the model's internal token-generation timestamp.

Full response time

Elapsed client time from the same request start until the SSE [DONE] marker. It includes connection setup, network transfer, provider queueing, generation and any gateway overhead. A stream without [DONE] is a failed sample.

For each metric, p50 and p95 use the nearest-rank rule on sorted successful samples: select positions ceil(0.50 × N) and ceil(0.95 × N), counting from one. For an even N, this p50 selects the lower of the two middle values. The script reports milliseconds using a monotonic local clock. A smaller number is better only when workload, model, output length, route and sample conditions are comparable.

Streaming exposes partial output earlier than a single completed response. That does not prove lower total generation time, and client-visible TTFT also contains network and transport delay.

Run the benchmark on your own node

Use Node.js 20 or newer and a valid key for an endpoint you are allowed to test. Download the complete script and inspect it before use. It has no package dependencies. Supply BENCH_API_KEY through your shell environment or secret manager; never put a key in the command arguments or a shared result. The command below makes 2 warmup requests plus 20 measured requests, which may be billable.

curl -fsSLo benchmark-chat-latency.mjs \
  https://flashapi.wiki-power.com/tools/benchmark-chat-latency.mjs

# Set BENCH_API_KEY in your local environment or secret manager first.
node benchmark-chat-latency.mjs \
  --endpoint "https://YOUR_PROVIDER/v1/chat/completions" \
  --model "YOUR_MODEL_ID" \
  --label singapore-vps \
  --samples 20 --warmup 2

Replace the endpoint and model placeholders with your provider's actual Chat Completions URL and model ID. This small tool supports bearer-key authentication and a query-free URL. The endpoint must use HTTPS; HTTP is accepted only for local loopback testing. The --label value is your own location label, not an automatic geolocation check. Run node benchmark-chat-latency.mjs --help for bounds and options.

Output shapeMeasured successes: N/20; failures: F
TTFT: p50 … ms; p95 … ms
Full response: p50 … ms; p95 … ms

The ellipses are placeholders. We have not measured or published FlashAPI latency.

A test protocol for Southeast Asian routes

  1. Choose real origins. Run from the same type of host your users or application use in Singapore, Jakarta, Ho Chi Minh City and/or Bangkok. Record provider endpoint region and whether a proxy, VPN or corporate network is involved.
  2. Keep the workload fixed. Use the script's fixed English prompt, text-only chat request and 64-token output cap for every route. Confirm that each route accepts the same request and model version.
  3. Warm up, then sample sequentially. The default is 2 unreported warmups and 20 measured requests, 250 ms apart. Repeat at different local times. Run separate experiments for production concurrency; this script measures one request at a time.
  4. Preserve failures. Note HTTP failures, timeouts, missing text or missing [DONE]. They do not enter p50/p95, but must be reported beside the successful count. If failures are frequent, a percentile from successes alone is misleading.
  5. Publish the setup with the numbers. Include client country and host type, time window, endpoint region, model/version, request settings, successful and failed sample counts, p50/p95 for both metrics, and any rate limiting or cache behavior.

A 20-sample p95 is only a directional signal: nearest-rank p95 is the 19th sorted success. More independent samples and repeated time windows are needed before treating tail latency as stable. Node reads can coalesce nearby SSE events, so the script timestamps client-observed chunks rather than provider emission times.

For a customer support purchasing example, see the Singapore model cost and p95 checklist.

Interpret a low-latency claim carefully

Output length changes full time

A short answer can finish sooner even when its model is slower per generated token. Keep the prompt and output cap fixed, then compare actual response lengths if you can access token usage.

Warm paths can look different

Connection reuse, DNS, provider caches, prompt caches and regional routing can shift measurements. Describe the setup and repeat at different times rather than publishing one run as a universal result.

Tail latency needs enough successes

p50 describes a typical successful request. p95 focuses on slower successful requests, but a small sample makes it volatile. Report timeout rate and errors alongside it.

Gateway overhead must be measured

Once an integration exists, compare direct provider and gateway routes from the same origin under equivalent conditions. Regional network distance and provider queueing may outweigh API gateway processing time.

FlashAPI's unified API is in development. No upstream provider is connected, and there are no FlashAPI latency benchmarks or SLA claims today. We will publish a route-specific method and results after an upstream pilot can actually be tested.

Read the proposed unified API contract ↗

Protocol references

The script reads data-only chat completion SSE chunks and the first non-empty text delta. The event framing follows the SSE format.

Have a Southeast Asian latency target?

Tell us the country, workload, model constraints and acceptable p95. We can use that specification to prioritize the first provider pilots.

Share your requirements