GLOSSARY
What is decision latency?
Decision latencyDecision latency is the time a System One model takes to answer a request, usually reported as the median (p50) and the 95th percentile (p95) in milliseconds, on stated hardware.
Latency is why System One models exist. A decision in a request path has a budget of tens of milliseconds, not seconds. Reported figures run from a few milliseconds on a data-centre GPU to a few hundred on a laptop CPU.
They depend on the hardware, the length of the state and the number of options, so read every figure with the setup it was measured on. The registry records median_latency_ms and p95_latency_ms, and each model page names the conditions.
Read more
Next term: Confidence gating · Every System One model