Start with the human moment
The right latency target comes from the interaction, not the model benchmark. A voice assistant, visual safety alert, and background document analysis have different definitions of “fast enough.” First identify the moment when delay becomes noticeable, disruptive, or unsafe.
Measure from the event the user creates to the result the user can perceive. That end-to-end path is the product latency. A fast kernel or model server is helpful, but it is not the same measurement.
Optimize the experience clock, not only the inference clock.
Allocate the latency budget
Split the target across every stage that can add delay. The exact allocation depends on the modality, but naming each stage prevents hidden time from accumulating between teams and services.
- Device capture, encoding, and request preparation
- Network travel to the selected runtime and back
- Authentication, routing, queueing, and admission control
- Preprocessing, model inference, and postprocessing
- Streaming, rendering, actuation, or application response
Design for the tail
Average latency can hide the requests that damage the experience. Track percentiles by site, workload version, hardware profile, and request shape. A system that is fast most of the time but periodically stalls may feel less reliable than a slightly slower, consistent system.
Treat queue depth and network variability as first-class signals. They often explain tail latency better than raw accelerator utilization. Placement should consider current service health, not just whether capacity exists.
Use placement as a performance control
When the network consumes too much of the budget, moving inference closer to the request can recover time without changing the model. Local placement can also reduce variance by shortening the number of network segments the request must cross.
Keep a fallback path for workloads that can tolerate it. A smaller nearby model may protect the interactive path while a larger cloud model handles complex or asynchronous work. The latency budget makes that routing policy explicit.