← Back to the AI Glossary 📖 Infrastructure & Ops

Throughput

The total volume of requests or tokens an AI system can process in a given amount of time, as distinct from latency, which measures the delay per individual request.

A system can have low latency but also low throughput (fast for one user, but doesn't scale well to many at once) — the two aren't the same thing.
One of 60 free AI glossary terms
Plain-language definitions for the AI jargon you'll actually run into — no email needed, ever, for this section.
Browse the Full Glossary →
More in Infrastructure & Ops
Rate Limit
A cap on how many requests (or how much usage) an account can make to an AI service within...
GPU (Graphics Processing Unit)
Specialised computer hardware, originally built for video game graphics, that turned out t...
Cold Start
The extra delay before an AI system responds to its very first request in a while, as oppo...
API (Application Programming Interface)
A defined way for one piece of software to talk to another — in AI, it's how developers se...