Definition
Inference is the phase in which a trained AI model is put to work: it receives an input, such as a question, a document or an image, and produces an output, such as an answer, a classification or a generated text. Training happens once and costs a fortune. Inference happens every time someone uses the model and is billed per request or per token. When your team "calls the AI", they are running inference.
For a company, inference is the operational cost of AI. Every support answer, every summarised call, every agent step is an inference call with a price, a latency and a failure mode. Teams choose between hosted inference from AI labs and cloud providers, which is simple and elastic, and self-hosted inference on open-weight models, which gives control over data and can be cheaper at very high, steady volumes.
Inference cost per token has fallen sharply year after year thanks to better hardware, smaller distilled models and techniques such as caching and batching, while reasoning models push the other way by spending more tokens per answer. The misconception is that the model price is the AI cost. Inference volume from agents running many steps is what shows up on the invoice, so monitoring and budgeting matter.
In practice
A media company runs inference on 50,000 articles a day to tag and summarise them. Switching routine tagging to a small model and keeping the large model for editorial summaries cut its monthly AI bill substantially without hurting quality.
Why it matters
Inference is the recurring cost line of AI, the way cloud hosting is for software. Knowing what drives it lets you scale AI use without an unpleasant surprise from finance.
Frequently asked questions
- What is the difference between training and inference?
- Training is the one-off process of teaching a model from data, which takes weeks and enormous computing power. Inference is using the finished model to answer requests, which takes milliseconds to seconds per call. Companies mostly pay for inference; only AI labs and large enterprises pay for training.
- How can I reduce AI inference costs?
- Use the smallest model that meets your quality bar for each task, cache repeated context, shorten prompts, batch non-urgent jobs, and monitor token use per feature. Route easy requests to cheap models and reserve frontier models for the hard ones.