01
Related
Where to read next on the same subject.
02
Short answer
Price one completed user task, including failed attempts and the work around the model. Count input and output tokens, retrieval, validation, retries, tracing, and human escalation. Then compare that cost with the quality and time the task delivers. A low token bill can hide an expensive failure path.
03
Follow one support question
Suppose a support assistant answers questions about delivery exceptions. It retrieves the current shipping policy, drafts an answer, validates its format, and either responds or sends the case to a specialist. The figures below are invented to show the calculation; they are not a BluePi customer result or a provider quote.
Generation: Input and output tokens by model and prompt version. Retrieval: Search, embeddings when created, reranking, and storage. Validation: Rules or model calls used to check the proposed answer. Failure path: Retries, fallbacks, timeouts, and human escalation. Operations: Tracing, retained samples, review time, and serving capacity.
The provider’s price per million tokens belongs in the calculation, but it is only one input. If an answer needs a second model call after a validation failure, count both calls. If the answer reaches a specialist, count the handoff and the specialist’s time. For a self-hosted model, count idle capacity as well as busy minutes. Compare providers on the same request mix and quality target.
04
Calculate cost per completed task
For an API model, start with this expression for each generation call:
input tokens × input price + output tokens × output price
Add the other model and infrastructure calls in the path. Sum the costs across all attempts, including failures. Divide by tasks that reach the agreed outcome, not merely by requests that received an HTTP 200 response.
Here is a simple model for 100 questions. Assume each first answer uses 3,500 input tokens and 250 output tokens. At hypothetical rates of $1 per million input tokens and $4 per million output tokens, one generation costs $0.0045. If 15 requests need one full retry, generation costs $0.5175 for all 115 calls. Assume retrieval and tracing add $0.15 across the batch. Ten questions then go to a specialist for five minutes each at an assumed labor rate of $0.50 per minute: $25. The observed total is $25.6675.
If 80 answers were resolved by the assistant, 10 by the specialist, and 10 still need rework, the current cost is about $0.29 per completed question ($25.6675 divided by 90). The unfinished 10 still carry cost, and their eventual rework will raise the total. State that completion rule beside the number. Counting only the initial model calls would give a very different, and much less useful, answer.
Keep quality next to cost. A smaller model that reduces model spend but increases wrong-answer rework may raise the cost per successful resolution. The same is true of aggressive context trimming that removes the policy exception the assistant needed.
Separate measured amounts from estimates. Token usage and provider charges may be available per call, while specialist time is often estimated from a sample. Put both in the same model, but mark the measurement method beside each line. An apparently precise cost per task is misleading when its largest line is a rough labor assumption. Revisit that assumption with the support team before using the number to justify a model change.
The denominator also needs an owner. For support, a completed task could mean the customer got an answer that resolved the issue, not simply that the assistant sent a reply. A later correction or reopened ticket can change that status. Pick an observation window, decide how to count handoffs and reopenings, and keep the rule steady when comparing releases. Otherwise a team can report a lower cost per completion by ending conversations earlier.
05
Find the expensive branch before changing models
Break cost down by task type and outcome. A small group of long conversations, repeated retrieval calls, or retries after format failures may account for a disproportionate share of spend. Inspect traces from that branch and fix the cause you can see.
If the prompt repeats instructions on every turn, test a shorter version against the evaluation set. If retrieval returns passages the answer never uses, measure whether a tighter retrieval step preserves the right evidence. If failures trigger the same full-price call again, ask whether a rule, a clearer prompt, or a targeted fallback fixes the failure more reliably.
Change one part at a time and compare cost *and* answer quality on the same requests. Lower token counts are useful only when the task still succeeds.
Show the cost distribution rather than only the average. The median request may be cheap while a few conversations repeatedly call tools, accumulate history, and escalate to a human. Report the typical cost, an expensive tail, and the share of tasks in each branch. A targeted fix to that tail can matter more than changing the model for every routine question.
06
Cache only work that can safely repeat
Caching can avoid repeated work, but eligibility matters more than the cache technology. A public policy answer may be reusable when the policy version and customer context match. “What is the status of my refund?” is tied to a person and a point in time; it should not be served from a shared answer cache.
For each cacheable response, record what makes it valid: the source version, tenant or audience, prompt and model release, and expiry rule. Test invalidation when a policy changes. On BluePi engagements, exact-match caching has saved roughly 5 to 30 percent of total spend in workloads with substantial repeated, stable questions. That is an observed range, not a planning assumption for every application. Measure the repeat rate and wrong-cache incidents on your own traffic. An uncached correct answer is cheaper than a cached answer that creates a support incident.
07
Route only where the quality evidence supports it
A smaller model may be sufficient for a narrow classification step while a complex, source-grounded answer needs a different tier. Define the classes of work first, test each model on the relevant cases, and record which errors trigger escalation. Include the cost and latency of the router and fallback in the comparison.
Do not assume that an automatic “quality check” can reliably catch every weak-model answer. Check the escalation rule against reviewed failures, particularly the mistakes that look confident. If the router sends a difficult case to the wrong model, the saving on the first call can disappear in rework.
08
Review the number with the people who own the outcome
A useful monthly view shows cost per completed task by workflow, answer quality, retry and escalation rates, context size, and trace-storage cost. Add a few representative expensive traces. Finance can see the cost curve; the product and support owners can see whether the service is actually resolving the work.
The review often exposes a bloated system prompt, retrieval returning too much context, or a routine task sent to the top model tier. We have fixed individual instances of those problems in about 30 minutes once the trace showed the cause. That is the edit, not the time needed to test and release it.
When traffic is steady, we have also seen provider discounts change the arithmetic materially. Take a credible volume forecast into that conversation, then compare any commitment and minimum-spend terms with the measured workload. A lower unit rate is useful only if the committed volume is real.
Treat experiments as part of the monthly bill. A new prompt should be tested on the same evaluation cases as the current one, and the cost of those runs belongs in the operating budget. So do review time and the storage needed to inspect failures. Those costs may be small compared with customer-facing traffic, but omitting them makes a safe release process look like an unexpected overrun when usage grows.
The first cost review does not need a new platform. Select one task, define what completion means, price every branch you can observe, and mark the branches still estimated. That gives the team a model it can improve, rather than an invoice it can only react to.