LLM Cost Optimisation: The Cheapest Request Is the One You Never Send
When teams start trying to reduce the cost of an AI application, they often begin in the same place: the model. Can we use a cheaper model? Shorten the prompt? Reduce the context? Cut the output tokens?
Those are useful questions, but they come surprisingly late in the optimisation process.
Before optimising the request you send to a large language model, ask a more fundamental question:
Did this request need to reach a language model at all?
That changes LLM cost optimisation from a prompt-engineering exercise into an architecture problem.
A production AI application should not automatically follow the path:
Request → LLM → Response
It should decide when probabilistic reasoning is actually required. If a cache, database query, deterministic rule, conventional API or precomputed result can answer the request correctly, there is little value in paying a model to generate the same answer again.
The cheapest LLM request is the one you never send.
LLM Cost Optimisation Starts Before the API Call
Consider an internal AI assistant that answers questions about company policies. An employee asks about an expense approval threshold. The application retrieves the relevant policy, sends it to a model and generates an answer. A few minutes later, another employee asks essentially the same question using slightly different words.
Should the application retrieve the same policy, construct another prompt, send another model request and generate another version of the same answer? Perhaps. But probably not by default.
The important optimisation opportunity is not necessarily reducing the second prompt from 2,000 tokens to 1,500. It is recognising that the application may already have everything it needs to answer the question without making the second model call. That principle applies well beyond internal assistants.
Not Every Question Requires Intelligence
Language models make it tempting to treat every user request as a reasoning problem. Many are not. If someone asks for your opening hours, the answer probably does not require an LLM. Neither does retrieving an order status, looking up an account balance, calculating 10% GST on an amount, validating an identifier or applying an explicit eligibility rule. These are deterministic problems with authoritative answers.
A request such as diagnosing an unusual support issue, comparing several options with competing trade-offs or synthesising information across multiple documents is different. In those cases, language-model reasoning may add genuine value.
What is the cheapest reliable path capable of answering this request correctly?
Sometimes that path ends at an LLM. Often, it should not.
Think in Execution Paths, Not Prompts
Instead of designing an AI application around one universal model call, design it around several possible execution paths. A request might be answered by deterministic logic, a system-of-record lookup, a cached response, retrieval from an authoritative source, a precomputed result or, when those paths are insufficient, an LLM.
This makes the model one component in the application rather than the application itself. The objective is not to avoid AI. It is to reserve probabilistic reasoning for the parts of the system where probabilistic reasoning creates value.
Use Deterministic Software for Deterministic Problems
Traditional software is exceptionally good at things such as arithmetic, date calculations, validation, permissions, thresholds, identifier lookups, status checks, pricing rules and explicit business logic. An LLM can often perform these tasks too. That does not make it the best component for them.
If an application needs to determine whether an expense exceeds an approval threshold, a few lines of deterministic code are cheaper, faster, easier to test and easier to audit than asking a language model. Natural language can still provide the interface.
Natural language → structured intent → deterministic execution → optional natural-language explanation
The model can help interpret what the user wants without being responsible for calculations or decisions that conventional software can make more reliably.
Don't Ask the Model for Data Your Systems Already Know
The same principle applies to information retrieval. Imagine a customer asks: “Has invoice 10427 been paid?” If your accounting system already contains the authoritative invoice status, sending that data through an LLM merely so the model can restate it adds another component, another delay and another cost.
Question → intent detection → system lookup → direct answer
An LLM may still be useful when the question requires interpretation — for example, asking why several invoices remain outstanding or identifying patterns across a customer’s payment history. But retrieving a known value is not reasoning. The architecture should recognise the difference.
Cache Repeated Context
Some model calls genuinely are necessary, but even then, the entire request may not need to be treated as new every time. Production AI systems often send large amounts of repeated context with each request: system instructions, tool definitions, policy documents, code, schemas, reference material, examples and conversation history.
Major model platforms now provide prompt or context caching mechanisms specifically to reduce the cost and latency associated with repeatedly processing the same input.
The architectural lesson is broader than any individual provider’s implementation:
Static context should not be treated as new context on every request.
If part of the prompt changes slowly while another part changes on every request, design the application so those characteristics are visible. That creates opportunities for caching rather than repeatedly paying to process identical information.
Cache Answers When Regeneration Adds No Value
Prompt caching avoids repeatedly processing the same input. Response caching can go further by avoiding the model call altogether. If customers repeatedly ask variations of “How long is the warranty?”, “What’s your warranty period?” and “How many years of warranty do I get?”, regenerating a stable answer every time may provide no meaningful value.
Traditional caching works when requests are identical. Semantic caching extends the idea by determining whether a new request is sufficiently similar to one that has already been answered. Microsoft, for example, supports semantic caching in Azure API Management’s AI gateway, where vector similarity can compare incoming prompts with previous requests before forwarding them to an LLM.
This can be powerful, but similarity is not identity. A semantically similar question may still require a different answer. Cached information can become stale, permissions can differ between users, and the similarity threshold itself can produce incorrect matches.
Semantic caching therefore needs appropriate expiry periods, invalidation when source information changes, identity and permission boundaries, and careful thresholds for what constitutes a reusable answer. Caching is an optimisation, not permission to return an answer that is merely close enough.
Retrieve Less Before You Generate
Retrieval-augmented generation is often described as a way to give models more information. A better architecture also asks how retrieval can help give the model less.
If a user asks a question across a thousand-page document set, sending everything to a model is rarely sensible. Retrieval can identify the small subset of information relevant to the request. Sometimes that retrieved information is sufficient to answer the question directly. If retrieval locates an authoritative structured value or an exact policy statement, the application may be able to return it without generation. If interpretation or synthesis is required, only the relevant information needs to continue to the model.
Retrieval should therefore do more than improve model answers. It should reduce the amount of work the model is asked to perform.
Precompute Work That Changes Slowly
Another common source of unnecessary model calls is repeatedly generating information that changes much less frequently than it is requested. Imagine an executive dashboard with an AI-generated summary of yesterday’s sales performance.
Ten managers open the dashboard during the morning. Should the system generate the same analysis ten times? If the underlying dataset has not changed, probably not.
Generate the summary when the data changes, store it, and reuse it until the next refresh. The same pattern can work for recurring summaries, classifications, product descriptions, document extraction, embeddings, risk assessments and other outputs where the source information changes less frequently than the output is consumed.
Compute when necessary. Store when useful. Invalidate when the source changes. Reuse until it does.
If You Still Need an LLM, Choose the Right Capability
After deterministic logic, system lookups, caching, retrieval and precomputation have been considered, some requests will still require a language model. Even then, they do not necessarily require the same model. Classification, extraction, ranking and simple transformations may be handled effectively by smaller, cheaper models, while complex analysis or difficult reasoning may justify a more capable model.
But model routing should come after the first architectural decision:
Does this task require an LLM at all?
Only then should the application ask:
What level of model capability does it require?
Routing a task to a cheaper model is useful. Routing it away from a model entirely can be better.
A Better Request Pipeline
Put these ideas together and the architecture starts to look very different from Request → LLM → Response. A request can move through a series of decisions:
User request → deterministic logic → system-of-record lookup → valid cache → retrieval → precomputed result → appropriate model capability, only when reasoning is genuinely required.
The exact order will vary by application. A cache may sit ahead of a database lookup in one system and behind an authorisation check in another. What matters is that the LLM is no longer the automatic destination for every request. It becomes one execution path among several.
Measure Avoided Calls, Not Just Token Usage
Most AI cost dashboards focus on tokens, model usage and total spend. Those metrics matter, but they only measure what happened after the application decided to use a model. A more mature system should also measure what happened before that decision.
Useful metrics include total application requests, requests resolved without an LLM, deterministic-path rate, cache hit rate, retrieval-only resolution rate, precomputed-result hit rate, LLM invocation rate, cached versus uncached input, model distribution, latency by execution path and cost per successful business transaction. These metrics reveal something token dashboards cannot: how dependent is the application on the model?
A team might reduce average prompt size by 20% and consider the optimisation successful. Another architectural change might prevent 40% of requests from reaching the model in the first place. Those are very different outcomes.
Why did this request require a model?
Don't Optimise Away the Reason You Used AI
There is an obvious danger in taking this argument too far. A sufficiently determined engineering team could build layers of rules, caches, classifiers and routing logic until the architecture becomes more complicated than simply making the model call.
Caches can become stale. Deterministic rules can become brittle. Retrieval can return the wrong information. Smaller models can reduce quality. Precomputed results can lag behind their source data. Sometimes the simplest and most reliable architecture genuinely is a direct model request.
The objective is not to minimise LLM calls at any cost. It is to avoid paying for probabilistic computation when it provides no meaningful benefit. That means measuring the system before optimising it and evaluating cost alongside quality, latency, reliability and operational complexity.
Architecture Before Model Optimisation
Model pricing is highly visible, so it naturally attracts attention when teams begin optimising AI costs. But the cost of an AI application is not determined by model pricing alone. It is also a consequence of architecture.
Before asking which model is cheaper or how many tokens can be removed from a prompt, ask what can be calculated, retrieved directly, cached, precomputed or answered from an authoritative system — and what genuinely requires probabilistic reasoning. Those decisions determine how much work ever reaches the model.
That is why effective LLM cost optimisation starts before the API call. Because the best optimisation may not be a better prompt or a cheaper model. It may simply be better architecture.
And the cheapest LLM request is still the one you never send.
References
OpenAI — GPT-5.4 model and API pricing — https://platform.openai.com/docs/models/gpt-5.4
OpenAI — Prompt caching — https://platform.openai.com/docs/guides/prompt-caching
Google — Gemini context caching — https://ai.google.dev/gemini-api/docs/caching
Anthropic — Prompt caching — https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
Microsoft — AI gateway in Azure API Management — https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities
Microsoft — Enable semantic caching for LLM APIs in Azure API Management — https://learn.microsoft.com/en-us/azure/api-management/azure-openai-enable-semantic-caching
Amazon Web Services — Prompt caching for faster model inference in Amazon Bedrock — https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html
This article was written by Keith Jenneke, Principal Consultant at Cypher Agency. Keith leads Cypher's Data, Integration, and AI Engineering practice, building governed Modern Data Platforms that make data reliable, integrated, and analytics- and AI-ready, delivered across professional services, resources, and government sectors in Australia.
Comments