AI Architecture: Stop Sending Everything to an LLM
For the last few years, one question has dominated almost every AI conversation: where can we use an LLM?
It is understandable. Large language models are unusually flexible. They can summarise documents, classify text, extract information, generate code, compare alternatives, answer questions and reason across messy inputs. When one technology can do that many things, the temptation is to make it the centre of every step in a workflow.
That is also how expensive, slow and surprisingly fragile AI systems get built.
A better question is: where does AI reasoning actually add value?
That small change in framing leads to a very different architecture. Instead of pushing every task through a language model, we separate the work. Deterministic code handles deterministic rules. Search retrieves information. Embeddings and similarity techniques identify relevant information. Databases filter and aggregate. Validation logic enforces constraints. The LLM is then used where its strengths matter: interpretation, synthesis, ambiguity and reasoning.
The result is not less AI. It is better AI engineering.
When the prototype architecture becomes the production architecture
A common AI prototype is beautifully simple. Take an input, assemble a large prompt, send everything to the model and ask it to produce the required answer.
For a proof of concept, that can be exactly the right decision. It gets an idea in front of users quickly and proves that the model can perform the difficult part of the task.
The problem starts when the prototype pattern quietly becomes the production architecture.
As usage grows, the application begins sending increasingly large amounts of context with every request. The system prompt grows. More examples are added. More source material is included “just in case.” Conversation history accumulates. Tool descriptions are attached. Outputs from earlier stages are passed wholesale into later stages.
Eventually, the model is being asked to process information that the application itself could have reduced, filtered, calculated or validated before the request was made.
That is where token consumption stops being merely a pricing metric and becomes an architectural signal.
The model was not the bottleneck. The design was.
On a recent client engagement, we encountered the pattern in a workflow that relied heavily on a language model. The specific business process is not important; the architectural lesson is.
Large payloads were flowing into the model for processing. Individual requests were consuming thousands of tokens and the application began running into throughput constraints. The immediate options looked familiar: increase model capacity, request higher limits, change the deployment configuration or move work to a model with different throughput characteristics.
Those options can be valid. But before scaling the model layer, we stepped back and asked a more basic question: which parts of this workload actually require a language model?
Once the workflow was decomposed, the answer was: fewer than the original design assumed.
Some steps were deterministic. They could be implemented in conventional code with predictable results. Some involved identifying similar or relevant items, where indexing, embeddings, cosine similarity or other retrieval techniques were a more natural fit. Some involved filtering structured attributes, which a database or ordinary application logic could do efficiently.
Only the final part of the workflow genuinely benefited from the LLM: taking a deliberately selected set of information and applying contextual reasoning to it.
That changed the optimisation problem completely. Instead of asking how to make the LLM process a large payload faster, the architecture reduced the amount of work that needed to reach the LLM at all.
Use the cheapest reliable computation first
This suggests a useful design principle for AI systems: use the cheapest reliable computation that can correctly perform each step.
“Cheapest” here does not only mean API cost. It includes latency, operational complexity, predictability, testability and the cost of failure.
If a rule is known and stable, deterministic code is usually preferable to asking a probabilistic model to rediscover that rule on every request.
If you need an exact calculation, calculate it.
If you need to filter records by known attributes, query them.
If you need to find semantically similar records or content, retrieval or embeddings may narrow the search space before reasoning begins.
If a result can be cached safely, do not regenerate it simply because an LLM is available.
And when the task genuinely involves ambiguity, interpretation, synthesis, natural language or judgement across context, that is where the language model earns its place.
This is not a rigid hierarchy. Real systems have exceptions. But it is a much stronger default than “send everything to the model and see what it says.”
A four-question test before every LLM call
A practical way to apply the principle is to ask four questions for each stage of an AI-enabled workflow.
First: can deterministic software solve this? Rules, transformations, validation, calculations, schema enforcement, permissions and known business logic usually belong here. Conventional code is repeatable and straightforward to test.
Second: can search, retrieval or an algorithm reduce the problem? Before a model reads a large corpus, retrieve the relevant portion. Before it compares hundreds of records or options, use filters or similarity techniques to create a shortlist. Before it reasons over raw data, aggregate or structure that data where appropriate.
Third: can we reuse something we have already computed? Caching is sometimes overlooked because model calls feel cheap during development. At scale, repeatedly asking the same expensive question is waste. Cache stable outputs, pre-compute reusable representations and avoid reprocessing unchanged information.
Fourth: does the remaining step require reasoning? This is the point where LLMs become powerful. The model can compare nuanced evidence, interpret unstructured language, synthesise multiple sources, explain trade-offs or generate a human-friendly response.
The goal is not to minimise model usage at any cost. The goal is to make every model call purposeful.
Token optimisation is bigger than prompt optimisation
When teams hit token limits or rising inference costs, the first response is often prompt optimisation. Shorten the system prompt. Remove unnecessary examples. Reduce output length. Choose a smaller context window.
Those are useful techniques, but they operate late in the chain.
The bigger question is why the application assembled so much context in the first place.
Selective retrieval can keep irrelevant documents out of the context window. Routing can prevent every request from being sent through the same expensive reasoning path. Semantic shortlisting can reduce the number of documents, records or options a model must inspect.
That is a more useful definition of token optimisation: not “how do I make this prompt a little shorter?” but “what is the smallest high-quality context this reasoning step actually needs?”
Sometimes the best token optimisation is deleting ten words from a prompt. Sometimes it is eliminating the model call entirely.
Long context is capability, not permission
The rapid expansion of context windows has made it technically possible to place enormous amounts of information in front of a model. That capability is valuable, particularly when the relationship between distant pieces of information matters.
But a large context window should not become permission to skip information architecture.
More context is not automatically better context. Irrelevant information consumes capacity, increases processing, can add latency and gives the model more material from which to form an answer. Selective retrieval exists precisely because applications usually benefit from presenting the most relevant information rather than everything available.
The model should receive enough context to make a good decision, not every piece of context the system happens to possess.
Determinism is a feature
There is another reason to keep some work outside the LLM: determinism.
Language models are probabilistic systems. That is part of what makes them useful. They can handle inputs that do not fit neatly into predefined rules and produce useful responses across an enormous range of situations.
But probabilistic behaviour is not an advantage for every task.
A permission check should not be creative. A financial calculation should not vary because the wording of a prompt changed. A schema validation should not be approximately correct. A hard business constraint should not depend on whether the model interpreted an instruction the way you expected.
Production AI systems therefore need a deliberate boundary between probabilistic reasoning and deterministic enforcement.
The LLM can propose. Code can validate. The LLM can interpret intent. The application can enforce permissions. The LLM can generate a structured result. A schema validator can decide whether that result is acceptable.
This division makes AI systems easier to test, observe and trust.
The architecture pattern: reduce, reason, validate
For many applications, a useful high-level pattern is: Reduce → Reason → Validate.
Reduce means doing as much reliable narrowing as possible before invoking expensive reasoning. Parse inputs. Remove irrelevant content. Query structured systems. Retrieve relevant passages. Apply known filters. Use embeddings or search where they are appropriate. Reuse cached results.
Reason means giving the LLM a focused task with focused context. At this stage, the model should be solving the part of the problem that benefits from its capabilities rather than compensating for work the surrounding application failed to do.
Validate means treating the model output as part of a software system. Check schemas. Apply business rules. Confirm permissions. Test critical facts against authoritative systems where necessary. Record the decision path and the model invocation so the behaviour can be evaluated later.
This pattern can be applied to document processing, support automation, internal knowledge assistants, coding agents, workflow agents and many other use cases.
What Better AI Architecture Looks Like
The first wave of enterprise generative AI rewarded teams that could connect to a model and produce a convincing demonstration.
The next wave rewards teams that can engineer the complete system around the model.
That means measuring token consumption, latency and throughput at the workflow level. It means understanding which calls can be routed to smaller or faster models. It means retrieving context deliberately rather than dumping entire datasets into prompts. It means using conventional software for deterministic behaviour and preserving the LLM for the parts of the process where probabilistic reasoning creates value.
It also means evaluating outcomes, not simply model responses.
Good architecture balances quality, cost, latency, reliability and maintainability.
The objective is not the fewest tokens. It is the most value per unit of AI.
AI is a component, not the whole solution
An LLM is not an application architecture. It is a powerful component within one.
The strongest AI systems are likely to look increasingly ordinary in some parts: APIs, databases, queues, caches, search indexes, identity controls, deterministic services, observability, tests and business rules. The language model sits among those components, performing the work that benefits from language and reasoning.
As models become more capable, it will become easier to make them do tasks that conventional software could already perform. The engineering challenge will be resisting the assumption that capability automatically means suitability.
A model might be able to search, calculate, filter, classify, transform, validate and reason in one enormous prompt. That does not mean it should.
Before you send another token
The question for AI teams is shifting. It is no longer enough to ask whether a model can perform a task. In many cases, the answer will be yes.
The more important questions are whether it is the right component for that task, what information it genuinely needs, what can be reduced before the call, what should remain deterministic, and how the result will be validated afterwards.
This is where AI engineering starts to look less like prompt experimentation and more like systems engineering.
Because the long-term advantage will not come from finding more places to insert an LLM. It will come from building systems that combine models, software, algorithms and data deliberately — each doing the work it is best suited to do.
So, before you send another token, ask one question: Is this actually a reasoning problem, or is it an engineering problem?
This article was written by Keith Jenneke, Principal Consultant at Cypher Agency. Keith leads Cypher's Data, Integration, and AI Engineering practice, building governed Modern Data Platforms that make data reliable, integrated, and analytics- and AI-ready, delivered across professional services, resources, and government sectors in Australia.
Comments