Context Engineering Is an Architecture Problem, Not a Prompt Problem
When an AI application gives a bad answer, one of the first things I want to see is the context.
What instructions did the model receive? What information was retrieved? What tools were available? What did those tools return? What state was the application in? What conversation history was included? What was left out?
That usually tells me more than staring at the final prompt.
Prompt engineering still matters. Clear instructions matter. Examples matter. Output formats matter. But once you move beyond a simple chatbot and start building AI into real applications, the prompt is only one part of a much larger system. The model needs the right information, at the right time, in the right structure, with the right permissions.
That is the practical job of context engineering. And increasingly, I think it is better understood as an architecture problem than a prompt-writing problem.
What is context engineering?
A useful way to think about context engineering is as the design of the information environment around the model.
That environment might include:
System instructions
The user’s current request
Application state
Retrieved documents
Operational data
Conversation history
Tool definitions
Tool results
Examples
Output schemas
Permissions
Memory from previous interactions
The final prompt is simply where some of those things eventually meet.
Consider a customer asking an AI support agent:
Where is my order, and can you refund the shipping charge because it is late?
A useful answer may require the system to know who the customer is, identify the correct order, retrieve its current fulfilment status, understand the shipping policy, know whether a refund is permitted, see any relevant previous conversation and expose the correct tool for issuing the refund.
None of those problems are solved by writing:
You are a helpful customer service assistant.
The hard part is building the system that assembles the right information around that instruction. That is context engineering.
A large context window is not an information architecture
The obvious response to context problems is often to give the model more information.
Modern models can accept increasingly large amounts of context, so technically we can put a lot in there. That doesn’t mean we should.
Imagine connecting an AI application to SharePoint, Google Drive, Slack, CRM records, meeting transcripts, policies, customer history and operational databases.
You could try to load as much as possible into the context. But what have you actually achieved? You have given the model a bigger pile of information to sort through.
Some of it may be useful. Some may be duplicated. Some may be stale. Some may conflict with newer information. Some may belong to another customer or project. Some may not be authoritative. Some may not even be information the current user should be able to access.
More context is not automatically better context. It also costs more tokens, increases latency and makes the behaviour of the application harder to understand.
When something goes wrong, debugging becomes:
Somewhere inside these thousands of tokens was probably the information the model needed.
That is not much of an architecture. A large context window gives you capacity. Context engineering determines what deserves to occupy that capacity.
Context is assembled, not written
This is the shift I think matters most. In a basic chatbot, we tend to think about the prompt as something we write. In a production AI application, context is something the system assembles.
A request might begin with a relatively stable set of system instructions. Then the application adds the current user request. It identifies the user and their permissions and adds relevant application state.
A retrieval service finds information related to the request. Another service might retrieve live operational data.
The model receives a set of available tools. If it calls one, the result becomes additional context. Relevant conversation history might be included. Previous state or memory might also be loaded.
By the time the model makes the important decision, the context may have been constructed by half a dozen different components. That makes context engineering an application architecture concern.
Each source needs a reason for being there. Each source needs a selection mechanism.
Each source needs an appropriate structure. And ideally, you should be able to explain why a particular piece of information was presented to the model.
The architecture needs to decide what the model sees
One pattern I increasingly prefer is to start with the minimum context required for the next decision and make additional information discoverable when needed. Instead of loading everything upfront, the application progressively supplies context.
For example, an internal AI assistant might initially know:
Who the user is
Their permissions
The task they are performing
The application’s current state
A small set of core instructions
Which information sources and tools are available
If the user asks about a customer, the system retrieves that customer’s information. If the conversation moves to a contract, it retrieves the relevant contract. If the model needs current account information, it calls the appropriate tool. If it needs a policy, the relevant policy is retrieved.
The model does not need every customer, every contract and every policy sitting in the context just because those things might theoretically become useful. This creates something closer to just-in-time context.
The architecture determines what becomes relevant as the task develops. That is generally easier to govern, easier to observe and easier to debug than treating the context window as a storage location.
Retrieval is only part of context engineering
Retrieval-augmented generation is one of the first techniques people encounter when building AI applications. Put documents in a vector database. Search for relevant chunks. Add them to the prompt. But retrieval alone does not solve the context problem.
You still need to answer questions such as:
What gets indexed?
How is the content chunked?
What metadata is attached?
Which sources are authoritative?
How is freshness represented?
What happens when two documents disagree?
How many results should be returned?
How should they be ranked?
Does the current user have permission to see them?
Should operational data come from retrieval at all, or should it come from the source system through a deterministic API?
Those decisions can have a larger impact on the final answer than the wording of the prompt.
For example, imagine an employee asking an assistant:
What is our current parental leave policy?
The retrieval system finds three documents. One is the current HR policy. One is an old policy from three years ago. One is a manager’s presentation that summarised the old policy. Semantic similarity might consider all three highly relevant. But relevance isn’t enough.
The architecture needs some understanding of authority, document type, effective date and potentially the employee’s location. That is information architecture and governance. The LLM should not have to infer all of it from a pile of text.
Tools are also context
I think is easy to underestimate the role tools play in context engineering and how important they are.
When we give an agent access to a tool, we normally provide the model with information describing:
What the tool does
When it should be used
What parameters it accepts
What those parameters mean
What the tool returns
That information consumes context.
Give an agent five well-defined tools and the choice may be straightforward. Give it 150 vaguely described tools and you have created a context problem. The model now needs to understand a large capability catalogue before it can even decide what action to take.
This is why good tool architecture matters. Tool names should be clear. Descriptions should be specific. Parameters should be unambiguous. Return values should contain the information the model actually needs. And the model should ideally only see tools that make sense for the current task.
If an agent is answering a question about an invoice, it probably does not need twenty marketing automation tools in its working context. Again, selection matters more than volume.
Context also needs permissions
This becomes particularly important when AI applications connect to internal business systems.
Suppose an employee asks an AI assistant about a customer. Your retrieval layer finds six highly relevant records. Five are ordinary CRM records with standard information. The sixth record contains confidential financial information that the employee is not authorised to view. Technically, that sixth record is excellent context. Architecturally, it should never reach the model.
Permissions need to be applied before context assembly, not after the model has already seen the information. That means identity and access control become part of context engineering.
The application needs to understand:
Who is asking?
Which organisation or tenant do they belong to?
What role do they have?
Which data are they allowed to access?
Which actions are they allowed to perform?
Context is not just about relevance. It is about relevant and permitted information. That distinction becomes critical as agents gain access to more business systems.
Conversation history is another architecture decision
Chat interfaces make conversation history look deceptively simple. Keep appending messages. Send them back to the model. Repeat. But that only works for a while.
Then the conversation becomes long. Old instructions remain in the history. Decisions that are no longer relevant remain in the history. The same information gets repeated several times. Tool outputs accumulate. The user changes direction.
Temporary details begin competing with the information that matters now. Eventually you need to decide what survives. Should the full conversation remain? Should older sections be summarised? Which decisions should become durable memory? Which facts should be retrieved again from the source system rather than remembered? What should disappear completely after the task?
These are context lifecycle decisions. For long-running agents, this becomes even more important because the system may operate across many interactions rather than one chat session.
Memory cannot simply mean:
Save everything forever.
Good memory is selective. Good context is selective too.
Good context engineering reduces the model’s job
One of the most useful architectural questions I ask when looking at an AI workflow is:
What can the surrounding system determine before the model has to reason?
Suppose the application needs to answer a question about an invoice. Code can determine the authenticated user. Code can identify their organisation. A database query can retrieve the authoritative invoice. An API can retrieve payment status. A permissions service can determine whether the user can access the record. A deterministic calculation can work out how many days the invoice is overdue.
Then the model receives something much cleaner:
Here is the correct invoice.
Here is its current status.
Here is the relevant policy.
Here is what the user is allowed to do.
Here is the question.
Now reason about it.
This is a pattern I keep coming back to in AI architecture.
Remove ambiguity before probabilistic reasoning wherever practical.
The less unnecessary uncertainty we hand to the model, the less work we need the model to do. That generally makes the system easier to test and understand.
A practical context engineering architecture
I find it useful to think about context assembly as a series of layers rather than one enormous prompt.
Identity and permissions
Start with who is making the request. Establish the user, organisation or tenant, role and permissions. This determines which information and capabilities can be exposed later.
Task state
Understand what the user or agent is currently trying to achieve. What has already happened? What decision needs to be made now? What application or workflow state is relevant?
Core instructions
Provide the stable behavioural constraints the model needs. Role. Objectives. Boundaries. Required output format. Rules that apply consistently. These should generally remain clear and separate from dynamic business information.
Relevant information
Retrieve the smallest, useful set of information required for the current decision. Prefer authoritative sources. Preserve useful metadata such as source, date and document type when it helps the system reason about reliability. Avoid making the model resolve conflicts that the architecture can resolve first.
Tools
Expose the capabilities that are relevant to the task. Make their contracts clear. Return structured, useful results rather than dumping unnecessary API payloads into the context.
Context lifecycle
Finally, decide what happens after the interaction. What should remain in short-term working context? What should be summarised? What should become durable memory? What should be retrieved fresh next time? What should be discarded?
Thinking in these layers makes context engineering much easier to reason about than treating it as one giant string sent to an API.
How I debug a context problem
When an AI application behaves unexpectedly, I want a context trace. I want to see what the model actually saw at the point it made the decision. Not just the original user message.
I want:
The system instructions
The selected conversation history
Retrieved sources
Operational data
Application state
Available tools
Tool results
Permissions and identity information
Any memory injected into the request
Then I want to know why each piece was included. That lets me distinguish between two very different problems.
The system assembled the wrong context.
Or:
The system assembled the right context, and the model still made the wrong decision.
Those failures require different fixes.
If the correct policy was never retrieved, changing the prompt is unlikely to help. If the model received an obsolete policy, that is probably a retrieval or data-governance problem.
If the model called the wrong tool because ten tools have almost identical descriptions, that is a tool architecture problem.
If important information disappeared because conversation compaction removed it, that is a context lifecycle problem.
Only after I know the model had the right information does it make sense to focus heavily on the model’s reasoning or instructions.
Observability therefore becomes part of context engineering too.
If context is assembled dynamically, we need to be able to inspect that assembly.
Otherwise debugging becomes guesswork.
Bigger models won’t remove the need for better context
It is tempting to assume that better models and larger context windows will eventually make these architecture decisions unnecessary. I expect the opposite.
As models become more capable, we will connect them to more information, more tools and more business processes. The number of possible things we could put into context will grow much faster than the amount we should put into context for a particular decision.
An agent connected to five systems presents one level of complexity. An agent connected to fifty systems, thousands of documents and hundreds of tools presents another. The model may become better at navigating that environment. But the surrounding application still needs to determine access, authority, freshness, relevance, state and lifecycle.
Those are system responsibilities. A smarter model does not remove them.
The architecture around the model matters more than the cleverness of the prompt
Prompt engineering is not disappearing. There will always be value in giving a model clear instructions and designing interactions carefully. But prompt engineering is only one layer.
When AI systems move into production, the harder questions increasingly sit around the prompt.
Where does the information come from?
Why was it selected?
Is it current?
Is it authoritative?
Is the user allowed to see it?
Which tools should be available?
What state should persist?
What should be forgotten?
What happened when the model made the decision?
Those are architecture questions. And they become more important as AI applications become more capable. The model may not need a better prompt. It may need better architecture around its context. And that is the larger lesson of context engineering:
The goal isn’t to give the model more information. It’s to give it the right information, at the right time, in the right structure, with the right permissions.
This article was written by Keith Jenneke, Principal Consultant at Cypher Agency. Keith leads Cypher's Data, Integration, and AI Engineering practice, building governed Modern Data Platforms that make data reliable, integrated, and analytics- and AI-ready, delivered across professional services, resources, and government sectors in Australia.
Comments