← All posts

The model isn't the agent: why the harness matters

ai agentsarchitectureagentic systems

Originally published on LinkedIn.

For the last few years, most of the conversation around AI has centered on models.

Which model is smartest?
Which has the largest context window?
Which is best at coding?
Which performs best on the latest benchmark?

Those questions still matter. But as I spend more time working with AI agents, I’ve become convinced that we’re starting to focus on the wrong layer.

The model isn’t the agent.

The model is the reasoning engine.

The harness around the model is what turns that reasoning engine into an agent.

And increasingly, I believe the quality of that harness will determine whether we build interesting AI demonstrations or genuinely useful autonomous systems.

What do I mean by an agent harness?

At its simplest, an LLM receives context and generates a response.

An agent does something fundamentally different.

It can reason about a goal, choose an action, use tools, inspect the result, update its understanding, and decide what to do next.

That requires infrastructure around the model.

I think of an agent harness as the operating environment that gives the model the ability to work.

It can provide things such as:

  • Tools
  • Files and repositories
  • Commands and APIs
  • Memory
  • Context management
  • Skills
  • Planning
  • Permissions
  • Specialized sub-agents
  • Validation
  • Observability
  • Human approval points
  • Recovery when something goes wrong

This distinction is important.

A powerful model inside a weak harness is still limited.

A capable model inside a thoughtfully engineered harness starts looking much more like an autonomous worker.

One useful definition describes the harness as everything surrounding the agent’s tool-use loop: what tools are available, what requires approval, what context the model sees, and even what survives a failure. (github.com)

That framing resonates with me.

Autonomy doesn’t come from better prompts

For a long time, we’ve tried to make AI systems better primarily by improving prompts.

Then came RAG.
Then tool calling.
Then longer context windows.
Then agents.

But a fully agentic system needs more than any one of these.

Imagine giving an AI this objective:

Build this feature, verify that it works, and deliver it when it’s ready.

That’s only one sentence.

But think about everything required to actually accomplish it.

The system might need to:

  1. Understand the existing codebase.
  2. Read the product requirements.
  3. Develop a plan.
  4. Break the plan into tasks.
  5. Modify multiple files.
  6. Compile the application.
  7. Run tests.
  8. Diagnose failures.
  9. Search documentation when it gets stuck.
  10. Revise its approach.
  11. Run the tests again.
  12. Review its own implementation.
  13. Decide whether the original requirement has actually been satisfied.

The model provides reasoning throughout that process.

But the harness provides the environment in which the reasoning becomes action.

That’s the important distinction.

A harness is more than an orchestrator

When people hear this idea, it’s easy to immediately think “multi-agent orchestration.”

That’s part of it, but I think the harness is bigger.

Orchestration answers questions like:

  • Which agent should do this work?
  • What happens next?

The harness also answers:

  • What information can the agent see?
  • Which tools can it use?
  • What state should persist?
  • What constitutes success?
  • How does it know something failed?
  • Can it retry?
  • When should it stop?
  • When should another agent take over?
  • When must a human approve the next action?

This is where agentic architecture starts looking less like prompt engineering and much more like traditional systems engineering.

Microsoft’s current architecture guidance makes a similar distinction around orchestration, describing sequential, concurrent, group-chat, handoff, and other multi-agent patterns while warning that multi-agent designs introduce additional coordination overhead, latency, and failure modes. (learn.microsoft.com)

That’s an important warning.

Agentic doesn’t mean adding more agents.

It means giving the system the right amount of autonomy and the infrastructure to use that autonomy effectively.

The real architecture is the loop

To me, the fundamental building block of an agentic system looks something like this conceptually:

Goal → Plan → Act → Observe → Evaluate → Adjust → Repeat

The interesting thing is that the LLM doesn’t have to own every part of the loop.

In fact, I don’t think it should.

Some decisions need reasoning.

Others should be deterministic.

The model might decide what needs to happen.
The harness can determine whether it is allowed to happen.

The model might propose that a task is complete.
The harness can run tests to determine whether that’s actually true.

The model might choose a tool.
The harness can determine whether that tool is available in the current context.

The model might get stuck.
The harness can decide whether to retry, provide additional context, delegate to another agent, or escalate to a human.

The model provides intelligence. The harness provides structure.

That combination is where autonomy becomes useful.

Specialized agents become much more interesting

Once you have a harness, you also don’t necessarily need one massive agent capable of everything.

You can start creating specialists.

A planning agent.
A research agent.
A coding agent.
A testing agent.
A documentation agent.
A review agent.

Each can operate with different instructions, tools, permissions, context, and even different models.

An orchestration layer can then coordinate those specialists around a common objective.

That’s the basic idea behind multi-agent orchestration: specialized agents coordinate through a shared system to accomplish complex workflows. (ibm.com)

But specialization introduces an interesting design principle:

An agent should only receive the capabilities it needs to perform its role.

A documentation agent probably doesn’t need production database credentials.

A research agent probably doesn’t need permission to deploy an application.

A testing agent might need to execute code but not merge it.

Boundaries aren’t merely a security mechanism.

They can become part of the architecture.

Memory may be one of the most important parts

Another area I think is underrated is memory.

Today’s AI interactions are often heavily session-oriented.

But real work isn’t.

Projects last days, weeks, months, or years.

A mature agentic system needs ways to retain useful information beyond a single context window:

  • Decisions
  • Requirements
  • Architecture
  • Previous mistakes
  • User preferences
  • Completed tasks
  • Outstanding work
  • Lessons learned
  • Project state

This creates an important distinction between context and memory.

Context is what the model needs right now.

Memory is what the system has learned previously and can retrieve when appropriate.

Once you separate those ideas, the harness can become responsible for deciding what information belongs in the model’s working context instead of continually stuffing everything into the prompt.

Guardrails should live outside the model

This may ultimately be one of the biggest reasons harnesses matter.

If agents are going to become more autonomous, we can’t rely entirely on the model to police itself.

The harness can enforce deterministic boundaries around probabilistic reasoning.

Permissions.
Execution limits.
Approval gates.
Tool restrictions.
Cost limits.
Validation.
Audit trails.
Recovery mechanisms.

The model can still reason creatively inside those boundaries.

That gives us something powerful:

constrained autonomy.

Real-world agent platforms are already emphasizing these concerns. For example, a recent enterprise implementation using LangGraph and LangSmith describes orchestration alongside state management, human-in-the-loop workflows, evaluation, and observability. (prnewswire.com)

Bigger harnesses aren’t necessarily better

There is an important caveat.

If harnesses become the next architectural layer, it will be tempting to keep adding machinery around the model.

More agents.
More tools.
More context.
More planning.
More retries.
More orchestration.

That doesn’t necessarily produce a better system.

Recent research comparing coding-agent harnesses while holding the underlying models fixed found statistically equivalent success rates between the most elaborate and most minimalist harnesses on the larger task set, while costs differed by as much as threefold. (venturebeat.com)

That should make us pause.

The objective shouldn’t be to build the most sophisticated harness.

It should be to build the smallest harness that reliably produces the behavior we need.

From AI applications to agentic systems

I think we’re watching another abstraction layer emerge in software development.

We started by calling models.

Then we wrapped them in applications.

Then we gave them tools.

Now we’re building environments in which models can pursue goals through repeated reasoning and action.

That’s a fundamentally different architectural problem.

And it suggests that the competitive advantage of future AI products may not simply be access to the best model.

Several organizations will have access to the same frontier models.

The differentiation will increasingly come from everything surrounding them:

the tools, memory, context, workflows, permissions, evaluators, feedback loops, specialized agents, and orchestration mechanisms that make up the harness.

In other words:

Models provide intelligence. Harnesses turn intelligence into agency.

And I suspect that over the next few years, some of the most interesting engineering in AI won’t happen inside the models at all.

It will happen in the systems we build around them.

Want help applying this to your own project? Book a call.