AI makes software changes cheaper to produce, but harder to understand, verify, and govern at scale.

Here is a scenario most teams working with coding agents will recognize. An agent is given a task. It finds the right files, modifies several of them, and the change is reasonable. Unit tests pass. The service starts.

And then nobody can answer the question that actually matters: does the complete business flow still work?

Not because the change was careless, and not because the agent was lazy. Because the answer does not live anywhere. It is distributed across several repositories, two or three databases, a message topic, a dashboard, a runbook that is quietly out of date, and the memory of the engineer who built the integration and has since moved to another team.

That gap is usually described as a limitation of AI. I think that is the wrong diagnosis.

Many apparent limitations of AI-driven development are long-standing architectural weaknesses that a new kind of system participant has made visible.

They were survivable before because the only operators were humans, and humans compensate. They read Slack history. They remember that the reconciliation job silently fixes a class of mismatch. They know which of the two services is actually the source of truth despite what the wiki says. None of that is in the system. It is around it.

Architecture was always about managing change

It is worth remembering what the architectural styles we have accumulated were actually for.

Monoliths kept code, state, and execution close together, which made a change easy to reason about and hard to isolate. Domain-driven design introduced bounded contexts, drawing lines around business meaning rather than technical layers. Decomposition by volatility grouped together the things that change together. Microservices gave teams independent ownership, deployment, and scaling. Event-driven architecture reduced temporal coupling so that producers and consumers did not have to be available at the same moment.

Different as they are, these approaches were answering a small set of related questions:

  • Where should responsibility live?
  • Which components change together?
  • Which team owns each capability?
  • How can parts of the system evolve independently?

Every one of those questions is about organizing change around human teams. That was the right optimization target, and for the most part it worked.

Decomposition distributed knowledge, not just computation

The part that gets less attention is what else decomposition distributed.

A single business operation — one customer-visible intent — now routinely crosses multiple repositories, a synchronous API or two, a message topic, a stream processor, several independent databases, a scheduled job, a pile of Kubernetes resources, an external vendor platform, and three or four owning teams.

Computation was not the only thing that got spread out. So did:

  • Business knowledge, including what “done” means
  • Runtime state and the failure modes attached to it
  • Test responsibility
  • Operational ownership
  • The source of truth

I work mostly on real-time data pipelines and vendor integration layers, and the honest version of this is that “did this operation actually complete?” is almost never answerable from a single place. You answer it by knowing where to look, in what order, and which signal to distrust. That knowledge is real, it is valuable, and it is almost entirely undocumented.

Experienced engineers compensate for this so automatically that they stop noticing. A new engineer cannot, which is why onboarding takes months. An agent cannot either — and unlike the new engineer, it will not accumulate that knowledge by sitting near the people who have it.

A new actor in the architecture

An agent is not just a faster autocomplete, and it is worth being precise about the difference rather than hand-waving at it.

A traditional development tool operates on artifacts under continuous human direction. An agent, given a goal, may discover what capabilities the system exposes, interpret an ambiguous requirement, modify several components, select and invoke tools, operate infrastructure, observe the runtime result, decide whether the task is complete, and continue without being told what to do next.

I want to be careful here, because this is exactly the point where these discussions usually inflate. I am not claiming agents currently do all of that reliably. In my experience they do some of it well, some of it badly, and the boundary moves. The architectural question does not depend on resolving that:

How should a system be designed when a probabilistic, semi-autonomous machine is one of its operators?

The system now has to support machine understanding and machine action, not only human development and user traffic. That is a genuinely new requirement, and it is a requirement on the system, not on the model.

The cost inversion

The clearest effect of AI on day-to-day engineering is that the local cost of implementation has dropped sharply. Scaffolding a service, integrating another middleware, generating schemas and API clients, writing deployment configuration, producing a first pass of tests, wiring up one more system — all of that is meaningfully cheaper than it was.

What has not changed is everything that comes after:

  • Ownership cost
  • Coordination cost
  • Operational complexity
  • Verification cost
  • Migration cost
  • Compliance obligations
  • Long-term maintenance

AI lowers the cost of adoption. It does not lower the cost of ownership.

These two costs were never equal, but they used to be correlated in a useful way. Adding a queue was expensive to implement, and that expense acted as a crude filter: someone had to want it enough to spend a week on it. Remove the implementation cost and the filter goes with it. The ownership cost stays exactly where it was, and it is now paid by a team that never had to argue for it.

The result is a rate mismatch. Teams can produce architectural change faster than the system — and the organization around it — can safely absorb it. That is not an argument against using AI. It is an argument that the constraint has moved, and the old proxy for “is this worth it” no longer works.

The low-level problems do not go away

None of this suspends the ordinary engineering underneath.

Threads still block. Locks still contend. Transaction boundaries still determine what is atomic and what merely looks atomic. Message ordering is still a per-key property, not a global one. At-least-once delivery still means you will process the same event twice, and idempotency is still the only thing standing between that and duplicated business effects. Retries still amplify load exactly when the system is least able to take it. Eventual consistency still means a read-after-write can be wrong for a while. Partial failure is still the normal case in a distributed system, not the exception, and compensation is still hard because the external effect you need to undo may not be undoable.

For an agent, the practical consequence is sharp:

A successful tool call is not a successful business operation.

The API returned 200. The message was produced. The write committed. None of that establishes that the operation the user asked for actually happened, once, correctly, everywhere it needed to.

So the architecture has to state, explicitly, things that teams usually leave implicit:

  • When an operation is considered complete
  • Which component owns the transaction
  • Whether retrying is safe, and under what conditions
  • How concurrent actions conflict
  • Which intermediate states are observable
  • How failures are compensated
  • How the final result can be verified

I have never seen a system where those answers were free. They are design decisions, and high-level architecture depends on them being made by someone who understands the low-level behavior well enough to make them correctly.

Four properties of an agent-operable system

If I try to compress what a system needs in order to be operated by a machine rather than only by people who remember things, it comes down to four properties.

Legibility

The system has to make its structure and meaning discoverable: which services and environments exist, what each is responsible for, who owns which data, what the integration contracts are, which version is actually deployed, which store is authoritative, and where the intended design and the runtime reality have diverged.

The last one matters more than it sounds. Most systems have a described design and a different actual behavior, and the gap is precisely what an agent cannot infer. Code availability is not the same thing as an understandable system.

Semantic observability

Technical telemetry has to be connected to business meaning. Knowing that an endpoint returned 200, that a message was consumed, and that a database write succeeded tells you that machinery moved. It does not tell you which business operation those events belonged to, whether the whole workflow completed, whether the same work was processed more than once, whether the resulting state satisfies the business rules, or how the effects propagated across service boundaries.

This is the property I see missing most often, including in systems that are otherwise well instrumented. Dashboards full of green technical metrics are entirely compatible with a business flow that is quietly broken.

Verifiability

An agent has to be able to prove that the outcome it was asked for actually occurred. That requires reproducible environments, controllable test data, stable integration entry points, assertions that span services, final states you can query, acceptance criteria that are explicit enough to execute, and a chain of evidence connecting a change to its result.

Passing unit tests is not equivalent to completing a business task. That is true for humans too; it is just that humans usually know it and compensate by poking at the system manually. An agent will take the green check at face value unless the architecture gives it something better.

Governability

Machine action has to stay bounded and accountable: agent identity, delegated authority, least-privilege capabilities, policy enforcement, explicit approval boundaries, environment restrictions, audit trails, and actions that can be reversed or compensated.

I have argued before that if something must happen, you should not ask a model to remember it — you make it part of the mechanism. The same applies here, more strongly. Compliance cannot be bolted on after autonomy. It has to be part of the execution architecture, because by the time an autonomous actor is running, the prompt is not where the boundary lives.

An imagined failure: a system no context window can hold

What follows is a thought experiment, not a postmortem. I have not watched this specific trajectory play out end to end, and I am not describing any particular team. It is a composite built from failure modes I think are individually familiar, arranged to show how they compound.

Picture a team that adopts AI and does well with it at first. Delivery speeds up. People are pleased. Over a few quarters the organization accumulates new services, additional middleware, more asynchronous workflows, duplicated models that were faster to copy than to reconcile, compatibility layers around the duplicates, tools built specifically so agents can operate the system, and a growing set of local tests with no end-to-end assertions behind them.

Every individual change is defensible. Each one passes its own checks. Nobody does anything obviously wrong.

Then the second-order effects arrive. Ownership becomes ambiguous, because the service that writes a field is no longer the service that defines it. Two stores disagree about the same value and both have a reasonable claim. Business transactions cross boundaries that no document describes. Logs record local events accurately while saying nothing about global causality. Documentation and runtime behavior drift apart. Integration tests cover interfaces but cannot reproduce a production workflow.

Now a serious incident happens, and the team does the natural thing: it gives the agent more context. More repositories. More logs. More traces. More schemas. More historical decisions. More infrastructure access.

The additional context does reveal more facts. It also reveals more contradictions. The agent can explain each component convincingly on its own and still cannot establish a causal chain anyone should trust. The failure cannot be reliably triaged. The workflow cannot be safely replayed. The blast radius of a proposed fix cannot be bounded. No test can demonstrate that the correction worked. At that point neither the humans nor the agent can change the system with any confidence.

The system has not run out of context. It has run out of architecture.

Why a bigger context window does not fix it

It is tempting to read that story as a capacity problem, and capacity is the one thing that reliably improves. But information volume and architectural coherence are different quantities, and one does not convert into the other.

A larger context window cannot create clear ownership, consistent definitions, transaction boundaries, reliable causality, reproducible environments, observation points that were never instrumented, executable acceptance criteria, or safe authorization boundaries. None of those are facts waiting to be read. They are decisions that either were made or were not.

The same limit applies to the surrounding techniques, and it is worth being specific about which gap each one leaves:

  • Retrieval finds facts, but does not establish which fact is authoritative.
  • Reasoning analyzes the available evidence, but cannot create evidence that was never recorded.
  • Tool access enables action, but does not make the action safe.
  • More documentation adds claims, but does not make them consistent with runtime behavior.

I would not state this as a law. Models keep getting better at working around ambiguity, and some of what looks structural today will turn out to be a capability gap. But there is a difference between working around missing information and inventing it, and the second is exactly what you do not want in an incident. Model capability cannot compensate indefinitely for architectural ambiguity.

The human role moves up, not out

The conclusion people reach from this is often that AI removes engineers from the process. I think the opposite is closer to true, but the shape of the work changes.

Responsibility moves away from producing every implementation detail and towards defining business and data boundaries, judging whether a new piece of infrastructure is actually justified, designing failure and consistency semantics, establishing observable completion conditions, creating verification strategies, deciding what authority an agent holds, choosing where a human must approve, and defending long-term architectural coherence against a stream of individually reasonable changes.

That is not an escape upward into abstraction. Every one of those decisions rests on understanding transactions, concurrency, networking, storage, and failure behavior. You cannot design a compensation strategy you could not implement. The detail work becomes less of the output and more of the input.

Architecture has to become more intentional

Seen across the eras, the pattern is fairly consistent.

In the monolith era, architecture organized code.
In the microservice era, architecture organized distributed change.
In the agentic era, architecture must also organize machine understanding and machine action.

Which suggests treating AI readiness as a property of the system rather than of the model. Not “can this model handle our codebase” but “is this system understandable enough to navigate, observable enough to diagnose, verifiable enough to trust, and governable enough to operate safely.”

Those four are testable, incrementally. You do not need an agent in production to find out whether a business operation can be traced end to end, or whether an integration test can reproduce a production workflow. Most teams can answer those questions today and will not like the answers, which is useful information regardless of what they decide about agents.

Change is cheap; coherence is not

The risk I would actually flag is not the one that gets the headlines.

The dangerous outcome of AI-driven development is not that AI may write bad code. It is that AI may help us produce change faster than our architecture can absorb it.

Bad code is a known problem with known defenses, and the damage is usually local. Architectural decay is different: it is produced by changes that are individually correct, it accumulates below the level any single review looks at, and by the time it is visible the cost of reversing it is enormous.

None of which is an argument for slowing down AI adoption. It is an argument for being deliberate about what gets added, who owns it, how completion is defined, and how it will be verified — the questions that implementation cost used to force us to ask, and no longer does.

When implementation becomes cheap, architecture must become more intentional—not less.