How should enterprises design agentic infrastructure for production in 2026?

What enterprise agentic infrastructure design actually means

Enterprise agentic infrastructure design is the discipline of building the systems, interfaces, and controls that allow AI agents to act inside an organization with measurable autonomy. An agent is an artificial intelligence program that can pursue goals, use software or other tools, and take actions with some level of autonomy. In practice, that means an agent might query a database, draft a purchase order, update a customer record, or trigger a deployment, rather than simply returning text to a user. The infrastructure is everything around the model: identity, tool access, memory, orchestration, observability, evaluation, policy enforcement, and human review paths. Many organizations today treat the model as the product and everything else as plumbing, but production systems behave differently. The model is only one component, and in many enterprise workloads it is not even the most expensive or most failure-prone component. As of September 2026, vendors such as ASUS, IBM, Huawei Cloud, and Nvidia have all announced agentic infrastructure or platform offerings, which confirms that this is becoming a distinct architecture category rather than a passing feature. The design question is no longer whether to add agents, but where autonomy is justified and how to bound it.

Also worth reading: What Is a Sovereign AI Infrastructure Architecture and How Do Enterprises Build It? · How can enterprises optimize AI infrastructure costs by 2027 without sacrificing performance? · Which MCP server benchmarking tools comparison should I trust for production-grade AI agent infrastructure?

A useful definition separates an agentic system from a chatbot. A chatbot receives an input and returns an output. An agent receives a goal, decomposes it into steps, selects tools, executes them, observes results, and iterates until it reaches a stop condition or a human intervenes. That loop introduces state, time, and side effects, each of which creates engineering requirements that a chat interface does not have. For example, if an agent issues a refund, the system must know which customer identity it is acting as, what approval limits apply, whether the action is reversible, and how the organization will explain the decision later. The design process therefore starts with the action inventory rather than the model choice. Teams should document every action an agent might take, classify each by reversibility and business impact, and decide which actions require a person in the loop. This framing keeps the discussion grounded in operational reality instead of model benchmarks.

Why traditional enterprise architecture is being reconsidered

Enterprise systems were designed around predictable transactions and fixed workflows. A user opens an application, fills in fields, and the system validates and commits a change. Agents invert that relationship. The agent becomes the user interface, while the ERP, CRM, or mainframe remains the stable backend. Research from Bain and McKinsey describes this shift explicitly: enterprise resource planning systems continue to operate as reliable backends while AI agents mediate the interaction layer on top. This is attractive because it avoids rewriting decades-old systems of record, but it also creates a new integration burden. Every legacy system an agent touches must be exposed through a controlled interface, and every interface must carry authorization context, audit metadata, and error semantics.

The mainframe case makes this concrete. Eight Capital, a YC F25 company, has developed an agentic interface for mainframes and COBOL, which shows that enterprises with highly stable legacy estates are now treating agent access as a first-class design problem. The same reasoning applies to supply chain planning, healthcare administration, and marketing operations. An agent that can read a production database but cannot safely act on it is not useful; an agent that can act but cannot be traced is not acceptable in regulated environments. The architecture must therefore support both capability and restraint. In August 2025, a Show HN project called Datafruit, a YC S25 company, launched with AI for DevOps, illustrating how quickly agent tooling is moving toward operational systems such as CI pipelines and incident response. The opportunity is real, but so is the risk of connecting language models directly to systems that were never designed for probabilistic callers.

The practical consequence is that enterprise architects now spend more time on protocol design and less on screen design. Tool schemas, function calling contracts, event streams, and approval gateways become the equivalent of API design in earlier generations of software. This is why the industry is producing new authorization proposals for agents. A Show HN project named Grantex submitted an IETF draft for an open authorization protocol specifically for AI agents, which is a signal that the standards community recognizes a gap between human OAuth flows and machine delegation. Architecture teams that ignore this layer will find themselves building bespoke, undocumented permission systems that no auditor can evaluate.

The core layers of an agentic architecture

A production agentic platform usually has six layers, and each layer answers a different failure question. The model layer generates plans and selects actions, and it may include one large model for reasoning plus smaller models for classification, extraction, or routing. The orchestration layer runs the agent loop, manages retries, enforces budgets, and decides when to escalate to a human. The tool layer exposes business capabilities through typed interfaces, whether those are REST APIs, SQL gateways, message queues, or mainframe transactions. The memory and context layer stores conversation history, retrieved documents, and task state, with retention rules that prevent stale or sensitive information from persisting indefinitely. The evaluation and observability layer records every prompt, tool call, output, latency measurement, and cost. The governance layer applies policy, audit, and approval rules before and after execution. Teams that skip the governance layer often discover that the evaluation layer cannot reconstruct what happened, which makes incident analysis impossible.

Memory deserves particular attention because it is frequently misunderstood. Conversation history is not the same as business memory, and a vector store is not a system of record. If an agent remembers that a customer prefers email over phone, that is useful context; if an agent remembers an unverified discount authorization, that is a liability. A sound design separates ephemeral working memory from durable, auditable records. Durable facts should be written to systems designed for that purpose, with provenance attached. Retrieval should be scoped by identity and purpose, and sensitive fields should be masked before they reach the model. Many organizations begin with a single vector database and a single prompt, then rebuild this separation under pressure after the first compliance incident.

Tool design is the layer most likely to determine whether an agent is useful. Each tool should have a narrow purpose, explicit input validation, and a defined error contract. A tool that accepts arbitrary SQL is a liability; a tool that runs a parameterized query against an approved schema is manageable. Similarly, an agent that can send an email should use a tool that requires a recipient list, a template identifier, and a preview step, rather than a tool that accepts free text and an SMTP address. These constraints cost more engineering time initially, but they reduce failure rates later. A reasonable target for early deployments is a 95% success rate on low-risk tool calls and a clear escalation path for the remaining 5%, with the threshold tightened for financial or regulated actions.

Security, identity, and governance by design

Identity is the hardest problem in agentic infrastructure because agents act on behalf of users, services, and organizations simultaneously. A common mistake is to give the agent a single service account with broad permissions because individual permissions are tedious. This creates an account that can do anything the agent decides to do, which is the opposite of least privilege. A better model treats the agent as a delegated principal with a narrow scope, a time limit, and an audit trail that links each action back to the human requester. When an agent chains five tools, the system should be able to show which permission allowed each step. This is similar to the problem of agentic commerce, where generative AI, autonomous agents, APIs, and digital payment infrastructure interact, and where a mispriced or misdirected transaction is expensive.

Authorization should be enforced at the tool boundary, not only in the prompt. Prompt-level instructions are advisory; they can be ignored, misinterpreted, or overridden by injected content. A robust platform evaluates policy before each tool call, checks the agent's current identity, and requires approval for high-impact actions. In practice, this means separating read actions, draft actions, and commit actions, with different thresholds. A 2025 deployment of Cohere's agentic AI for hospitals by Ensemble illustrates the stakes in healthcare, where incorrect administrative actions can affect patient access, billing, or care coordination. The same pattern appears in financial services, where a payment instruction above a certain amount may require dual control. IBM Consulting's announcement of an enterprise-scale agentic AI platform natively integrated with AWS in 2025 reflects how vendors are responding to these requirements, but integration alone does not supply governance. The organization still owns the policy decisions.

Data handling requires equal discipline. Sensitive information should be filtered before it leaves the trust boundary, and logs should record enough detail for investigation without recording raw secrets. A practical default is to log tool names, parameters with sensitive values redacted, model versions, timestamps, and outcomes. Teams should set retention windows, for example 90 days for operational logs and longer for regulated audit records where law requires it. Access to those logs should itself be role-based. Governance also includes model change management, because a model update can alter tool selection behavior without any code change. A mature program versions prompts, tool schemas, and model identifiers together, and runs regression evaluations before promoting a change to production.

A practical phased implementation path

Start with a low-risk, high-volume workflow where success is easy to measure. Customer support triage, internal knowledge search, or ticket summarization are safer starting points than payments or clinical decisions. The objective of the first 90 days is not autonomy; it is instrumentation. Teams should establish baseline metrics such as task success rate, human correction rate, average latency, and cost per resolved task. If a human currently handles a request in 8 minutes and the agent reduces review time to 3 minutes, that is a measurable gain even if the agent only drafts rather than commits. A pilot that cannot produce these numbers will struggle to earn a production budget.

The second phase introduces tools and constrained action. Instead of letting the agent call every API, teams should expose three to five tools per workflow and require a dry-run mode that simulates the action without committing it. A 70% auto-execution rate with a 30% escalation rate may be acceptable for internal tasks, while financial actions should begin closer to 0% auto-execution and gradually increase only after evaluation. Teams should also define a budget cap per task, such as a maximum of 10 model calls or a fixed dollar ceiling, so a runaway loop cannot consume an entire month's allowance. These thresholds are design choices, not universal standards, but they force explicit trade-offs.

The third phase adds governed autonomy. Here the organization introduces policy checks, approval queues, and rollback mechanisms. Every agent action should have an inverse or a compensating action where possible, because an agent that sends an email cannot be unsent. For database changes, use staged commits; for infrastructure changes, use canary deployments; for payments, use idempotency keys so a retry does not duplicate a charge. Teams should run adversarial tests with prompt injection, data exfiltration attempts, and unexpected tool failures. A platform that passes 1,000 happy-path tests but has no injection test plan is not ready for production. By the end of the third phase, the organization should be able to answer three questions for any incident: which agent acted, under which policy, and how the action was reversed or contained.

Comparing build, buy, and managed options

The build-versus-buy decision is less about feature checklists and more about where the organization's differentiation lies. If the core value is in an internal workflow that competitors cannot see, the agent logic and tool design may deserve internal ownership. If the value is in standard infrastructure such as hosting, monitoring, or model serving, a managed service may be more economical. Hybrid designs are common, and many teams find that they own the orchestration and governance while renting the model runtime and vector search. The table below summarizes the trade-offs in a simplified way.

Cost estimates should be built from task volume, not seat count. If an agent handles 100,000 tasks per month and each task costs 0.02 dollars in model calls plus 0.01 dollars in infrastructure, the variable cost is roughly 3,000 dollars per month before human review. If human review costs 2 dollars per task, even a 50% automation rate can produce savings, but only if review time falls proportionally. Some managed vendors price per action, per token, or per seat, and these models can reward inefficient agent loops. Teams should negotiate usage caps and request transparent rate cards. TrueFoundry, founded in 2021 in San Francisco, is one example of a company focused on building, observing, and governing enterprise agentic applications, which reflects the market's emphasis on operational tooling rather than raw model access.

Common design mistakes and cost realities

The first common mistake is treating an agent like a deterministic integration. Language models produce variable outputs, so a workflow that assumes identical phrasing or identical tool arguments will fail. The second is allowing agents to browse arbitrary websites before reaching internal systems, which increases latency and attack surface. A 2-second response target may be acceptable for a search assistant but unrealistic for a multi-step agent that queries three systems. Set latency budgets per step and measure the 95th percentile, not the average. The third mistake is skipping idempotency. Retries are inevitable, and an agent that times out after submitting a request may submit it again, creating duplicate orders or tickets. Every mutating tool should accept an idempotency key.

Cost surprises often come from invisible work. Retrieval, tool calls, retries, and evaluation runs can multiply a single user request by 5 to 10 model invocations. Teams should track cost per successful task and cost per failed task separately, because a cheap failure that triggers human escalation may be more expensive than an expensive success. Another mistake is evaluating only answer quality. For agents, the relevant metrics are task completion, correct tool selection, policy compliance, and recovery from error. A model that writes elegant text but selects the wrong tool is not useful. Finally, many organizations underestimate maintenance. Tool schemas change, APIs deprecate endpoints, and models update behavior. A budget that assumes 10% monthly maintenance effort is more realistic than one that assumes no ongoing engineering.

Vendor claims also deserve scrutiny. Terms such as enterprise-grade, autonomous, and production-ready do not guarantee explainability or regulatory fit. Ask for evidence: audit logs, permission models, data retention policies, and incident response procedures. In the United Kingdom, the government's announced billion-level investment in AI infrastructure and AI Growth Zones signals public commitment, but regulatory expectations remain distinct from infrastructure commitments. The EU's AI Act introduces a different compliance environment, and companies operating across regions should map requirements rather than assume one global standard. Architecture advice is most valuable when it separates what a platform can technically do from what an organization is permitted to deploy.

When to act and what success looks like

Act now if your organization already has structured data, a clear owner for each workflow, and a measurable baseline for human performance. Those conditions matter more than model quality. If workflows are undocumented, responsibilities are unclear, or data quality is poor, an agent will amplify the confusion rather than remove it. A good first test is whether a human team can describe a process in a flowchart with fewer than 20 steps. If they cannot, fix the process before automating it. The same advice applies to system access. If the underlying API is undocumented, an agent cannot compensate for that gap; it will guess, and guessing is expensive.

Success in the first year should be measured in operational terms. A reasonable target is a 20% reduction in handling time for the pilot workflow, a 95% or higher success rate on low-risk tasks, and fewer than 1% of actions requiring emergency rollback. These are not universal benchmarks, and they should be adjusted for risk. In healthcare or financial services, the compliance target may be stricter, while internal IT tasks can tolerate more experimentation. Track human override rate as well, because a low override rate can indicate either genuine reliability or an agent that is not being challenged. Sample outputs for quality assurance, and compare against the pre-automation baseline quarterly.

The strategic decision is whether to treat agentic infrastructure as a platform program or a series of projects. A platform program amortizes identity, observability, and policy work across workflows, which pays off when there are more than three use cases. A series of projects is faster initially but creates duplicated tooling and inconsistent risk controls. Most enterprises end up with a hybrid: a small central team owning the platform, and domain teams owning the agents. That structure matches the direction of the market, where vendors such as IBM, Adobe, ASUS, Huawei Cloud, and WSO2 are building platforms and protocols rather than isolated demos. By September 2026, the differentiator is no longer access to a capable model. It is the quality of the architecture that decides what agents can do, what they cannot do, and whether anyone can explain the outcome six months later.