Multi-Agent Systems Software Design Patterns

Multi-Agent Systems Software Design Patterns for Production Architecture
Multi-agent systems rarely fail because a model cannot generate a reasonable answer. They fail because the surrounding software architecture treats probabilistic, stateful, distributed workers like synchronous functions.
A production agent platform has to coordinate multiple decision-makers, tools, databases, queues, users, and policy boundaries. It must remain correct when an agent retries a tool call, a worker crashes after committing a side effect, a model returns malformed structured output, or two agents update the same business entity concurrently. Latency and token cost matter, but correctness, recoverability, and observability matter more.
The central architectural problem is therefore not simply how to call a model. It is how to define ownership, communication, state transitions, and failure semantics across autonomous components. This article presents practical Multi-agent systems software architecture patterns for systems built with TypeScript, Python, OpenAI APIs, PostgreSQL, queues, gRPC, REST, and event-driven services.
Start with Capabilities, Not Personalities
The most common design mistake is decomposing a system into agents named after roles: researcher, writer, reviewer, and manager. These names are useful for product discussions, but insufficient for architecture. An agent boundary should represent a capability with an explicit contract, authority, and data ownership.
For example, an order automation platform might contain:
- An intake agent that converts user intent into a validated command.
- A policy agent that evaluates authorization, budget, and compliance constraints.
- A fulfillment agent that calls external providers.
- A reconciliation agent that compares provider state with the internal ledger.
- A notification agent that publishes user-visible updates.
Each agent should have a narrow responsibility and a bounded tool set. The policy agent should not be able to place an order. The notification agent should not mutate payment state. This is the same principle used in secure microservices design: minimize authority and make invalid actions structurally difficult.
A useful agent contract includes:
- Input schema and version.
- Output schema and confidence or status fields.
- Allowed tools and maximum execution budget.
- Data it may read and write.
- Timeout, retry, and cancellation behavior.
- Idempotency requirements.
- Human approval requirements.
The coordinator should not depend on free-form prose to determine whether an agent succeeded. Use structured outputs with explicit statuses such as
completedblockedneeds_approvalfailedOrchestration and choreography
There are two primary coordination patterns. In orchestration, a workflow service explicitly invokes agents and decides the next step. In choreography, agents subscribe to events and independently react to state changes.
Orchestration is easier to reason about for bounded workflows. A coordinator can enforce deadlines, maintain a step graph, and provide a single execution trace. It is usually the right starting point for payment, provisioning, or approval flows.
Choreography reduces central coupling and works well for reactions such as indexing, notifications, analytics, and reconciliation. Its cost is higher complexity: causal relationships become harder to discover, duplicate events are normal, and global progress is less obvious.
A mature platform often combines both. Use a durable orchestrator for the critical path and events for side effects. Do not allow every agent to invent its own workflow semantics.
State Synchronization: Durable Facts Over Shared Memory
Agents need context, but a giant shared conversation transcript is not a state model. Transcripts are expensive, difficult to query, and vulnerable to prompt injection or accidental disclosure. Separate durable business state from ephemeral model context.
A practical state model has four layers:
- Business state: orders, accounts, permissions, balances, and workflow status in a transactional database.
- Execution state: runs, steps, attempts, leases, deadlines, and tool-call records.
- Context state: summaries, retrieved documents, user preferences, and selected conversation turns.
- Telemetry state: traces, token counts, latency, model versions, and evaluation outcomes.
The database should be the source of truth for the first two layers. A vector store or cache can accelerate context retrieval, but it should not be authoritative for financial or workflow state.
Event Sourcing and CQRS
The Event Sourcing and CQRS pattern web is particularly effective when an agent workflow needs auditability and replay. With event sourcing, immutable events such as
OrderRequestedPolicyApprovedProviderChargeCapturedThe pattern is not free. Event schemas become long-lived public contracts, projections can lag, and correcting a bad decision may require compensating events rather than an update. Use it where history, replay, regulatory review, or asynchronous projections justify the operational cost. For simpler agent metadata, a conventional relational model with an append-only audit table is often sufficient.
An event should contain a stable identifier, aggregate version, causation ID, correlation ID, actor, timestamp, and schema version. Consumers must be idempotent because at-least-once delivery is the realistic default.
type DomainEvent = {
eventId: string;
aggregateId: string;
aggregateVersion: number;
type: string;
schemaVersion: number;
occurredAt: string;
causationId?: string;
correlationId: string;
payload: unknown;
};
Optimistic concurrency prevents two agents from silently overwriting one another. An update should include the version it read, and the database should reject the write if the current version has changed. The coordinator can then reload state, re-evaluate the step, or escalate to a human rather than guessing.
Communication Choices: gRPC, REST, and Events
Microservices communication gRPC vs REST is not a binary technology debate. Select the protocol according to the interaction semantics.
Use gRPC for internal, low-latency request-response calls where both sides are controlled. Protocol Buffers provide compact payloads, generated clients, deadlines, and explicit schema evolution. gRPC is effective for an orchestrator calling a policy or retrieval service many times during one workflow.
Use REST for public APIs, browser clients, third-party integrations, and operational simplicity. REST is easier to inspect with standard tools, works naturally with Next.js 15 route handlers and server actions, and is more accessible across organizational boundaries.
Use an event broker for durable asynchronous work. Events decouple producers from consumers and absorb bursts, but they introduce eventual consistency and require delivery, ordering, and replay policies.
A sensible division looks like this:
- Next.js 15 frontend to application API: HTTPS REST or a typed RPC layer.
- Orchestrator to internal agent services: gRPC with deadlines and generated types.
- Workflow transitions and side effects: durable queue or event broker.
- External providers: REST, webhooks, and provider-specific idempotency keys.
Every call needs a deadline. Retries without deadlines create retry storms and can multiply expensive model calls. Retry only transient failures, use exponential backoff with jitter, and distinguish transport failure from a valid agent response that says the task is impossible.
const result = await policyClient.evaluate(command, {
deadline: Date.now() + 1500,
metadata: {
'x-correlation-id': correlationId,
'x-idempotency-key': idempotencyKey
}
});
For streaming user interfaces, do not expose an internal agent socket directly to the browser. Convert internal progress events into an authenticated server stream, such as Server-Sent Events, through a Next.js 15 route handler or a dedicated gateway. The gateway should filter sensitive tool arguments and support reconnecting from a sequence number.
Building AI Agents with OpenAI API
Building AI agents with OpenAI API should be treated as integration with an unreliable external dependency, even when the API itself is highly available. Models can produce invalid arguments, exceed latency budgets, select an unsafe tool, or return a plausible but incorrect conclusion.
Wrap model access behind a provider interface. This prevents model-specific details from leaking into business logic and enables controlled fallback, replay, and evaluation.
from dataclasses import dataclass
from typing import Any
@dataclass
class ModelDecision:
status: str
tool_name: str | None
arguments: dict[str, Any]
rationale: str | None
class AgentModel:
def __init__(self, client, model: str):
self.client = client
self.model = model
async def decide(self, messages, tools) -> ModelDecision:
response = await self.client.responses.create(
model=self.model,
input=messages,
tools=tools,
temperature=0,
metadata={'component': 'policy-agent'}
)
decision = parse_and_validate(response)
return decision
The wrapper should enforce:
- Structured output validation before execution.
- Maximum tool calls per step and maximum total tokens per run.
- Model and prompt version identifiers in every execution record.
- Redaction of secrets and personal data before logging.
- A circuit breaker when provider errors exceed a threshold.
- Human approval for irreversible or high-value operations.
Tool execution must be outside the model call and inside a policy-controlled runtime. Treat tool arguments as untrusted input. Validate identifiers against the authenticated tenant, enforce authorization again at the tool boundary, and use allowlisted destinations for network access.
The critical transactional sequence is: reserve work, call the model, validate the decision, execute an idempotent tool, record the result, and publish the next event. If the process crashes after the external side effect but before recording completion, a retry may repeat the action. Solve this with provider idempotency keys, an operation ledger, or a reconciliation process. A database transaction alone cannot roll back a remote API call.
Production Protocols and Implementation Workflow
A reliable implementation workflow is more valuable than an elaborate prompt. Build the platform in this order:
- Define domain commands and events before designing agent prompts.
- Create one agent with one tool and a deterministic test fixture.
- Add schema validation, idempotency, timeouts, and trace propagation.
- Persist every run, step, attempt, model version, and tool result.
- Introduce a queue and worker leases before adding parallel agents.
- Add approval gates for irreversible actions.
- Load-test model latency, queue depth, database connections, and provider quotas.
- Replay recorded traces against new prompts and models before production rollout.
A worker lease prevents two consumers from processing the same step concurrently. The lease must expire, and ownership must be checked before committing results. PostgreSQL can support this with row locking and
SELECT ... FOR UPDATE SKIP LOCKEDRecommended baseline settings include:
- Request deadlines: 1–3 seconds for internal reads, 10–30 seconds for model steps, and explicit longer limits for asynchronous jobs.
- Worker lease duration: several times the expected step duration, renewed while active.
- Maximum attempts: bounded per failure class, with dead-letter routing after exhaustion.
- Idempotency retention: long enough to cover provider retries and reconciliation windows.
- Database pool size: limited according to database capacity, not worker count.
- Queue visibility timeout: longer than the maximum tool execution time, with heartbeat renewal.
For low-latency APIs, avoid putting every agent step in the synchronous HTTP request. Return a run ID quickly, process the workflow asynchronously, and stream status to the client. This protects frontend responsiveness and prevents load balancers from becoming workflow coordinators.
Use PostgreSQL for transactional state, read replicas for non-critical queries, and partition large execution or event tables by time or tenant where necessary. Do not introduce a distributed database before measuring contention, storage growth, and read patterns. Most early bottlenecks are unbounded transcripts, missing indexes, oversized connection pools, and synchronous calls to slow providers.
Failure Handling and Operational Metrics
Failure is a first-class workflow outcome. Distinguish at least four categories:
- Validation failure: malformed model output or invalid tool arguments; usually do not retry blindly.
- Transient infrastructure failure: timeout, unavailable dependency, or rate limit; retry with backoff.
- Business rejection: policy denied the action; record the reason and stop or request approval.
- Unknown outcome: the process may have crashed after an external side effect; reconcile before retrying.
Use correlation IDs across HTTP, gRPC, queue messages, database records, and model metadata. OpenTelemetry traces should include agent name, workflow ID, step ID, model version, token counts, queue wait, tool latency, and retry count. Avoid storing raw prompts by default when they may contain sensitive data; store hashes, redacted payloads, or encrypted artifacts with controlled retention.
Monitor more than model latency. The important metrics are:
- Workflow completion rate by agent and business outcome.
- P50, P95, and P99 end-to-end latency.
- Queue age, queue depth, and lease-expiration count.
- Tool-call success rate and duplicate side-effect rate.
- Model validation failure and fallback rate.
- Token usage and cost per successful workflow.
- Human approval rate and post-approval reversal rate.
- Event projection lag and reconciliation backlog.
- Policy violations, prompt-injection detections, and unauthorized tool attempts.
Create automated evaluations from production traces. Test factuality, tool selection, policy compliance, refusal behavior, and recovery after partial failure. A model upgrade should be treated like a distributed system change: canary it, compare outcome metrics, and retain a rollback path.
Conclusion
Reliable multi-agent systems software architecture is built from explicit authority boundaries, durable state transitions, typed contracts, and controlled side effects. Agents should make decisions, but deterministic services should validate permissions, execute tools, and commit business state.
Use orchestration for critical workflows, events for decoupled reactions, gRPC for controlled internal calls, REST for public boundaries, and event sourcing only where auditability and replay justify its complexity. Separate business state from model context, enforce idempotency at every external boundary, and make unknown outcomes recoverable through reconciliation.
The operational invariant is simple: every decision must be explainable, every side effect must be attributable, every retry must be safe, and every workflow must be restartable. If those properties are visible in your schemas, protocols, traces, and metrics, adding more agents becomes an incremental scaling problem rather than an architectural gamble.
