Building Autonomous AI Agents with OpenAI

Building Autonomous AI Agents with OpenAI: Architecture, State, and Production Reliability
Autonomous agents are often introduced as a prompt plus a loop: send a user request to a model, execute the returned function, append the result, and repeat. That prototype can work in a notebook. It fails in production because autonomy is not the same as reliability. The moment an agent can mutate data, call external services, or spend money, it becomes a distributed system with an uncertain decision-maker inside it.
The real engineering problem in building AI agents with OpenAI API is controlling nondeterminism at the boundaries. The model may select the wrong tool, produce a valid but unsafe argument, retry an operation that already succeeded, or consume the entire latency budget with a long reasoning loop. Network failures, duplicate webhooks, stale user state, rate limits, and partial database writes remain normal distributed-systems problems; the language model adds another probabilistic component to the system.
A production design therefore needs explicit state, typed capabilities, bounded execution, idempotency, policy enforcement, and observability. The model proposes actions. Application code validates and authorizes them. Durable infrastructure records what happened.
The Agent Is a Control Plane, Not the System of Record
An agent should be modeled as a control loop around a durable workflow:
- Receive an intent and authenticate the caller.
- Load the relevant conversation, account, and workflow state.
- Ask the model for either a user-facing response or a typed tool call.
- Validate the proposed call against a server-owned schema and policy.
- Execute the tool with a deadline and idempotency key.
- Persist the tool result and transition state atomically.
- Continue for a bounded number of turns or return a response.
The model is responsible for selecting among capabilities and composing information. It must not be the authority for identity, permissions, pricing, inventory, or transaction status. Those facts belong in databases and services that the application controls.
A useful state model separates immutable events from a materialized session view. Store user messages, assistant messages, tool calls, tool results, policy decisions, and workflow transitions as append-only records. Then maintain a compact conversation projection for fast reads. PostgreSQL is usually sufficient at the beginning: partition large event tables by tenant or time, index session identifiers, and use optimistic concurrency on the projection.
For high-volume deployments, Redis can cache the hot conversation projection, while PostgreSQL or an event store remains authoritative. Do not make Redis the only record of an agent run. A cache eviction or failover should increase latency, not erase the workflow.
A minimal relational model might contain:
- : run ID, tenant ID, status, version, deadline, and current step.
agent_runs - : ordered input, output, tool, error, and policy events.
agent_events - : tool name, canonical arguments, idempotency key, status, result reference, and timestamps.
tool_invocations - : current summary, token budget, last event sequence, and retention metadata.
conversations
The
versionOpenAI Tool Calling with Bounded Autonomy
The safest interpretation of function calling is that the model emits a proposal in a constrained language. The server translates that proposal into a real operation only after validation. This is the core of an OpenAI GPT-4o function calling tutorial, but production systems need several additional controls beyond the API example.
The following TypeScript pattern uses the Chat Completions interface with GPT-4o. The same architectural rules apply when using the newer Responses API.
import OpenAI from 'openai';
import { z } from 'zod';
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const tools = [
{
type: 'function' as const,
function: {
name: 'get_order_status',
description: 'Read the current status of an order owned by the authenticated user.',
parameters: {
type: 'object',
additionalProperties: false,
properties: {
orderId: { type: 'string', minLength: 1, maxLength: 64 }
},
required: ['orderId']
}
}
}
];
const GetOrderStatus = z.object({
orderId: z.string().min(1).max(64)
});
export async function runAgent(messages: OpenAI.Chat.ChatCompletionMessageParam[]) {
let history = messages;
for (let turn = 0; turn < 6; turn++) {
const completion = await openai.chat.completions.create({
model: 'gpt-4o',
messages: history,
tools,
tool_choice: 'auto',
temperature: 0.1,
max_tokens: 700
});
const message = completion.choices[0]?.message;
if (!message) throw new Error('empty_model_response');
history.push(message);
const calls = message.tool_calls ?? [];
if (calls.length === 0) return message.content ?? '';
for (const call of calls) {
if (call.type !== 'function' || call.function.name !== 'get_order_status') {
throw new Error('unsupported_tool');
}
let args: unknown;
try {
args = JSON.parse(call.function.arguments);
} catch {
throw new Error('invalid_tool_json');
}
const parsed = GetOrderStatus.safeParse(args);
if (!parsed.success) throw new Error('invalid_tool_arguments');
const result = await getOrderStatusForCurrentUser(parsed.data.orderId);
history.push({
role: 'tool',
tool_call_id: call.id,
content: JSON.stringify(result)
});
}
}
throw new Error('agent_turn_limit_exceeded');
}
Several details are non-negotiable:
- Set and validate again with Zod, JSON Schema, or an equivalent runtime validator.
additionalProperties: false - Bind authorization to the authenticated request context, not to arguments generated by the model. A tool called should receive the user identity from the server.
get_order_status - Use separate tools for reads and writes. A destructive action should never be hidden behind a generic database tool.
- Require confirmation for irreversible or financially significant actions. The confirmation should be a server-issued capability or workflow state, not merely the phrase yes in conversation history.
- Limit turns, tool calls per turn, total wall-clock time, and cumulative spend.
- Record the exact tool name, canonical arguments, model, prompt version, and policy result for replay and audit.
For writes, use an idempotency key derived from the durable run and logical action, such as
runId:stepIdState Synchronization and Workflow Execution
A low latency AI conversational agent should not execute every operation synchronously inside the HTTP request. Separate the interactive path from the durable workflow path.
The synchronous path should handle authentication, context retrieval, one or two fast model calls, read-only tools, and streaming the response. Long-running tasks such as document processing, browser automation, CRM synchronization, or approval workflows should become jobs. Return a run identifier and stream progress through Server-Sent Events or WebSockets.
A practical architecture for Next.js 15 looks like this:
- Next.js 15 App Router provides the chat UI, route handlers, authentication boundary, and streaming response endpoint.
- An agent service owns model orchestration, tool schemas, policy checks, and prompt versions.
- PostgreSQL stores conversations, events, workflow state, and idempotency records.
- Redis handles rate limiting, short-lived locks, and hot projections.
- A queue such as SQS, BullMQ, or Kafka runs durable background steps.
- Specialized microservices own payments, search, messaging, and third-party integrations.
- OpenTelemetry correlates the browser request, model calls, tool invocations, and queue jobs.
Do not hold a database transaction open while waiting for OpenAI or a third-party API. Persist an intent, commit, call the external dependency, and persist the outcome in a second transaction. For workflows that require stronger coordination, use a saga: each step has a forward action, a compensating action where possible, and a durable status.
Exactly-once execution is generally unavailable across independent services. Design for at-least-once delivery with idempotent consumers. A unique constraint on
tool_invocations(idempotency_key)Conversation history also requires deliberate management. Sending the entire transcript on every turn increases latency and token cost. Keep recent messages, a structured account snapshot, and a model-generated summary that is treated as lossy cache rather than truth. Retrieve authoritative facts from tools. Summaries should never be used as the source for balances, permissions, order status, or legal commitments.
Production Protocols and Implementation
The first production protocol is a request budget. Define a deadline at ingress and propagate it through every call. For an interactive request, a reasonable initial budget might be 2.5 seconds at the API boundary, with 200 milliseconds reserved for persistence and response serialization. If the deadline is exceeded, return a useful partial response or a run ID instead of allowing unbounded work.
Example configuration:
const policy = {
maxTurns: 6,
maxToolCalls: 8,
requestTimeoutMs: 2500,
modelTimeoutMs: 1800,
maxInputTokens: 12000,
maxOutputTokens: 700,
maxRetries: 2
};
Retries must be selective. Retry connection resets, 429 responses, and transient 5xx responses with exponential backoff and jitter. Do not retry invalid arguments, policy denials, authentication failures, or a tool whose side effect has unknown completion status unless its idempotency contract is explicit.
For Next.js streaming, emit structured events rather than only text tokens. A client should be able to distinguish a draft response, a tool request, a progress update, a final answer, and an error. This makes reconnects possible. The client sends the last event ID when reconnecting, and the server resumes from the durable event sequence instead of rerunning the model blindly.
Security requires treating all retrieved text as untrusted input. Prompt injection can arrive through a support ticket, web page, PDF, or calendar event. Use a capability-oriented tool layer:
- Classify tools as read, reversible write, and irreversible write.
- Apply tenant and user authorization in code for every invocation.
- Strip secrets and unnecessary personal data from model context.
- Use allowlisted domains and network egress policies for browsing tools.
- Require human approval for high-impact actions.
- Log policy denials without logging raw credentials or sensitive payloads.
An AI workflow automation full stack should also have a human control plane. Operators need to inspect a run, see the exact event timeline, replay a failed read-only step, cancel a job, and disable a tool or prompt version globally. Feature flags should control model versions, tool availability, and autonomy levels independently.
Measure the system as a distributed application, not just as a chatbot. Important metrics include:
- p50, p95, and p99 time to first token and time to final response.
- Model latency, queue latency, database latency, and external API latency separately.
- Tool-call success rate, validation rejection rate, policy-denial rate, and retry count.
- Agent completion rate within the turn and wall-clock budgets.
- Duplicate side-effect rate and idempotency conflict rate.
- Tokens per successful task, cost per run, and cost by tenant.
- Human escalation rate and user correction rate.
- Stale-state conflicts, dropped stream reconnects, and dead-letter queue depth.
Trace IDs should connect the user request to the model request, tool invocation, database event, and downstream service call. Store prompt and tool-schema versions so a production decision can be reproduced even after the application changes.
Evaluation should combine deterministic tests with scenario tests. Unit-test validators, authorization, idempotency, and state transitions. Build a fixture set containing ambiguous requests, prompt injections, malformed tool arguments, duplicate webhooks, timeouts, stale conversations, and partial failures. Run it on every prompt, model, or tool-schema change. A higher benchmark score is irrelevant if the new prompt increases unauthorized tool proposals or causes longer workflows.
Conclusion
Autonomous agents become dependable when autonomy is bounded by architecture. The model should propose typed actions, while application code owns identity, authorization, state transitions, and side effects. Durable events, optimistic concurrency, idempotent tools, bounded loops, deadlines, and selective retries convert probabilistic behavior into an operationally manageable workflow.
For teams building AI agents with OpenAI API, the highest-leverage work is rarely another prompt refinement. It is designing the state machine around the prompt. Start with read-only capabilities, instrument every transition, add human approval for irreversible actions, and move slow work to durable queues. Monitor latency, cost, tool reliability, duplicate effects, escalation, and stale-state conflicts. Those invariants are what let a low latency AI conversational agent evolve into a reliable automation platform rather than an expensive source of unpredictable side effects.
