← BACK_TO_LOGS

Integrating Large Language Models into Web Systems

•10 min read•KANAT NAZAROV
Integrating Large Language Models into Web Systems

Integrating Large Language Models into Web Systems: A Production Architecture

Integrating Large Language Models into Web Systems is not primarily an API-integration task. The first prototype is usually a single server action that sends a user message to a model and returns generated text. Production systems are different: they must preserve authorization boundaries, control spend, tolerate provider failures, synchronize conversational state, stream partial results, validate model output, and remain observable under changing model behavior.

The architectural mistake is treating an LLM as a deterministic application dependency. A relational database generally returns the same result for the same committed query. An LLM may produce different output for the same input, consume variable tokens, fail with provider-specific errors, and return syntactically valid but semantically unsafe data. It is better modeled as a probabilistic, high-latency, metered remote service operating behind a policy boundary.

This distinction changes where the integration belongs. The browser should not hold provider credentials or decide which tools a model may invoke. The application should not allow arbitrary generated text to become an SQL statement, authorization decision, or durable state mutation. A robust design places an AI middleware layer between product workflows and model providers, with explicit contracts for prompts, tools, budgets, retries, state, and validation.

The Core Architecture: Treat the Model as an Untrusted Decision Component

A practical request path for integrating LLMs into web applications looks like this:

  1. The client sends a user action to a Next.js 15 route handler or backend API.
  2. The application authenticates the user and resolves tenant, role, and resource permissions.
  3. An AI orchestration service loads the minimum required context from application databases and search indexes.
  4. A policy layer selects the model, prompt version, tool set, token budget, and timeout.
  5. The provider adapter makes the model request, optionally using streaming transport.
  6. The response is validated, persisted, and emitted to the client through Server-Sent Events or a WebSocket.
  7. Usage, latency, refusal, tool calls, and errors are recorded independently of the user-facing response.

The key boundary is the provider adapter. It normalizes different vendor APIs into an internal interface and prevents provider-specific assumptions from spreading through the product codebase.

export type ModelRequest = {
  tenantId: string;
  userId: string;
  conversationId: string;
  messages: Message[];
  tools: ToolDefinition[];
  maxOutputTokens: number;
  traceId: string;
};

export type ModelResult = {
  text: string;
  toolCalls: ToolCall[];
  inputTokens: number;
  outputTokens: number;
  finishReason: string;
  providerRequestId?: string;
};

export interface ModelProvider {
  generate(request: ModelRequest, signal: AbortSignal): Promise<ModelResult>;
  stream(request: ModelRequest, signal: AbortSignal): AsyncIterable<ModelEvent>;
}

This interface supports model routing without coupling business logic to one vendor. A low-cost model can handle classification and summarization, while a stronger model handles code generation or multi-step reasoning. Routing should be policy-driven, not hard-coded in UI components.

Synchronous and asynchronous work

Not every LLM task belongs on the request path. Interactive completion, extraction of a few fields, and short answers can use a low-latency API. Large document analysis, batch classification, report generation, and repository indexing should be represented as durable jobs.

For asynchronous workflows, write a job record before dispatching work to a queue. The worker must be idempotent: retries are normal, and a provider timeout does not prove that the provider did not complete the request. Use an idempotency key derived from tenant, operation, input version, and attempt-independent workflow identity.

const job = await db.aiJob.create({
  data: {
    idempotencyKey,
    type: 'document-extraction',
    status: 'queued',
    inputHash,
    tenantId,
  },
});

await queue.publish('ai.jobs', {
  jobId: job.id,
  deduplicationKey: idempotencyKey,
});

The worker claims the job with a lease, persists intermediate state, and records the provider request ID. If the process dies after the provider responds but before the database commit, reconciliation can inspect the request log or safely re-run with the same idempotency key where the provider supports it.

State Synchronization and Context Management

Conversation history is application state, not model state. A model API usually receives a serialized context on every request, so the system must define how messages, summaries, tool results, and user-visible events are stored. Storing only the final assistant text is insufficient for auditability and debugging.

A useful schema separates immutable events from derived conversation projections:

  • conversation
    : tenant ownership, participants, model policy, and lifecycle status.
  • message
    : role, content reference, token estimate, prompt version, and visibility.
  • tool_call
    : requested arguments, authorization result, execution status, and output reference.
  • generation
    : provider, model, latency, token usage, finish reason, and trace ID.
  • conversation_summary
    : compressed context with the source message range and summary version.

Use an append-only event record for the authoritative history, then maintain a materialized view for fast reads. This prevents concurrent browser tabs from overwriting messages through last-write-wins updates. Optimistic concurrency can reject stale writes using a conversation version number.

Context windows are a database and product problem as much as a model problem. Sending all historical messages increases latency and cost, while aggressive summarization can remove constraints that matter. A production context builder should combine:

  • Recent turns, because local conversational intent is usually most important.
  • A durable summary of older turns, including unresolved decisions and user preferences.
  • Retrieved domain facts, filtered by authorization before retrieval results reach the prompt.
  • The current user request and explicit output contract.
  • Tool definitions required for this turn, rather than every tool available to the product.

Prompt engineering for software developers should therefore be treated as interface design. A prompt is a versioned program specification with inputs, constraints, failure behavior, and output schema. Store prompts in source control or a prompt registry, assign versions, and evaluate changes against a fixed test set before rollout.

Avoid constructing prompts through uncontrolled string concatenation. Use typed sections and clearly delimited untrusted data. Data retrieved from a document may contain instructions that attempt to override system policy. Delimit it as evidence, not as an instruction source, and ensure tool authorization is enforced outside the prompt.

const prompt = {
  system: renderTemplate('support-agent', { version: 7 }),
  context: [
    { type: 'summary', text: conversationSummary },
    { type: 'evidence', text: retrievedChunks },
  ],
  user: userMessage,
  output: {
    schema: SupportDecisionSchema,
    onInvalid: 'return_safe_fallback',
  },
};

AI Middleware Rate Limiting Architecture

LLM traffic has two independent resource dimensions: request concurrency and token consumption. A traditional requests-per-second limiter controls neither sufficiently. One request may consume a few hundred tokens; another may consume hundreds of thousands through a long context and tool loop.

Use layered controls:

  • Per-user request limits to prevent accidental loops and abusive clients.
  • Per-tenant token budgets to enforce commercial plans and cost allocation.
  • Global provider quotas to avoid account-level throttling.
  • Concurrency limits per model and operation class to protect latency.
  • Queue limits for asynchronous work to prevent unbounded backlog.
  • Circuit breakers that stop sending traffic to a failing or overloaded provider.

A token-aware limiter can use a Redis-backed token bucket. The request estimates input plus maximum output tokens before execution, reserves capacity atomically, and refunds unused output capacity after completion. Estimation is imperfect, so maintain a safety margin and reconcile actual usage from provider metadata.

const reservation = await limiter.reserve({
  key: `tenant:${tenantId}:tokens`,
  requested: estimateTokens(messages) + maxOutputTokens,
  limit: plan.tokenLimitPerMinute,
  refillPerSecond: plan.tokenLimitPerMinute / 60,
});

if (!reservation.allowed) {
  throw new RateLimitError({
    retryAfterMs: reservation.retryAfterMs,
    code: 'AI_TOKEN_BUDGET_EXCEEDED',
  });
}

Do not place all throttling inside a single application process. Horizontal scaling makes local counters inaccurate, and serverless instances are short-lived. Redis scripts, a distributed quota service, or a queue with explicit admission control are appropriate coordination mechanisms. The limiter itself needs a failure policy: fail closed for expensive premium operations, or fail open with a conservative local limit for low-risk features when Redis is unavailable.

Retries require special treatment. Retry only transient failures such as connection resets, provider overload, and selected 5xx responses. Use exponential backoff with jitter and a strict attempt budget. Never blindly retry invalid requests, safety refusals, authentication failures, or context-length errors. Streaming requests are especially difficult: if the connection breaks after partial output, retrying may duplicate content. Persist stream state and either resume through provider support or mark the generation as interrupted.

A circuit breaker should track failures by provider and model, not only by endpoint. Its states are closed, open, and half-open. When open, route to a fallback model only if the fallback meets the operation's quality and data policy. Silent fallback to a weaker model can create a product regression, so expose the selected model and fallback reason in internal telemetry.

Building Custom GPT Tools for Developers

Tools are the controlled bridge between a model and deterministic software capabilities. Building custom GPT tools for developers should begin with narrow, typed functions rather than a general-purpose shell or database interface.

A safe tool has four properties:

  1. Its input schema is strict and rejects unknown or ambiguous fields.
  2. Its authorization is evaluated using the authenticated principal, not model-supplied identity.
  3. Its side effects are explicit, auditable, and idempotent where possible.
  4. Its output is bounded, structured, and filtered for secrets or irrelevant data.
const getBuildStatus = defineTool({
  name: 'get_build_status',
  description: 'Returns the status of a build the current user can access.',
  inputSchema: z.object({
    projectId: z.string().uuid(),
    buildId: z.string().uuid(),
  }).strict(),
  execute: async ({ projectId, buildId }, context) => {
    await authorization.require(context.user, 'build:read', projectId);
    return buildService.getStatus({ projectId, buildId });
  },
});

The orchestration loop must limit tool depth, total execution time, and cumulative cost. A tool result should not automatically become a new unrestricted prompt. Mark it as tool output, enforce size limits, and redact credentials before reinserting it into the model context.

For mutating tools, separate proposal from execution. The model can generate a deployment plan, but the user or a policy engine must approve the final action. High-impact operations should require an idempotency key, a dry-run capability, and an audit record containing the exact arguments and authorization decision.

Tool discovery also affects prompt size. Expose only tools relevant to the current route and user role. A developer assistant may have repository search, test execution, and issue lookup, while a customer support agent should have none of these capabilities. Capability selection is an application concern, not a model decision.

Production Protocols and Implementation Workflow

A reliable implementation can be delivered in the following sequence:

  • Define the user-visible failure behavior before integrating a provider. Decide whether the feature can degrade to cached data, a deterministic workflow, or a human handoff.
  • Establish an internal provider interface and keep vendor SDK calls in one adapter package.
  • Create typed input and output schemas. Validate model output at runtime with a library such as Zod, JSON Schema, or Pydantic.
  • Add authentication, tenant isolation, quota checks, and tool authorization before the model call.
  • Implement streaming only after persistence and cancellation semantics are correct. A fast broken stream is worse than a slightly slower reliable response.
  • Version prompts, model policies, and tool schemas. Record versions with every generation.
  • Build an evaluation set containing normal requests, ambiguous requests, prompt-injection attempts, long contexts, provider failures, and adversarial tool arguments.
  • Load-test the middleware with realistic token distributions, not only request counts.
  • Introduce feature flags for model, prompt, and tool changes, then roll out by tenant or percentage.

For a Next.js 15 application, keep route handlers thin. They should parse the request, authenticate, invoke an application service, and translate domain events into a response. Do not embed prompt construction, retry loops, and tool execution directly in a React component or route handler. Use a separate worker or service boundary when execution can exceed the platform's request duration.

Observability must include structured traces across the browser request, API route, context retrieval, provider call, tool execution, and persistence transaction. Monitor:

  • Time to first token and total generation latency.
  • Input, output, and cached token counts.
  • Cost per request, tenant, feature, and successful business outcome.
  • Provider error rate, timeout rate, retry count, and circuit state.
  • Tool-call frequency, rejection rate, execution latency, and side-effect failures.
  • Schema-validation failures, refusal rate, fallback rate, and user correction rate.
  • Queue age, worker lease failures, and dead-letter volume.

Do not log raw prompts or outputs by default when they may contain personal, financial, or proprietary data. Use redaction, sampling, encrypted restricted logs, and short retention periods. Trace IDs and hashes often provide enough correlation for operations without copying sensitive content into every telemetry system.

Conclusion

Integrating LLMs into web applications succeeds when the model is isolated behind deterministic application controls. The durable invariants are straightforward: authorization never comes from generated text, tool access is least-privilege, conversation state is persisted outside the model, retries are bounded and idempotency-aware, quotas account for tokens as well as requests, and every prompt and model choice is versioned.

The operational goal is not merely higher answer quality. It is predictable system behavior under provider outages, long contexts, malicious inputs, traffic spikes, and evolving models. Measure latency, token consumption, cost, failure modes, tool safety, and business outcomes together. With that discipline, an LLM becomes a replaceable probabilistic component inside a fault-tolerant web architecture rather than an uncontrolled dependency at the center of the product.

Leave a comment

Kanat Nazarov

Kanat Nazarov

Systems & Full-Stack Engineer
Kanat Nazarov is a product-minded Full-Stack Engineer. He is the co-founder of GetFusionChat, an enterprise-grade multi-tenant communication SaaS platform. Beyond architecting low-latency infrastructures and advanced AI systems, Kanat is a dedicated student of peak performance—optimizing his lifestyle through rigorous fitness, precise nutrition, and creative production in music. He operates on the core philosophy that breakthrough technical results stem from a high-functioning, deeply disciplined life.