← BACK_TO_LOGS

End-to-End AI Workflow Automation Architecture

•11 min read•KANAT NAZAROV
End-to-End AI Workflow Automation Architecture

End-to-End AI Workflow Automation Architecture

An AI workflow becomes an engineering problem the moment it must run without supervision. A demo can call a language model, generate prose, and write a document to a CMS. A production system must additionally answer harder questions: What happens when the model times out after producing a response? How do we prevent duplicate publishing when a webhook is retried? How do editors override an incorrect decision? How do we trace one user request across Next.js, a queue, an orchestration platform, several model calls, and Sanity CMS?

The correct design is not a single prompt chained to a CMS API. It is an AI workflow automation full stack: a set of synchronous APIs, durable state transitions, asynchronous workers, provider adapters, validation layers, and operational controls. The system should treat AI output as an untrusted intermediate artifact and publishing as a controlled distributed transaction.

This article describes an architecture for an automated AI publishing pipeline built around Next.js 15, a relational database, a queue or workflow engine, Make.com custom HTTP webhooks, and an Automated content engine Sanity CMS. The specific vendors can change. The invariants should not.

The Architectural Boundary: Control Plane Versus Data Plane

The first useful separation is between the control plane and the data plane.

The control plane owns intent, permissions, workflow state, idempotency, policy, and observability. It answers: who requested this run, what stage is it in, which version of the prompt was used, and is publication allowed? The data plane performs expensive or asynchronous work: research, retrieval, generation, image creation, fact checking, formatting, and delivery to external systems.

A typical request path looks like this:

  1. A user submits a topic, content type, locale, and desired publication date through a Next.js 15 interface.
  2. A low-latency API validates the request, authenticates the caller, creates a workflow run, and returns
    202 Accepted
    .
  3. The API enqueues a job or invokes a Make.com custom HTTP webhook with a signed payload.
  4. The orchestrator executes research, generation, structured validation, editorial checks, and asset creation.
  5. A dedicated publisher converts the approved internal document into Sanity mutations.
  6. Sanity returns a durable document identifier and revision metadata.
  7. The system records the result and emits a notification or schedules the next stage.

The initial HTTP request should not wait for model generation. Even a fast model call is subject to provider throttling, network variance, tool execution, and retries. A synchronous endpoint that waits 30 seconds creates unnecessary connection pressure and makes failure semantics ambiguous. Returning a run identifier immediately allows the frontend to poll, subscribe through Server-Sent Events, or refresh through a webhook-driven status update.

A minimal domain model might include these entities:

  • workflow_runs
    : one row per user-requested execution.
  • workflow_steps
    : individually retryable stages with attempt counts and provider metadata.
  • content_documents
    : the canonical internal representation of a draft.
  • idempotency_keys
    : request fingerprints and their associated result.
  • outbox_events
    : committed events waiting to be delivered to external systems.
  • audit_events
    : immutable records of state transitions and human decisions.

The database is the source of truth for workflow state. Make.com and Sanity are integrations, not authoritative state stores.

State Machines and Distributed Failure Semantics

AI pipelines fail in ways that ordinary CRUD applications do not. A model may return syntactically valid JSON containing unsupported claims. An external call can succeed while its response is lost. A retry can create a second CMS document. A webhook can arrive twice or out of order. These are distributed systems problems, regardless of whether the business feature is called content automation.

Represent the workflow as an explicit state machine rather than a set of loosely related booleans. For example:

queued -> researching -> drafting -> validating -> review_required
                                             -> approved -> publishing -> published
                                             -> rejected
Any active state -> failed_retryable -> queued
Any state -> cancelled

Each transition should be conditional on the current version of the row. Optimistic concurrency prevents two workers from advancing the same run simultaneously:

UPDATE workflow_runs
SET status = 'drafting', version = version + 1, updated_at = now()
WHERE id = $1
  AND status = 'researching'
  AND version = $2;

If the affected-row count is zero, the worker must stop and reload state. It must not continue based on stale assumptions.

Retries also require classification. Retryable failures include connection resets, HTTP 429 responses, provider 5xx responses, and temporary database unavailability. Non-retryable failures include invalid credentials, unsupported schema, policy violations, and malformed user input. Use exponential backoff with jitter, for example 2, 4, 8, 16, and 32 seconds, bounded by a maximum delay. Do not retry a failed model response indefinitely; cap attempts and move the run to

review_required
or
failed
with a diagnostic reason.

The most important pattern is idempotency. Every externally visible operation needs a deterministic key, such as:

const publishKey = `publish:${runId}:${contentVersion}`;

Persist the key before making the external request. If the request is retried, the publisher checks whether the key already has a Sanity document ID. For Sanity, use a deterministic document ID derived from the internal content ID rather than allowing every retry to create a random document:

const documentId = `article.${contentId}`;

A create-if-not-exists mutation or a conditional patch can then make publication effectively idempotent. Publication is not literally atomic across your database and Sanity, so record an intermediate

publishing
state and reconcile ambiguous outcomes by querying Sanity before retrying.

Designing the AI Workflow Automation Full Stack

Next.js 15 as the Control Surface

Next.js 15 is well suited to the operator and authoring surface, but it should not become the workflow engine. Route handlers or server actions should perform authentication, authorization, validation, and transaction creation. They should not contain long-running model orchestration.

A request handler should be deliberately boring:

export async function POST(request: Request) {
  const user = await requireUser(request);
  const input = createRunSchema.parse(await request.json());
  const key = request.headers.get('idempotency-key');

  const result = await runs.create({
    userId: user.id,
    input,
    idempotencyKey: key,
  });

  return Response.json(
    { runId: result.runId, status: result.status },
    { status: result.reused ? 200 : 202 },
  );
}

Use runtime validation with Zod or an equivalent schema library. TypeScript types alone do not protect an HTTP boundary. Apply rate limits per user, organization, and IP. Store prompt templates and model configuration by version, so a later investigation can reproduce which instructions, tools, and temperature settings generated a document.

For status updates, polling is often sufficient for low-volume systems. At scale, publish state changes through a message broker and expose Server-Sent Events. WebSockets are useful for bidirectional collaboration, but they add connection-management complexity that most publishing workflows do not need.

Make.com Custom HTTP Webhooks as an Integration Edge

Make.com is effective for connecting SaaS systems, notifications, approvals, and low-code business rules. Its webhook should be treated as an integration boundary, not as the canonical workflow database.

A robust webhook contract includes a schema version, event ID, timestamp, tenant ID, run ID, and signature:

{
  "eventId": "evt_01J...",
  "schemaVersion": 1,
  "eventType": "content.run.created",
  "runId": "run_123",
  "tenantId": "team_456",
  "occurredAt": "2025-01-15T10:30:00Z",
  "payload": {
    "contentId": "cnt_789"
  }
}

Sign the raw request body with HMAC-SHA256. The receiver verifies the signature and rejects stale timestamps to reduce replay risk. Make scenarios should begin with a fast acknowledgment path. If downstream processing is slow, the webhook handler should enqueue work and return immediately rather than holding the Make request open.

Make may retry delivery, and network intermediaries can create duplicate requests. Deduplicate by

eventId
in a database table with a unique constraint. Do not rely only on a Make scenario filter; the database constraint is the durable guarantee.

Use Make for orchestration when the process is primarily integration-centric and has moderate volume. Move latency-sensitive, high-volume, or highly stateful logic into application workers. A visual scenario becomes difficult to reason about when it contains dozens of branches, model-specific recovery paths, and transactional requirements.

Sanity as the Publishing Projection

Sanity should contain the editorial projection of a document, not every intermediate model trace. Keep prompts, raw provider responses, tool logs, and evaluation artifacts in controlled internal storage. Store only the fields required for editing, review, localization, and rendering in Sanity.

A useful content schema includes:

  • Stable internal content ID.
  • Workflow run ID and generation version.
  • Title, summary, body blocks, and SEO metadata.
  • Editorial status such as draft, review, approved, or published.
  • Source references and claim-review status.
  • Generation timestamp and model identifier.
  • Scheduled publication time.

Sanity’s block content model is preferable to pushing unstructured HTML when editors need to modify sections. Convert model output into a validated Portable Text representation. Reject unsupported marks, invalid references, oversized blocks, and malformed links before sending mutations.

The Automated content engine Sanity CMS integration should use least-privilege credentials, environment-specific datasets, and a dedicated publishing client. Draft and production content must be separated. A worker should never publish merely because generation succeeded; it should publish only after policy and editorial gates have passed.

Production Protocols and Implementation

The following protocols make the automated AI publishing pipeline operationally credible.

  • Transactional outbox: Write the workflow transition and an outbound event in one database transaction. A separate dispatcher delivers the event to Make, a queue, or a notification service. This prevents a successful database commit from losing its corresponding message.
  • Inbox deduplication: Store received event IDs with a unique index. Mark the event processed only after the associated state transition commits.
  • Schema enforcement: Require structured model output using JSON Schema or provider-native structured output. Validate again in your application because provider guarantees are not a substitute for business validation.
  • Prompt and model versioning: Store
    promptVersion
    ,
    model
    ,
    temperature
    , tool versions, and retrieval configuration on every run.
  • Budget controls: Set maximum token, tool-call, image-generation, and wall-clock budgets per run. Abort or escalate when a workflow exceeds them.
  • Human gates: Require review for regulated topics, unsupported claims, low confidence, missing citations, or changes to already published documents.
  • Secret isolation: Keep model keys and Sanity tokens in a secret manager. Never include them in browser bundles, Make payloads, logs, or generated content.
  • Payload limits: Enforce maximum topic length, attachment size, webhook body size, and generated document size before expensive processing.
  • Dead-letter handling: After the retry budget is exhausted, move the step to a dead-letter queue with the last error, correlation ID, and safe replay command.

A worker should use timeouts at every network boundary. One global timeout is insufficient because research, model generation, and CMS mutation have different latency profiles. Circuit breakers prevent a failing provider from consuming all worker capacity. Bulkheads isolate expensive image generation from lightweight text validation.

Observability must be tied to a correlation ID propagated through HTTP headers, queue messages, logs, and Sanity metadata. Track at least:

  • End-to-end completion rate and time from request to publication.
  • Per-step latency, retry count, timeout rate, and queue wait time.
  • Model token usage, cost per successful document, and provider error rate.
  • Percentage of runs requiring human review.
  • Duplicate webhook and duplicate publication attempts.
  • Validation rejection rate and post-publication correction rate.
  • Sanity mutation failures and reconciliation lag.

Do not log complete prompts or generated documents by default. They may contain personal, confidential, or licensed information. Log hashes, sizes, versions, and structured error categories, with access-controlled payload sampling for debugging.

Scaling, Consistency, and Security Trade-offs

The database workload is usually write-heavy during orchestration and read-heavy in dashboards. Index

workflow_runs
by tenant, status, and updated time. Index steps by status and next-attempt time for polling workers. Use connection pooling, but size it against database capacity rather than worker count. If the audit log becomes large, partition it by month or archive it to object storage.

Queue-based scaling is preferable to spawning unbounded serverless tasks. Set concurrency separately for each provider. For example, a model provider may allow high request concurrency while Sanity mutation throughput remains constrained. Backpressure should be explicit: when the publishing queue is full, keep runs in

approved
rather than pretending they are published.

Consistency must be visible to users. The UI should distinguish

queued
,
generating
,
review_required
,
publishing
, and
published
. Do not show a success notification when the database says approved but the CMS projection is still pending. Eventual consistency is acceptable when it is represented honestly and reconciled automatically.

Security includes prompt-injection resistance. Treat retrieved web pages, documents, and tool output as untrusted data. Delimit sources, prohibit tools from changing authorization state, and require deterministic application-side checks for publication. The model may recommend publication; only the control plane can authorize it.

Conclusion

A reliable end-to-end AI workflow automation architecture is a distributed system with probabilistic components. The core design is therefore based on deterministic boundaries: a transactional control plane, explicit state machines, idempotent external operations, durable queues, versioned prompts, schema validation, and human gates.

Use Next.js 15 for a responsive control surface, not long-running orchestration. Use Make.com custom HTTP webhooks for integration edges with signed, versioned, deduplicated events. Use Sanity as an editorial projection backed by a publisher that can safely reconcile ambiguous outcomes. Keep model output untrusted until it passes structural, policy, and editorial validation.

The operational definition of success is not that a model generated text. It is that every run can be explained, retried safely, cancelled, reviewed, and reconciled; that duplicate delivery does not create duplicate content; and that engineers can measure latency, cost, quality, and failure recovery. Those invariants turn an AI demo into an AI workflow automation full stack that can operate at production scale.

Leave a comment

Kanat Nazarov

Kanat Nazarov

Systems & Full-Stack Engineer
Kanat Nazarov is a product-minded Full-Stack Engineer. He is the co-founder of GetFusionChat, an enterprise-grade multi-tenant communication SaaS platform. Beyond architecting low-latency infrastructures and advanced AI systems, Kanat is a dedicated student of peak performance—optimizing his lifestyle through rigorous fitness, precise nutrition, and creative production in music. He operates on the core philosophy that breakthrough technical results stem from a high-functioning, deeply disciplined life.