AI / Agent Platform

Case Study: AI Agent Platform

A product where customers give an assistant a task and it works on it: calling a model repeatedly, invoking tools, reading the customer's own documents, and producing a result minutes later. From an infrastructure standpoint this is a worker pipeline whose jobs are long, expensive, non-deterministic and hold credentials to systems that cost money — which makes it the worker pipeline with the sharpest edges. The agent behaviour itself belongs to the Agentic Engineering domain; what follows is how you *run* one in production.

Requirements

Accept a task, return immediately, and let the work run for minutes without holding a connection.
Call a hosted model API with credentials that are rotatable and never present in an image or a repository.
Execute model-chosen tools — including generated code — without giving them the platform's own privileges.
Ground answers in the customer's documents, isolated per tenant.
Attribute cost per run and per tenant, and stop a run that exceeds its budget.
Explain, after the fact, what a run did: every step, every tool call, every token.

Deliberately not requirements

Half of a design is what it refuses to do. These are the refusals.

Out of scope, on purpose

No model training or fine-tuning — this platform consumes models, it does not produce them.
No sub-second responses: the product's promise is minutes of autonomous work, not chat latency.
No self-hosting of the main reasoning model at this stage; that is a deliberate later decision with its own economics.

How the design got here

In order. Each stage leads with the problem that forced it.

Stage 1

Call the model inside the request

Forced by

The requirement, in its first form: a single-turn assistant endpoint. One model call, three seconds, a response. There is no queue, no worker, no run state and no tracing, and for single-turn question answering this remains the right design far longer than most teams believe. A great deal of AI infrastructure is built past this point without the problem that justifies it.

One hop out to the model provider. That is the entire platform.PROVIDER-NEUTRAL

Clientpublic

API instancesprivate— holds the model credential in memory; one outbound call per request

NAT gatewaypublic— the only path to the model provider — and a single point of failure for the whole product

Model APIpublic— metered per token; latency measured in seconds and highly variable

Client→API instances· POST /askcrosses boundary

API instances→NAT gateway

NAT gateway→Model API· inference requestcrosses boundary

Decision	Reason	Alternative	Trade-off
Synchronous, in-request model calls.	One call that fits inside an HTTP timeout needs no orchestration. Everything after this stage is machinery for work that does not fit in a request.	Asynchronous runs from the start, which is more infrastructure than the feature and delays learning whether anyone wants it.	Model latency is now request latency, and it is variable in a way ordinary backends are not — a slow provider minute is a slow product minute, and there is nothing you can do about it from inside the request.
Aggressive timeouts and a circuit breaker on the model call from day one.	The model provider is a third party on the critical path of every request. Without a bounded timeout, a provider slowdown becomes thread-pool exhaustion and takes down endpoints that never call a model.	A generous timeout so long requests can finish, which converts a provider incident into a full outage of your API.	Some legitimately long generations are cut off, and the timeout has to be re-tuned whenever prompts or models change. It is a permanent maintenance item.

Stage 2

Runs as durable objects on a queue

Forced by

The assistant became multi-step. A task calling three tools and the model six times took four minutes; the load balancer closed the connection at sixty seconds, the client treated it as a failure and retried, and the retry executed the same tool side effects a second time — including sending a customer email twice.

The run outlives the request that created it, and becomes something you can address, resume and audit.PROVIDER-NEUTRAL

Clientpublic

APIprivate— creates a run keyed by an idempotency key, returns 202 with a run id, streams status

Run storeprivate— run state, step history, current step — durable, so a worker crash is resumable

Run queueprivate— visibility timeout longer than the longest step; heartbeat to extend it

Agent runtime workersprivate— executes the loop: model → tool → model; checkpoints after every step

Model APIpublic

Client→API· POST /runs (idempotency key)crosses boundary

API→Run store· create run

API→Run queue· enqueue

Run queue→Agent runtime workers

Agent runtime workers→Run store· checkpoint each step

Agent runtime workers→Model API· inferencecrosses boundary

Decision	Reason	Alternative	Trade-off
A run is a durable object with an idempotency key, not a request.	Work that outlives a connection needs an identity. The key makes a client retry return the existing run instead of starting a second one that repeats every side effect.	Fire-and-forget background threads in the API process, which needs no new components and loses every in-flight run on the next deploy.	A state machine to design, store and expose in the API contract, plus the honest question of what a "partially completed" run means to a user — a question with no clean answer when the steps had external side effects.
Checkpoint after every step, and make steps individually replayable.	Steps are minutes long and expensive. A worker lost at step five should resume at step five, not re-run four model calls you have already paid for.	Restart the run from the beginning on failure, which is far simpler and multiplies both cost and latency on every interruption.	Every step must be checkpointable, which constrains how the agent loop is written, and replaying a step whose side effect already happened requires the tools themselves to be idempotent — application work, not infrastructure.
Queue visibility timeout longer than the longest step, with a heartbeat.	A run that takes minutes will otherwise be redelivered mid-flight and executed twice concurrently, which for an agent means duplicated tool calls in the real world.	A short timeout, which recovers faster from crashes and duplicates long-running work.	A long timeout delays recovery from a genuine worker crash; the heartbeat mitigates that and adds a failure mode of its own, where a hung worker keeps heartbeating a run that is going nowhere.

Stage 3

Model credentials as managed secrets with spend caps

Forced by

The model API key was baked into the worker image, so anyone with registry pull access had it — and the image was mirrored into a staging account with wider access. The realization that landed: a leaked model key is not only a data-access problem, it is a *metered billing instrument*. A stolen key is somebody spending your money at machine speed.

Credentials become fetched, scoped, capped and observable.PROVIDER-NEUTRAL

Agent workersprivate— no credential in the image, the environment or on disk

Worker identityinternal— may read exactly the model credential for its own environment

Secret managerprivate— per-environment model keys; every read recorded in the audit trail

Model gatewayprivate— the only component holding the provider key: adds per-tenant budgets, rate limits, retries and token accounting

Model APIpublic

Spend alertsprivate— per-key and per-tenant thresholds, alerting on rate as well as total

Agent workers→Worker identity· assume

Agent workers→Model gateway· inference request with tenant + run id

Model gateway→Secret manager· read provider key

Model gateway→Model APIcrosses boundary

Model gateway→Spend alerts· token and cost accounting

Decision	Reason	Alternative	Trade-off
An internal model gateway that is the only holder of the provider key.	It gives one place to rotate the credential, one place to enforce per-tenant budgets and rate limits, one place to count tokens, and one place to swap providers. Without it, every one of those becomes a change in every worker.	Each worker calls the provider directly with its own credential, which is one less hop and scatters the key, the accounting and the rate limiting across the fleet.	A new component on the critical path of every inference call, which must be highly available and adds latency. It is also a chokepoint: a bug in it stops all AI functionality at once.
Separate keys per environment, with independent spend caps.	It bounds the financial blast radius of a leak and stops a staging load test from consuming the production budget — a genuinely common way to discover you had one budget.	A single organizational key, which is simpler to manage and makes every incident a full-spend incident.	More keys to rotate, more places for a mismatch to cause a confusing "unauthorized" in one environment only.
Alert on spend rate, not only on cumulative spend.	A monthly budget alert fires long after a leaked key or a runaway loop has done its damage. Tokens per minute against a baseline detects both within minutes.	Provider budget alerts on the monthly total, which are free, easy, and lagging by days.	A rate baseline must be maintained as usage grows, and it will page during legitimate spikes — a customer running a large batch looks exactly like a runaway loop until someone looks.

Stage 4

Tool execution in a sandbox

Forced by

A tool that executed model-generated code ran inside the worker process, with the worker's network access and the worker's identity. That put the platform one prompt injection away from an agent reading the production database credential out of the environment and exfiltrating it through an ordinary outbound HTTP call. Nothing had been exploited; the design simply had no boundary in it.

The trust boundary the whole platform depends on: model-influenced code runs with nothing of yours.PROVIDER-NEUTRAL

Agent runtimeprivate— orchestrates the loop; holds identity and secrets

Tool brokerprivate— validates the tool call against a schema and an allowlist before anything executes

Execution sandboxinternal— ephemeral, no ambient credentials, read-only root, CPU/memory/wall-clock limits, destroyed after one call

Egress allowlist proxyprivate— the sandbox reaches only explicitly permitted destinations; everything else is refused and logged

Platform secretsprivate— unreachable from the sandbox — by network policy and by identity, not by convention

⚠ the asset the sandbox exists to protect

Approved tool endpointspublic

Agent runtime→Tool broker· proposed tool call

Tool broker→Execution sandbox· execute with scoped, short-lived token

Execution sandbox→Egress allowlist proxy· outbound attempt

Egress allowlist proxy→Approved tool endpoints· allowed destinations onlycrosses boundary

Decision	Reason	Alternative	Trade-off
Model-influenced execution happens in a separate compute boundary with no ambient credentials.	Treat every tool input as attacker-controlled, because a prompt injection in a customer document makes it exactly that. The only durable defence is that the code has nothing worth stealing in its environment.	Run tools in-process with careful input validation, which is faster and cheaper and relies on being able to enumerate every hostile input — the assumption that fails.	Per-call sandbox startup adds latency to every tool invocation, and you now operate a container isolation story with its own patching and escape surface. This is the most expensive stage in the design, in both compute and complexity, and it is the one that is not optional.
Egress from the sandbox goes through an allowlist proxy.	Exfiltration needs a path out. An allowlist turns "the agent can call anything" into "the agent can call the four things we approved", and every denied attempt becomes a signal.	Open egress with monitoring, which detects exfiltration after it has happened.	Every new tool integration needs an allowlist change, which is friction that engineers will route around if the process is slow. The proxy is also a single point of failure for all tool calls.
Hard CPU, memory and wall-clock limits per tool call, enforced by the platform.	Generated code loops. Without a limit, one tool call can hold an instance indefinitely, which is both a cost incident and a capacity incident.	Trust the tool implementation to bound itself, which works for the tools you wrote and not for the code the model wrote.	Legitimate long-running tools must be redesigned as asynchronous jobs of their own, and the limits need tuning per tool — a permanent source of "it works locally" reports.

Stage 5

Retrieval over customer documents

Forced by

Answers were confidently wrong because the model had no access to the customer's own material. Putting the whole corpus in the prompt was impossible for large tenants and, for the ones where it fit, made a single run cost more than the customer paid in a month.

A second stateful system, with per-tenant isolation as its first requirement.PROVIDER-NEUTRAL

Document storageprivate— customer uploads, per-tenant prefixes

Embedding pipelineprivate— chunk, embed, index — a worker pipeline with the same retry and DLQ needs as cs-worker-pipeline

Vector indexprivate— memory-shaped: sized by vectors × dimensions, not by request rate

Agent runtimeprivate

Model gatewayprivate— embedding calls are metered too, and there are far more of them than completions

Document storage→Embedding pipeline· object-created event

Embedding pipeline→Model gateway· embed chunks

Embedding pipeline→Vector index· upsert with tenant id

Agent runtime→Vector index· query, always tenant-filtered

Decision	Reason	Alternative	Trade-off
Tenant isolation is enforced at query time and asserted in tests.	A vector index does not naturally respect tenancy — nearest-neighbour search will happily return another customer's chunk. A missing filter here is a cross-tenant data leak delivered inside a plausible-sounding answer, which is the hardest kind to notice.	An index per tenant, which is unambiguous and expensive, and hits collection limits at scale — the right answer for a small number of large, sensitive tenants.	Shared indexes mean the filter is load-bearing, so it needs tests that assert a query cannot return another tenant's data, plus a review discipline on every query path.
The vector index is a stateful service with a memory-shaped cost, treated like a database.	It needs capacity planning, backups, version upgrades and a rebuild plan. Teams routinely treat it as a library and are surprised when it becomes the second most expensive and least understood component they run.	A vector extension inside the existing PostgreSQL, which removes an entire component and is entirely adequate at moderate corpus sizes — genuinely the first thing to try.	A dedicated index scales further and costs a permanent always-on memory footprint. Rebuilding it after a model change means re-embedding the whole corpus, which is a large one-off bill nobody budgets for.
Re-embedding is a planned operation with its own budget.	Changing the embedding model invalidates every stored vector. If that is not planned, the platform is locked to its first embedding model forever.	Never change the embedding model, which is a real strategy and a slowly accumulating quality debt.	A dual-index period during migration doubles the memory cost, and the re-embedding run is a large, spiky compute and token expense.

Stage 6

Self-hosted small models on GPU capacity

Forced by

The main reasoning model on a hosted API was fine. The *supporting* models were not: embedding and re-ranking were called two hundred times per run, and at hosted per-call pricing a single run cost more than the customer's monthly subscription. The volume was high, the models were small, and the workload was steady — the three conditions that make self-hosting arithmetic work.

A GPU tier for high-volume small models only, with the reasoning model still hosted.PROVIDER-NEUTRAL

Model gatewayprivate— routes by model: small models internal, reasoning model to the provider

GPU inference poolprivate— batches requests to keep the accelerator busy; model weights baked into the image or pulled from storage at boot

Model weightsprivate— tens of gigabytes — pull time dominates instance startup, so scale-out is minutes

Hosted reasoning modelpublic— stays hosted: low volume per run, and self-hosting a frontier model is a different business

Utilization metricsprivate— GPU utilization and batch size — the only numbers that decide whether this stage was a good idea

Model gateway→GPU inference pool· embedding and re-ranking

Model gateway→Hosted reasoning model· reasoningcrosses boundary

Model weights→GPU inference pool· load at boot

GPU inference pool→Utilization metrics

Decision	Reason	Alternative	Trade-off
Self-host only the small, high-volume models.	A GPU instance is expensive per hour and cheap per request if it stays busy. Two hundred embedding calls per run keep it busy; a handful of reasoning calls per run do not.	Keep everything hosted, which has zero operational burden and pays per call forever — the right answer at low volume and the reason to check the arithmetic rather than the fashion.	You take on GPU capacity management, driver and runtime versions, model version rollout, and a scarce instance type that may be unavailable in your region when you need to scale. Utilization below roughly half turns the whole saving into a loss.
Batch requests at the inference server.	Accelerators are throughput devices: a batch of thirty-two short requests costs little more than one. Without batching you are renting a GPU to do one thing at a time, which is the most expensive possible way to use it.	One request at a time for the lowest per-request latency, which is correct for interactive single-user inference and wasteful for a pipeline.	Batching adds queueing latency per request, and the batch window is a direct latency-versus-cost dial that must be set with the product's expectations in hand.
Treat model weights as a deployment artifact with a slow scale-out path.	Pulling tens of gigabytes of weights dominates instance startup, so GPU capacity cannot be autoscaled reactively the way a stateless API can. Scale-out is minutes, and planning must account for it.	Pull weights on demand per request, which is flexible and makes every cold instance uselessly slow for its first several minutes.	A baked image is large and slow to build, and every model update becomes a full image rebuild and rollout — model versioning is now a deployment problem.

Stage 7

Per-run tracing and cost attribution

Forced by

Two failures with the same root cause. A prompt change tripled token usage per run and nobody noticed for nine days, until the invoice arrived. And when a customer reported a wrong answer, there was no way to see which tool call or which retrieved chunk produced it — the run was a black box that had already been discarded.

A run is only debuggable if the platform recorded it while it happened.PROVIDER-NEUTRAL

Agent runtimeprivate— emits a span per step: prompt reference, tool call, retrieved chunk ids, tokens in and out, duration

Model gatewayprivate— the authoritative token and cost meter, tagged with tenant and run id

Trace storeprivate— one trace per run, retained on a schedule; this is the largest data producer on the platform

Cost attribution storeprivate— cost per run, per tenant, per model — the number the business prices against

Alertingprivate— tokens per run against baseline, run failure rate, sandbox denial rate, spend rate

Agent runtime→Trace store· spans

Model gateway→Cost attribution store· metered usage

Cost attribution store→Alerting· unit-cost drift

Trace store→Alerting· failure and denial rates

Decision	Reason	Alternative	Trade-off
Every run produces a trace: steps, tool calls, retrieved context ids, tokens.	Agent behaviour is non-deterministic, so "reproduce it locally" is not a debugging strategy. The recording made during the run is the only evidence that will ever exist.	Log the final output only, which is compact and makes every quality investigation guesswork.	Traces are large — often larger than all other data the platform produces — so retention is a real cost decision, and they contain customer content, which makes them a data-protection surface with access controls and deletion obligations of their own.
Cost is attributed per run and per tenant at the gateway.	On a platform whose marginal cost is tokens, unit economics is an operational metric. Without per-tenant attribution you cannot price the product, cannot detect an unprofitable customer, and cannot tell growth from regression.	Read the provider's monthly invoice, which is accurate, aggregate, and arrives far too late to act on.	Every call must carry tenant and run tags, which is a discipline that decays unless it is enforced at the gateway rather than requested of callers.
Alert on tokens per run against a baseline, and on a per-run budget ceiling that aborts the run.	It catches the nine-day prompt regression on day one, and it stops a single looping agent from spending without limit. A hard ceiling is the only defence against a failure mode whose cost is unbounded.	Trust prompt review and a monthly budget, which is how the nine-day regression happened.	A ceiling occasionally kills a legitimate complex run, so it must be visible to the user and adjustable per tenant — and the baseline must be reset deliberately after every intentional prompt change, or it pages on every release.

What would break this

Every design has a load, a failure or an organization size at which it stops being the right one.

Breaking points

Runs that must last hours rather than minutes. Queue visibility timeouts and heartbeats stop being adequate, and you need a durable workflow engine with explicit resumable state — a different class of infrastructure.
Strict data residency or a customer who forbids their data leaving your infrastructure. That forces self-hosted reasoning models and turns the GPU stage from an optimization into the foundation of the platform.
Tools with irreversible real-world side effects — payments, deletions, external postings. No sandbox fixes that; it requires human approval steps and a two-phase design in the agent loop itself.
A tenant whose corpus dominates the vector index. Shared-index economics stop working and that customer needs dedicated capacity, which is a pricing conversation before it is an infrastructure one.
Sustained GPU demand in a region where the instance type is scarce. Capacity, not cost, becomes the constraint, and reserved capacity or a second region becomes the answer.
Trace volume outgrowing the application it observes. Sampling becomes necessary, and every sampling decision is a decision about which future incident you will be unable to investigate.

Cost shape

Drivers and relative weights. Never a price.

An agent platform's bill: tokens dominate until GPU capacity or trace storage quietly overtakes them. Shapes only — model pricing changes constantly.COST-VARIES

Model API tokens spiky

driven by tokens in + tokens out, per step, per run — multiplied by every retry and every re-plan · The marginal cost of the product. A prompt change alters this line more than any infrastructure decision in the design, which is why per-run token accounting is an operational metric and not a curiosity.

GPU inference capacity · the surprisefixed

driven by GPU instance-hours, largely independent of how busy the accelerator is · Priced by the hour whether or not it is doing anything, so utilization is the entire argument. Below roughly half utilized, the hosted API you replaced was cheaper.

Agent runtime workers usage

driven by instance-hours × run duration — and runs spend most of their time waiting on the model · Agent workers are mostly idle-but-occupied, which makes concurrency per worker, not CPU, the number that matters for this line.

Tool sandboxes usage

driven by sandbox invocations × their duration and reserved resources · The direct price of the security boundary. It is worth stating plainly that this line exists because the alternative was giving generated code your credentials.

Vector index fixed

driven by memory footprint = vectors × dimensions, always on · Grows with the corpus and never shrinks. A re-embedding migration temporarily doubles it.

Traces and logs · the surpriseusage

driven by spans per run × runs × retention · Every step of every run recorded in full is frequently more data than the product itself produces, and it is billed on ingestion and again on retention. The first team to be surprised by this is always the one that instrumented well.

Embedding calls spiky

driven by chunks embedded on ingestion, plus one per query · Individually trivial and enormous in aggregate during a bulk ingestion or a re-embedding run.

Storage, queue, database, egress fixed

driven by the ordinary infrastructure underneath all of it · Almost invisible next to the rest — a useful reminder that on this platform, optimizing the classic infrastructure line items is optimizing the wrong thing.

Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.

Lessons behind this design

Infrastructure for Model and Agent Workloads GPU and Accelerator Infrastructure Hosted APIs, Managed Inference or Your Own Cluster Secrets in Infrastructure Infrastructure Trust Boundaries Least Privilege in Infrastructure Human vs Workload Identity Containers in Operation Infrastructure Observability Cost Engineering Cost per Service and the Attribution Problem Choosing a Compute Model