AI & GPU Infrastructure
Long-running jobs that outlive a request, model credentials as first-class secrets, tool sandboxes as isolation boundaries, accelerator scheduling and memory limits, batching and utilization, and the choice between a hosted model API, managed inference and a GPU cluster you own.
An agent run is a long-lived job with external dependencies, privileged credentials and untrusted tool output. The infrastructure that supports it is a queue, a worker pool, a sandbox and a trace — not a bigger web server.
Q · What does a model or agent workload need from infrastructure that an ordinary request/response service never asked for?
An optional advanced module. GPU instances are rented by the hour and bill identically whether they are saturated or idle, so utilization and batching are the entire cost story — and the model fitting in memory is the binding constraint, not raw throughput.
Q · When does a workload need an accelerator, and what changes about scheduling, memory and cost when it does?
Four provider-neutral categories for running a model, compared on cost shape, latency, control, data boundary and scaling — and an honest account of which one a team of five should actually pick.
Q · Should this workload call a hosted model API, deploy to managed inference, self-host on our own compute, or run a dedicated accelerator cluster?