Connectionsvcpuschedulingsteal timenoisy neighbourcloudoversubscription

What a vCPU Actually Is

A vCPU is not a core. It is a schedulable thread of execution that the hypervisor multiplexes onto physical hardware, sharing that hardware with other guests. This is why cloud instance performance varies, why steal time exists, and why the count in the instance description does not translate into a guaranteed amount of compute.

Follow the mechanism

Software view, hardware view

The gap between what you wrote and what the machine does is where this whole domain lives.

The question
When a cloud instance advertises eight vCPUs, what have I actually been given — and why does identical code sometimes run at different speeds on it?
What you wrote
The instance has eight vCPUs, so eight threads can run simultaneously and each gets a full core's worth of compute.
What the hardware does
Eight schedulable contexts that the hypervisor places onto physical execution resources when it chooses. Those resources may be hardware threads sharing a core, may be shared with other tenants, and may be taken away mid-execution.
Almost every "our benchmark is inconsistent in the cloud" investigation ends here. Understanding that a vCPU is a scheduling entity rather than a hardware guarantee reframes capacity planning, benchmark methodology and instance selection all at once.
SourceCompilerInstructionsFront EndExecutionRegistersCachesMemoryI/OBehavior

A vCPU is a scheduling abstraction

The hypervisor schedules vCPUs onto physical CPUs much as an operating system schedules threads onto cores (Hardware Threads Are Not OS Threads). A vCPU that is not currently running is simply a runnable context waiting for the hypervisor to place it. From inside the guest this is invisible: the guest kernel believes it has a CPU and schedules its own threads onto it, unaware that the CPU underneath periodically stops existing.

Two independent things determine what that vCPU is worth. The first is what it maps to: on many platforms a vCPU corresponds to a hardware thread rather than a full physical core, so two vCPUs may be two SMT siblings sharing one core's execution resources (SMT: Two Contexts, One Core) — which is emphatically not the same as two cores. The second is who else is using it: if the host is oversubscribed, your vCPU competes with other guests' for physical time.

Steal time is the guest-visible symptom of the second effect: time during which the vCPU was runnable but the hypervisor was running something else. It is one of the most useful and least-watched metrics in cloud operations, because it distinguishes "my code is slow" from "my code was not running" — a distinction no amount of application profiling will make for you.

scheduled byontowaits forplacesalso competingGuest application threadsOther guests' vCPUsGuest kernel schedulervCPU — a schedulable contextHypervisor schedulerPhysical core or hardware thread
UserLLMAgentToolDataDecisionHumanGuardrail

Why identical code runs at different speeds

Several independent sources of variance stack up, and separating them is the whole diagnostic skill. Steal time means your vCPU was not running. Cache and bandwidth contention means it was running but sharing L3 and memory bandwidth with neighbours, so the same instructions took longer (What a Second Core Actually Adds, When the Memory Bus Is the Bottleneck). SMT siblings mean another thread was consuming execution resources on the same physical core. Frequency and thermal state mean the core may have been running at a different clock than it was a minute ago (The Clock Is a Variable, The First Ten Seconds Lie).

On top of these sit two more that catch people out. Cloud fleets are heterogeneous: the same instance type may be backed by different CPU generations, so two instances of the same size can genuinely differ in per-core performance. And migration can move a running guest to different physical hardware, resetting cache warmth (Cache Warmth and the Real Cost of Migration) and possibly landing on a different NUMA topology (NUMA: Not All Memory Is Equally Far).

The practical consequence for benchmarking is severe: a single run on a single cloud instance is close to meaningless. Results need repetition across instances and across time, reported with a distribution rather than a number, and paired with steal time so that host contention can be distinguished from genuine code regression. This is the cloud-specific instance of the general warning in Every Way a CPU Microbenchmark Lies.

Sources of performance variance in a virtualised instance
SourceWhat it meansGuest-visible signalWhat actually helps
Steal timeThe vCPU was runnable but not scheduledSteal time counter risesLarger or dedicated instances; less oversubscribed classes
Cache and bandwidth contentionNeighbours consuming shared L3 and memory bandwidthSame instructions, more cycles; higher miss ratesDedicated hosts; larger instances that own more of a socket
SMT sibling activityAnother thread on the same physical coreLower IPC with no change in miss ratesInstance types that map vCPUs to full cores
Frequency and thermal stateThe core is not at the clock you assumedCycles per second varies between runsLonger warm-up; report distributions; avoid short benchmarks
Fleet heterogeneityDifferent CPU generation behind the same instance typeConsistent difference between instancesPin to instance families that specify the processor
Live migrationGuest moved to different physical hardwareSudden change in behaviour, cold cachesExpect it; do not treat one instance as a stable lab

What this is not: containers

It is worth being explicit, because the two are constantly conflated. A container is not hardware virtualization. There is no guest kernel, no vCPU and no hypervisor scheduling layer: container processes are ordinary processes on the host kernel, isolated by namespaces and constrained by resource limits, and they are scheduled by the host kernel's own scheduler.

That has direct performance consequences of a different shape. A CPU limit on a container is enforced by the kernel's scheduler as a quota over a period, so a container that exhausts its quota is throttled until the next period — which produces latency spikes with a very different signature from steal time. There is no nested page-table walk, because there is no second translation layer. And the isolation boundary is the kernel's system-call surface rather than a hardware-enforced one (The Hardware That Makes Virtual Machines Possible).

Which to choose, how the security postures compare, and how the operational models differ are all cloud-domain questions, treated properly in Containers vs Virtual Machines and Choosing a Compute Model. The hardware point to carry across is narrow and worth remembering: when you read "CPU" in an instance or container specification, find out which mechanism is doing the enforcing, because the two produce different failure modes and are diagnosed with different signals.

  • vCPU — a schedulable context multiplexed by a hypervisor; contention shows up as steal time.
  • Container CPU limit — a scheduler quota enforced by the host kernel; contention shows up as throttling.
  • Different diagnostics — steal time and throttling counters are different metrics with different meanings.
  • Different translation cost — VMs pay a nested page walk; containers do not.
  • Different boundary — hardware-enforced versus the kernel system-call surface.

Key points

  • A vCPU is a schedulable execution context, not a dedicated core, and may map to a hardware thread sharing a physical core.
  • Steal time measures the interval during which your vCPU was runnable but the hypervisor ran something else.
  • Performance variance in the cloud stacks several independent causes: steal, shared cache and bandwidth, SMT siblings, frequency, fleet heterogeneity and migration.
  • A single benchmark run on a single instance is close to meaningless; results need repetition and a distribution.
  • Containers use no hardware virtualization: CPU limits are scheduler quotas, and throttling is not steal time.

Follow the mechanism

The path through the machine, hop by hop — and the conclusions it invites that are wrong.

  1. 1
    Guest thread → guest kernel: the guest schedules its threads onto what it believes are CPUs.
  2. 2
    vCPU → hypervisor queue: the vCPU becomes a runnable context competing with other guests' contexts.
  3. 3
    Hypervisor → physical CPU: it is placed on a core or hardware thread for a time slice, then possibly preempted.
  4. 4
    Physical CPU → shared resources: while running, it shares last-level cache, memory bandwidth and possibly a core with neighbours (SMT: Two Contexts, One Core).
  5. 5
    Preemption → steal time: intervals when the vCPU was ready but not running are accounted as steal and are visible in the guest.
What people conclude from this — wrongly
  • "Eight vCPUs means eight cores of compute" — they may be four cores' worth of SMT siblings, shared with other tenants.
  • "CPU utilisation is only 60%, so we have headroom" — utilisation does not account for stolen time or shared-cache contention.
  • "The benchmark regressed" — it may have been a different host, a different CPU generation, or a noisier neighbour.
  • "Container CPU limits work like vCPU allocation" — one is a scheduler quota with throttling, the other is hypervisor scheduling with steal.

Consequences, controls and cost

What it causes
  • • Identical code shows different performance across instances of the same advertised type and across time on one instance.
  • • Application profiling attributes time to code that was not actually executing, because the guest cannot see that it was descheduled.
  • • Capacity planning based on vCPU counts overestimates available compute when instances are oversubscribed or SMT-backed.
  • • Latency-sensitive services see tail latency driven by host contention that no application change will fix.
What you can do
  • • Watch steal time as a first-class metric; it separates "not running" from "running slowly" and nothing in the application can tell you that.
  • • Use dedicated hosts or instance types that map vCPUs to full physical cores where consistency matters more than cost.
  • • Benchmark across multiple instances and multiple times, and report distributions rather than single numbers.
  • • Pin to instance families that specify the processor generation when comparability matters.
  • • For containers, watch throttling counters rather than steal time — different mechanism, different metric.
How to see it
  • • Track steal time continuously; a rise correlating with latency is host contention rather than a code regression.
  • • Record the CPU model reported inside the guest alongside benchmark results, since fleets are heterogeneous.
  • • Compare IPC across runs: unchanged miss rates with falling IPC suggests SMT sibling or frequency effects rather than a memory problem.
  • • For containers, monitor throttled periods and throttled time, which are the equivalent signal for quota enforcement.
What it costs
  • • Dedicated instances and full-core mappings remove variance at significantly higher cost.
  • • Oversubscription is what makes cloud economics work, so demanding consistency means paying for idle capacity.
  • • Robust benchmarking across instances and time costs substantially more machine time than a single run.
  • • Pinning to specific processor generations restricts capacity availability and can raise cost or reduce region choice.

Scope

§224 — what these claims are specific to.

What these claims are specific to
  • PLATFORM-SPECIFICWhether a vCPU maps to a hardware thread or a full core, how aggressively hosts are oversubscribed, and which metrics are exposed differ by cloud provider, instance family and hypervisor.
  • GENERALThe structural point — a vCPU is a scheduling entity multiplexed onto shared hardware — holds across hypervisors and providers.
  • SIMPLIFIEDThe scheduling diagram omits hypervisor scheduling policy, NUMA-aware placement and CPU pinning, all of which change placement behaviour in ways specific to each platform.

Misconceptions

Claim
“A vCPU is a core, so eight vCPUs is eight cores of compute.”
Reality
A vCPU is a schedulable context. On many instance types two vCPUs are SMT siblings on one physical core, sharing its execution resources — which does not deliver two cores of throughput. On oversubscribed hosts they additionally compete with other tenants for physical time.
Claim
“If CPU utilisation is low, the instance has spare capacity.”
Reality
Utilisation measures time your vCPU spent executing, not time it was available. Stolen time does not appear as utilisation, and contention for shared last-level cache and memory bandwidth makes the same instructions take longer without changing the utilisation figure at all.
Claim
“Container CPU limits and VM vCPU allocation are the same idea.”
Reality
They are different mechanisms with different failure modes. A container limit is a kernel scheduler quota over a period, and exceeding it causes throttling with a characteristic sawtooth latency profile. A vCPU is scheduled by a hypervisor, and contention appears as steal time. They are diagnosed with different counters.

Apply it