Scheduled Jobs
Cron in a single process is a timer. Cron on three instances is three timers, and the job runs three times.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
How does recurring work run exactly once when several identical instances are all convinced it is their turn?
Every night at 02:00, generate invoices for accounts whose billing period closed. Once. Not once per instance.
Register an in-process scheduler at startup — a cron library, a timer, a framework annotation — and have it call the job function. It works perfectly in development.
It works perfectly because development runs one instance. Deploy three and you have three schedulers, three timers and three invocations of the same job at 02:00 — three invoice runs, and customers billed three times.
- It works perfectly because development runs one instance. Deploy three and you have three schedulers, three timers and three invocations of the same job at 02:00 — three invoice runs, and customers billed three times.
- Rolling deploys make it worse in a way that is hard to see: for a window during every deploy, old and new instances are both running, so the effective instance count is higher than your configured replica count (Rolling Deployments).
- If instead you avoid duplication by running exactly one instance, you have a single point of failure with no redundancy — and the job silently does not run at all on the night that instance is being restarted.
- The schedule lives in application memory, so nothing anywhere records that a run was due. A job that never fired produces no evidence of not firing.
- Timezones and daylight saving turn "02:00 daily" into a run that happens twice on one night a year and not at all on another (Configuration: Separating Code From Environment).
What is actually happening
- A scheduler answers two questions: when is this due and who runs it. An in-process cron answers the first and assumes the second, which is exactly the assumption that fails once there is more than one process.
- There are three shapes for the second question. Leader election: instances coordinate and one wins the right to fire, usually via a lease in a shared store. External trigger: something outside the fleet fires — a platform scheduler, a Kubernetes CronJob, a managed cron service — and the fleet just receives a call. Claim on execution: every instance wakes up, and all of them try to atomically claim a row representing this specific occurrence; exactly one succeeds.
- Claim-on-execution is the most robust of the three because it needs no coordination protocol and no external dependency: an occurrence is identified by (job name, scheduled time), a unique constraint makes claiming it atomic, and the losers do nothing. It is the same construction as Job Idempotency, applied to a schedule.
- An external trigger is not automatically safe. A platform scheduler that retries a failed invocation, or that guarantees at-least-once firing, will invoke your endpoint twice — so the endpoint still needs to claim the occurrence.
- Whatever fires the job, the right shape is trigger enqueues, worker executes. The scheduler should place one message on a queue and return; then the work inherits retries, visibility, dead-lettering and observability from the queue instead of running inside a timer callback with none of them (Job Queues).
- A missed run is a distinct failure from a duplicated run, and it needs its own detection. Nothing about a timer that did not fire generates a signal — the absence has to be checked for deliberately.
Three instances, one occurrence
The picture is the lesson. Each instance has an identical, correct scheduler. Each one wakes at 02:00. Each one is right about the time. The only question the schedule cannot answer is which of them should act, and the answer has to come from somewhere they all share.
Making the occurrence an addressable thing — a name plus a scheduled timestamp — turns "who runs it" into a database question with an atomic answer. The losers are not blocked or coordinated; they simply find the row already taken and exit.
Who gets to run it?
All four options below produce correct behaviour; they differ in what they depend on and in how they fail. The question to ask is which failure you would rather debug at 02:00: a stale lease, a platform retry, or a claim row that was inserted by a process that then died.
The last option is worth taking seriously. A job that is genuinely idempotent and cheap does not need coordination at all — letting all three instances run it is a legitimate design when the effect is a no-op after the first.
What decides which process executes this occurrence?
when You already have a transactional database. All instances fire; one atomic insert wins.
cost One row and one unique index per occurrence; the claim needs its own lease so a crash between claim and execution does not suppress the run.
when Many scheduled jobs, and you want one designated instance rather than a claim per job.
cost A lease, renewal, and a failure mode where the leader is alive but wedged — it holds the lease and does nothing.
when The platform already offers one and you want the schedule out of the application entirely.
cost A platform dependency, at-least-once firing on retry, and an endpoint you must authenticate (Core Objects, and Why Each One Exists).
when Simplicity matters more than availability of the schedule itself.
cost A single point of failure: while it is restarting or crashed, nothing fires and nothing says so.
when The job is genuinely idempotent and cheap: a cache refresh, a health sweep, a metrics rollup that recomputes from source.
cost N times the load. Only valid when the effect really is a no-op after the first, which is worth verifying rather than assuming.
How scheduled work fails
Scheduled jobs fail in a specific way: quietly, at night, and in a component nobody has opened in months. The rows below are the recurring shapes, and the response column is the part worth copying into a runbook.
The second row is the one to internalise. A job that stops running produces no error, no log line and no metric — the only signal is the absence of the signal, and detecting absence is something you must build on purpose.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Scaling from one instance to three | Customers billed three times | Three in-process schedulers, no claim | Atomic occurrence claim; make the job idempotent regardless |
| A scheduler thread dies or a config change drops the entry | Nothing — for weeks | Absence generates no signal | Alert when an expected occurrence has no completion record (Alerts Worth Waking Someone For) |
| Data growth | The nightly job stops finishing before the next one starts | Duration has grown past the interval; two runs now overlap | Chunk into per-entity jobs; add a concurrency guard on the occurrence |
| A rolling deploy at the scheduled minute | Duplicate runs only on deploy days | Old and new instances both live during the rollout | The claim covers this too — it is the same mechanism, which is the point |
| A daylight-saving transition | A double run or a missing run, once a year | Local-time schedule over a 23- or 25-hour day | Schedule in UTC; decide explicitly what a repeated local hour means |
| A manual re-run during an incident | Duplicate side effects while already degraded | The job was never idempotent; the schedule was the only thing preventing duplicates | Idempotency on the job itself, not on the trigger (Job Idempotency) |
How to build it
Most important first.
- Never rely on an in-process timer for correctness in a multi-instance deployment. Assume every instance will fire (Making an Existing Service Stateless).
- Identify each occurrence explicitly:
invoice-run:2026-08-25T02:00Z. That identifier is what makes the run claimable, deduplicable and auditable (Duplicate Detection). - Claim it with an atomic insert against a unique constraint. Whoever inserts, runs; everyone else exits quietly. This is the whole of the deduplication story.
- Have the trigger enqueue a job rather than do the work, so the actual execution gets the queue's retries, timeouts and dead-letter path.
- Make the job itself idempotent anyway. Claims can be lost, replayed by an operator, or re-run manually during an incident — and manual re-runs during incidents are exactly when a non-idempotent job does damage.
- Store schedules in UTC and convert for display. Choose explicitly what happens on the days that have 23 or 25 hours, rather than discovering the choice your library made.
- Alert on absence: record every completed run and alert when the expected next occurrence has not been claimed within a tolerance window (Alerts Worth Waking Someone For).
- Handle catch-up deliberately. If the fleet was down at 02:00, decide whether the 02:00 run happens late, is skipped, or is coalesced with the next one — and write the decision down.
What can go wrong
- N instances, N executions — the defining failure, and it scales with your replica count.
- A rolling deploy briefly doubling the instance count, so a job duplicates only on deploy days.
- A missed run with no alert, discovered when a customer asks where their invoice is.
- A long job overlapping its own next occurrence, so two runs execute concurrently against the same rows (Backend Races).
- A leader lease held by an instance that is alive but wedged, so the leader neither runs the job nor releases the lease.
- A leader lease that expires mid-run, producing two leaders and two concurrent executions (A Mutex on Server A Does Nothing About Server B).
- Daylight-saving transitions producing a double run or a skipped run once a year, in a job nobody has looked at since it was written.
- A scheduled job that ran fine at small volume timing out at large volume, half-completing every night (Pagination That Survives a Large Table).
- Three instances firing simultaneously and all attempting the same occurrence — the defining race, resolved by an atomic claim.
- A leader lease expiring mid-execution, so a second leader starts while the first is still running.
- A long run overlapping its own next scheduled occurrence.
- A claim inserted but the run never started because the process died between claim and execution — needs a lease on the claim, not just a row.
- A manual re-run racing the scheduled run during an incident (Duplicate Detection).
- Scheduled jobs typically run with the widest credentials in the system and no user context. Scope them per job rather than giving the scheduler a role that is the union of every job it might run (Least Privilege in Infrastructure).
- An endpoint that exists to be called by a platform scheduler is an unauthenticated remote trigger unless you authenticate it. Require a signed header or a platform identity, not a secret path (API Keys).
- Log every run with the occurrence id, who or what claimed it, and the outcome. A scheduled job that touches money needs an audit trail as much as a user action does (Audit Trails).
- A job that reads a schedule or a target from configuration is executing configuration. Validate it at startup rather than discovering a malformed cron expression at 02:00 (Validate at Startup, Fail Loudly).
- "Cron runs once because cron runs once." The cron *expression* describes a time. What runs at that time is every process that has registered it (Stateless Services).
- "We only have one instance." That is true until the first autoscale event, the first rolling deploy, or the first time someone runs the app locally against production configuration.
- "A Kubernetes CronJob guarantees exactly one execution." It creates a Job at the scheduled time; concurrency policy, missed-deadline handling and pod restarts all affect how many pods actually run the work. Read the concurrency policy rather than assuming it (Core Objects, and Why Each One Exists).
- "If the job did not run, we would notice." Nothing generates a signal for work that did not happen. Absence needs an explicit alert.
- "The job is scheduled, so it does not need idempotency." Manual re-runs during incidents are the most likely duplicate execution of all, and they happen when the system is already unhealthy.
Operating it
- Record start and completion of every occurrence with its identifier. That table is simultaneously your deduplication mechanism, your audit trail and your missed-run detector.
- Alert on absence: no completed run for an occurrence that should have happened. Almost nobody has this alert, and a silently dead scheduler is a common multi-week outage (Alerts Worth Waking Someone For).
- Count claim attempts versus claim wins. Wins should be exactly one per occurrence; attempts tell you how many instances are firing, which is a useful sanity check on your mental model.
- Duration per run, plotted against the interval between runs. A duration approaching the interval is an overlap incident waiting to happen.
- Lag from scheduled time to actual start. Growing lag means the trigger is queueing behind something.
- Emit a deploy marker so a change in run behaviour can be correlated with the release that caused it ("What Changed?" — Deploy Markers and the Invisible Deploys).
- Duplication scales linearly with instance count, so the bug gets worse exactly as you add redundancy. This is why it is usually discovered during a scale-up.
- At 10x data, a nightly job that comfortably finished starts to approach its window. Chunk it — enqueue one message per account rather than one message for all accounts — so it becomes parallelisable and resumable (Worker Scaling).
- At 100x, the schedule fans out: the trigger enqueues thousands of per-entity jobs, and the interesting problem moves to the queue rather than the timer (Backpressure).
- Across regions, "run once" means once globally, not once per region. A per-region scheduler with no global claim runs the job once per region (Multi-Region Deployment).
- Claim-on-execution needs no coordination service and costs one row and one unique index per occurrence.
- Leader election concentrates the decision and adds a lease, a renewal path, and a failure mode where the leader is alive but not working.
- An external platform scheduler removes the problem from your code and adds a platform dependency, a differently-shaped retry behaviour, and an endpoint that must be authenticated.
- Trigger-enqueues-worker-executes gives the job retries and observability, and adds a queue hop between "it is 02:00" and "the work started".
- Chunking a large job makes it resumable and parallel, and turns one failure into many partial ones that need aggregate reporting.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe when/who split, and the need for an atomic claim per occurrence, hold for any language and any deployment.
- CLOUD-SPECIFICPlatform schedulers differ in ways that matter. A Kubernetes CronJob creates a Job object whose concurrency policy (Allow, Forbid or Replace) decides what happens when a run overruns, and a missed schedule beyond the starting-deadline window is skipped. Managed cron services generally invoke a target and retry on failure, which is at-least-once firing. A cloud scheduler firing an HTTP endpoint gives you no execution isolation at all. In all three cases the occurrence claim stays your responsibility.
- FRAMEWORK-SPECIFICFramework schedulers (Spring @Scheduled, Celery beat, node-cron, Rails recurring tasks) are in-process by default and duplicate across instances. Some ship an optional distributed lock — ShedLock, a single beat process, a Redis lease — and whether it is enabled is a configuration detail, not a property of the annotation.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — leader election and lease-based coordination, and why a lease that can expire mid-work never gives you a guarantee of single execution on its own.