Cron Jobs in Production
Six failure modes that scheduled work has and request handling does not: duplicate execution, missed execution, overlap, timezones, long-running jobs, and no observability at all.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
A job runs on a schedule. What are all the ways that goes wrong, given that nobody is watching it?
Scheduled work is written like a script and deployed like an afterthought, but it mutates production data on its own initiative, with nobody present, and its success is indistinguishable from its absence.
Add a cron entry on the server. It runs the job every night; if something goes wrong the error will be in the log.
Nobody reads that log. A job that has not run for three weeks produces exactly the same amount of output as one that succeeded: none that anyone sees (An Alert Should Demand Action).
- Nobody reads that log. A job that has not run for three weeks produces exactly the same amount of output as one that succeeded: none that anyone sees (An Alert Should Demand Action).
- The host is not eternal. It is replaced, drained or scaled down, and the schedule leaves with it — usually without anyone noticing, because the symptom is an absence.
- Run the same entry on two hosts for redundancy and the job now runs twice, concurrently, on the same data (Job Idempotency).
- Last night's run has not finished when tonight's starts, so two copies compete over the same rows. Data volume grows, so this arrives eventually on every job that touches a growing table.
- The host clock is in local time. Twice a year an hour is repeated or skipped, and the job either runs twice or not at all (Timezone and DST Failures).
- The job holds a database connection for forty minutes and blocks something it does not know about, at a time nobody is awake to correlate it (The Connection Budget).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Scheduled work inverts the properties that make request handling easy to operate. A request has a caller who notices failure, a natural timeout, a trace, and a retry policy owned by someone. A scheduled job has none of those by default.
- Its two most important signals are negative. "Did not run" and "ran but did nothing" both look exactly like a quiet, healthy system, which is why absence must be alerted on explicitly.
- Every scheduler makes a delivery guarantee, and most give at-least-once in practice regardless of what the interface implies — a trigger that fails to be acknowledged is retried, a leader that pauses is replaced, and both produce a second execution (Job Scheduler Reliability).
- Overlap is a function of runtime versus interval, and runtime usually grows with data while the interval stays where someone set it. Every hourly job over a growing table is on a path to overlapping itself.
- Production time is UTC. A schedule expressed in a local timezone has two ambiguous hours a year, and a job triggered by local wall-clock time is a job with a built-in annual incident (Production Time Is UTC).
The six failure modes
Read the symptom column carefully. Three of these six produce no error output at all, and one of them produces output that looks like success — which is why "check the logs" is not an operational strategy for scheduled work.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| The host is replaced or scaled down | Nothing. Downstream data quietly goes stale | Schedule lived on the host, not in a durable scheduler | Schedule in a durable system; alert on absence of a completion signal |
| The same entry runs on two hosts | Double side effects: two emails, two charges, doubled counters | Redundancy of the trigger rather than of the execution | Lease or leader election; make the work idempotent regardless (Job Idempotency) |
| Last run has not finished when the next starts | Lock contention, rising duration, eventually neither run completes | Runtime grew with data while the interval stayed fixed | Lease held for the run's duration; the second run declines and says so |
| Clock change or local-time schedule | A run happens twice, or not at all, on two nights a year | Wall-clock scheduling in a timezone with daylight saving (Timezone and DST Failures) | Schedule in UTC; convert only for display |
| A long job holds resources | Unrelated queries slow down nightly; connection pool exhausted | No maximum runtime, no chunking, no connection budget (The Connection Budget) | Enforce a timeout; process in batches with the resource released between them |
| Upstream change makes the input empty | Job succeeds in two seconds, every night, doing nothing | Success measured as exit code rather than as outcome | Emit items processed; alert when it is zero on a job that should never process zero |
The crontab line and everything it does not say
The naive form is genuinely fine for a developer machine. Every problem with it in production comes from what the line does not express: no lease, no timeout, no record, no signal, and a timezone that belongs to the host rather than to the work.
1# Naive: works, until any of six things is true.20 2 * * * /usr/local/bin/reconcile-orders.sh >> /var/log/reconcile.log 2>&13 4# What is missing, made explicit. The wrapper is the production part.5#6# TZ=UTC schedule in UTC, not host-local7# flock -n /var/lock/reconcile only one run at a time, on this host8# timeout 45m a bounded run; a stuck job releases its lease9# --batch-size 500 bounded blast radius per unit of work10# --since-checkpoint resume rather than restart after a partial run11# heartbeat start / finish so absence is detectable, not just failure12#13TZ=UTC140 2 * * * flock -n /var/lock/reconcile \15 timeout 45m \16 /usr/local/bin/reconcile-orders \17 --batch-size 500 \18 --since-checkpoint \19 ; /usr/local/bin/report-run reconcile-orders "$?"20 21# Even this only coordinates runs on ONE host. Two hosts with this entry22# still run it twice: flock is a local lock. Cross-host exclusion needs a23# shared lease, and the job still needs to be idempotent either way.The last comment is the one that catches people. Host-level locking looks like it solves duplicate execution and solves only the single-host case — which is exactly the case that stops being true the moment someone adds a second host for reliability.
What a production job needs beyond the schedule
A scheduled job is a service with a strange invocation pattern. The list below is what turns a script into something you can operate, and each item maps to one of the failure modes above.
- 1Acquire a lease
A shared, expiring lock that prevents concurrent and overlapping runs across all hosts.
fails by Host-local locking, which does nothing across machines.
evidence Two simultaneous invocations produce one execution and one logged decline.
- 2Record the run
Write a start record with an identifier before doing any work.
fails by No record, so a partial run is invisible afterwards.
evidence Every run appears in a queryable history with its outcome.
- 3Resume from checkpoint
Continue from where the last run stopped rather than from the beginning.
fails by Restarting from scratch, so a job that cannot finish never finishes (Backfills).
evidence Killing it midway and re-running reaches the correct end state.
- 4Work in bounded batches
Process a fixed number of items, commit, release, repeat.
fails by One long transaction holding locks for the entire run.
evidence Other workloads show no nightly latency increase.
- 5Enforce a maximum runtime
Abort and release the lease at a hard limit.
fails by A stuck run holding the lease forever, so every subsequent run declines and nothing happens.
evidence A deliberately hung run is terminated and the next scheduled run proceeds.
- 6Emit outcome signals
Duration, items processed, items failed, and a completion heartbeat.
fails by Exit code only, which cannot distinguish success from doing nothing.
evidence A dashboard shows the last run's counts, and zero is visibly abnormal.
- 7Alert on absence
Page when no completion arrives within the expected window.
fails by Alerting only on failure, which never fires when the job does not run at all (Alert on Symptoms, Not on Causes).
evidence Disabling the schedule produces a page.
- 8Be pausable without a deploy
Disable the schedule from a control surface during an incident.
fails by Requiring a pipeline run to stop a job that is actively making things worse.
evidence The pause path has been used in a drill.
How to do it properly
Most important first.
- Make every job idempotent so a duplicate execution is harmless, and design for that rather than trying to guarantee exactly-once (Job Idempotency).
- Take a lease before doing work and hold it for the duration, so a concurrent or overlapping run declines rather than competes.
- Give every job a maximum runtime and enforce it. A job with no timeout is a job that can hold a lease or a connection indefinitely.
- Alert on absence: expect a completion signal within a window and page when it does not arrive. This is the single highest-value change for most scheduled work.
- Schedule in UTC, always, and convert for display only.
- Emit the same signals as a service: start, end, duration, items processed, items failed — and make "processed zero items" distinguishable from "did not run" (Instrumentation: From Code to Signal).
- Chunk long jobs so they can be interrupted and resumed rather than restarted from the beginning (Backfills).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A job that mutates shared data at full speed with nobody watching is contained by nothing on its own. Containment is a lease, a batch size, a maximum runtime, and a run record that makes a bad run reversible as a set.
What can go wrong
- Silent non-execution: the schedule is gone and the first signal is downstream data being stale.
- Silent partial execution: the job died halfway, having processed some records, and the next run starts from the beginning or from an inconsistent position.
- Duplicate concurrent execution producing double side effects — a payment applied twice, an email sent twice, a counter incremented twice.
- Overlapping runs contending for the same locks, each making the other slower until neither finishes within the interval.
- A job that succeeds while doing nothing, because its input query silently returned no rows after an upstream change.
- Retries with no backoff hammering a dependency that is already struggling (Retry Storms: The Load You Generated Yourself).
- A batch job scheduled at midnight local time, in a system where midnight local happens twice in October.
- "Cron is simple." Cron is a simple trigger attached to work with the hardest operational properties in the system: unattended, mutating, and silent on success.
- "It has run every night for two years, so it works." It has run every night for two years on data that was smaller and a schema that was different. Both change without telling the job.
- "If it fails, it will retry tomorrow." Only if the job is idempotent and the missed window does not matter. For anything that processes a time window, a missed run is a permanent gap unless the next run knows to cover it.
- "Two hosts running the same cron entry is redundancy." It is duplicate execution. Redundancy for scheduled work requires a lease or a leader, not a second copy of the trigger (Job Scheduler Reliability).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Each run emits a completion record with duration and counts, and those records are on a dashboard.
- Deliberately preventing the job from running produces a page within its expected window.
- Running two copies simultaneously in a test produces one execution and one clean decline.
- Killing the job halfway and re-running it produces the correct end state.
- A scheduled job mutating data has no rollback unless one was designed: mark what a run changed, so a bad run can be identified and reversed as a set.
- Batch with checkpoints so a bad run can be stopped partway and only the completed batches need reversing.
- Pausing the schedule must be possible without a deploy, and it is the first action when a job is implicated in an incident (Stop the Harm Before You Understand It).
- This lesson is about automation that already exists — the work is making it observable, bounded and safe to run twice.
- Automate the absence check: a dead-man signal that pages when an expected completion does not arrive.
- Do not automate unbounded remediation from inside a job. A job that notices a problem and fixes it at scale, unattended and at three in the morning, is the automation trap with a schedule attached (The Automation Trap).
- Idempotency and leasing require state — a run record, a lock table, a marker per processed item — which is real design work on something that started as a script.
- Chunking makes a job more complex and slower overall, in exchange for being interruptible and resumable.
- Alerting on absence adds an alert that can be noisy if the window is set too tight, which is a real cost against a real class of silent failure.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALAll six failure modes exist for any scheduled work on any platform. What changes is which ones the platform handles for you — a managed scheduler may guarantee a single active run, and none of them make your job idempotent, bound its runtime or tell you it did nothing.
- PLATFORM-SPECIFICHost crontab dies with the host and is invisible to the orchestrator. A cluster-scheduled job survives node loss but can start a second run when a node partitions. A managed cloud scheduler removes the host problem and typically gives at-least-once delivery, which moves the entire burden onto the job being idempotent. Know which of the three you have before assuming a guarantee.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.