Cron Job Failed Silently: How to Detect Missed Runs

Quick answer: Cron does not page you. A job can skip, hang, or exit 0 after doing nothing — and the crontab still looks correct. Detect silent failure with a heartbeat: ping Seiri only after real success; alert if the check-in never arrives.

Someone searched “cron job failed silently” because the damage already started: missing backups, unsent invoices, stale ETL — and neither cron nor the script complained loudly enough.

What “failed silently” usually means

Pick the matching failure mode before you dig:

1. Never ran — crontab missing, wrong user, daemon down, schedule typo. See cron job not running.
2. Ran and lied — exit 0 with skipped work, ignored upload errors, empty dump.
3. Started and never finished — deadlock, NFS stall, waiting on stdin, no timeout.
4. Failed with a log nobody saw — mail to a dead local mailbox, log rotated, >> /dev/null.

Error trackers help for (2) only when an exception or non-zero exit is actually surfaced. They miss (1) and often (3). That is why silent cron is a dead man’s switch problem, not an APM problem. Seiri is push/heartbeat monitoring — Did it run? — not uptime polling.

Diagnosis checklist

Work top-down. Stop when you find the break.

1. Confirm the expected schedule and user

crontab -l
sudo crontab -l
ls -la /etc/cron.d /etc/cron.daily 2>/dev/null
grep -R "backup\|etl\|invoice" /etc/cron.* 2>/dev/null

Wrong user is classic: you edit deploy’s crontab while production runs as root.

2. Confirm cron executed something

grep CRON /var/log/syslog | tail -80
# Debian/Ubuntu often:
journalctl -u cron -n 80 --no-pager
# RHEL-family:
journalctl -u crond -n 80 --no-pager

Look for CMD (…your command…) near the expected time. No line → scheduling/daemon problem. Line present → script/environment problem.

3. Run the command exactly as cron would

sudo -u deploy -H env -i bash -c '/usr/local/bin/backup.sh'
echo $?

Cron’s environment is minimal (PATH especially). Interactive success proves little.

4. Inspect exit handling inside the script

# bad: upload failure ignored
pg_dump ... > dump.sql
aws s3 cp dump.sql s3://bucket/   # no set -e; failure still exits 0 from shell quirks

Prefer:

#!/usr/bin/env bash
set -euo pipefail

and explicit checks ([[ -s "$dump" ]]) before claiming success.

5. Check for hangs

ps aux | grep -E 'backup|pg_dump'
# if you wrap with timeout:
timeout 2h /usr/local/bin/backup.sh

A hung previous run plus flock can make the next cron “succeed” by skipping.

6. Ask whether any monitor got a ping

If you have no heartbeat today, you are debugging blind for the next incident. Add one before you close the ticket.

Monitoring fix: heartbeat on the success path

Create a monitor in cloud.seiri.app (free forever). Ping only when the job truly succeeded:

0 2 * * * /usr/local/bin/backup.sh && \
  curl -fsS -m 10 --retry 3 \
  https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success \
  || curl -fsS -m 10 \
  https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/fail

Behavior:

• Script fails early → /fail (if the || path runs as intended) or silence if you only use && success pings.
• Script never starts → silence → Seiri missed-check-in alert after grace.
• Script hangs past grace → silence → same alert.

Success-only form (simplest missed-run detector):

0 2 * * * /usr/local/bin/backup.sh && \
  curl -fsS -m 10 --retry 3 \
  https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF

No egress HTTPS? Mail after success to YOUR_REF@ping.seiri.app — email heartbeat monitoring.

Wrapper that fails closed:

#!/usr/bin/env bash
set -euo pipefail
/usr/local/bin/backup.sh
curl -fsS -m 10 --retry 3 \
  "https://ping.seiri.app/webhook/${SEIRI_REF}/success"

Crontab:

0 2 * * * SEIRI_REF=YOUR_ORG:YOUR_REF /usr/local/bin/backup-and-ping.sh

Match Seiri’s schedule to cron; set grace above normal runtime. Route alerts to the same Slack/PagerDuty channel you already trust.

Logging that does not create false comfort

Redirecting cron mail to /dev/null hides failures. Shipping logs to a central system helps after you look — it does not page on absence.

# weak: errors discarded
0 2 * * * /usr/local/bin/backup.sh >/dev/null 2>&1

# better: keep logs AND heartbeat
0 2 * * * /usr/local/bin/backup-and-ping.sh >>/var/log/backup.log 2>&1

Central logging + Seiri is complementary: logs explain why; heartbeats tell you that the window was missed when there is no log line at all.

systemd timers disguised as cron

Some hosts migrated to timers. Operators still search “cron job failed silently.” Check:

systemctl list-timers --all
systemctl status backup.timer backup.service
journalctl -u backup.service -n 50 --no-pager

The monitoring fix is identical: OnSuccess-style ExecStop is fragile; ping Seiri from the service ExecStart script after verified work, or:

# drop-in idea — prefer ping inside the script for correctness
[Service]
ExecStart=/usr/local/bin/backup-and-ping.sh

Prevent the next silent miss

1. Absolute paths in crontab; no reliance on interactive PATH.
2. set -euo pipefail (or language equivalent); check artifact size/checksums.
3. flock + timeout for jobs that must not overlap or hang forever.
4. One Seiri monitor per critical job — not one ping for the whole server.
5. Keep crontab in config management so lines cannot vanish unnoticed.
6. After host migration, verify a ping lands before signing off.
7. Do not >/dev/null 2>&1 without another signal path (heartbeat).
8. Treat “SUCCESS with empty work” as a bug in the script, not a monitoring gap alone.

Framework schedules (Laravel, Celery Beat, Sidekiq-cron) fail the same way when the outer trigger dies — see Laravel, Celery, Sidekiq. Kubernetes: CronJob monitoring.

Related

• Cron job monitoring · complete guide
• Backup failed silently
• Heartbeat monitoring explained