Backup Failed Silently: Detect Missing Backups Before You Need Them

Quick answer: A backup “succeeds” in cron terms if the process exits 0 — even when the dump is empty, the upload never happened, or the crontab never fired. Ping Seiri only after a verified artifact exists (and preferably after upload). If the check-in stops, you are alerted before restore day.

The worst time to discover a backup failed silently is during recovery. This is one of the clearest fits for a dead man’s switch: absence of a success signal is the incident.

Seiri is heartbeat / push monitoring — Did the backup job check in? — not disk uptime and not APM.

How backups fail without alarms

Walk this list against your last incident (or your current script):

1. Upload failed, exit 0 — aws s3 cp failed but the shell continued without set -e.
2. Credentials expired — dump skipped or wrote zero bytes; cron still “ran.”
3. Disk full mid-dump — partial .sql.gz left on disk; “file exists” checks pass.
4. Crontab removed after host migration — nothing runs; cron job not running.
5. Kubernetes CronJob suspend: true left on after a maintenance window.
6. Mutex / withoutOverlapping / flock stuck — subsequent runs skip quietly.
7. Wrong database / wrong cluster — script backs up empty staging credentials in prod env mixups.
8. Encryption or checksum step skipped after a refactor — artifact unusable, job green.

Presence of a file on disk is a weak signal. An explicit success check-in after verification is stronger. See also cron job failed silently.

Diagnosis checklist

1. Did the scheduler fire?

crontab -l
grep CRON /var/log/syslog | grep -i backup | tail -20
# K8s:
kubectl get cronjobs,jobs -A | grep -i backup

No schedule fire → fix cron/CronJob first; monitoring below still required afterward.

2. Is there an artifact, and is it non-empty?

ls -lh /var/backups/
find /var/backups -type f -mtime -1 -ls
# empty file check
[[ -s /var/backups/db-latest.sql.gz ]] && echo ok || echo EMPTY

3. Did the remote copy land?

aws s3 ls s3://my-backups/ --recursive | tail
# or rclone, gsutil, az storage — confirm size and timestamp

Compare local size to remote size. Mismatch → silent partial failure.

4. Can you restore a recent artifact to a scratch instance?

A green job that produces unrestorable blobs is still a silent failure. Schedule restore drills; monitor those jobs too (scheduled job monitoring).

5. Exit code and shell options

bash -x /usr/local/bin/backup-database.sh; echo exit:$?

If critical steps lack set -euo pipefail, fix that before anything else.

Monitoring fix: heartbeat the verified success path

Hardened backup script

#!/usr/bin/env bash
set -euo pipefail

dump="/var/backups/db-$(date +%F).sql.gz"
umask 077

pg_dump "$DATABASE_URL" | gzip > "$dump"
[[ -s "$dump" ]] || { echo "empty dump" >&2; exit 1; }

# optional: minimum size guard (tune to your DB)
min_bytes=1048576
size=$(wc -c < "$dump")
(( size >= min_bytes )) || { echo "dump too small: $size" >&2; exit 1; }

aws s3 cp "$dump" "s3://my-backups/$(basename "$dump")"
# optional: head object and compare content-length

# caller pings Seiri only if we reach here

Crontab with Seiri

0 2 * * * /usr/local/bin/backup-database.sh && \
  curl -fsS -m 10 --retry 5 \
  https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success \
  || curl -fsS -m 10 \
  https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/fail

Or success-only (missed run = silence → alert):

0 2 * * * /usr/local/bin/backup-database.sh && \
  curl -fsS -m 10 --retry 5 \
  https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF

Create the monitor in cloud.seiri.app (free forever). Schedule 0 2 * * * with grace covering dump + upload + variance (often 30–90 minutes depending on DB size).

No HTTPS egress

0 2 * * * /usr/local/bin/backup-database.sh && \
  echo "backup ok $(date -u +%FT%TZ)" | \
  mail -s "db-backup" YOUR_REF@ping.seiri.app

See email heartbeat monitoring.

Application schedulers

Same placement rule — ping after verify:

• Laravel: Http::get('https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF') at end of app:backup-database, or pingOnSuccess(...).
• Celery / Sidekiq: ping after upload confirms.
• Kubernetes CronJob: curl in the Job container after /backup.sh, or seiri-kube-agent (kubernetes-agent).

Verify restores on a schedule

Heartbeat proves “job reported success.” It does not prove “restore works.” Add a periodic restore-to-scratch job with its own Seiri monitor. If the drill is skipped, you want that alert as much as a missed nightly dump.

Practices that prevent silent backup failure

1. set -euo pipefail (or fail-hard equivalents).
2. Reject empty / undersized dumps before upload.
3. Confirm remote object size when the API allows.
4. timeout + flock so hung dumps cannot block forever without detection (heartbeat grace still catches silence).
5. One Seiri monitor per backup lane (primary DB, object storage, configs) — do not share one ping across unrelated jobs.
6. Config-manage the crontab / CronJob; treat missing schedules as deploy bugs.
7. Alert to PagerDuty/Slack — not only email to a shared inbox humans ignore.

Related

• Cron job monitoring · complete guide
• Heartbeat monitoring explained
• Kubernetes CronJob monitoring