Dead Man's Switch Monitoring for Engineers

A dead man’s switch is a fail-safe: if the expected signal stops, the system assumes something is wrong and acts. On a train, releasing the switch stops the locomotive. In operations, if your cron job, worker, or batch pipeline stops checking in, you get an alert — before a customer complaint or a failed restore three weeks later.

The engineering question is always the same: Did it run?

This is the same idea as heartbeat monitoring. The industry also calls it push monitoring, missed-cron detection, or dead man’s switch monitoring. Seiri is a hosted dead man’s switch for scheduled and background work — not website uptime polling, not APM, and not a log management platform. Start free at cloud.seiri.app.

What a dead man’s switch means in software

Physically, a dead man’s switch requires continuous presence. Release it and the machine fails safe.

Digitally, your job must keep “holding the switch” by sending heartbeats on a known schedule. Stop sending them — deploy wiped the crontab, the pod was evicted, the script hung on I/O, someone set suspend: true — and the monitor trips.

The important inversion:

• Traditional alerts fire when something bad appears (error log, 5xx spike, OOM kill)
• A dead man’s switch fires when something good disappears (the check-in)

Silence is the incident. That is why this pattern owns cron job monitoring and scheduled job monitoring use cases that uptime tools miss. Product hub: dead man’s switch.

Why silent failures need this pattern

Schedulers are built to be quiet. Cron does not open a ticket when a line is missing. Kubernetes does not page you because last night’s CronJob object was never created. Queue frameworks mark a recurring job “ok” if the last run exited 0 — even if that was days ago and the schedule definition changed underneath.

Failure mode: never started

Expected: process-payments every 15 minutes
Actual:   crontab entry removed in a config management refactor
Visible:  host healthy, other crons still firing
Invisible: payment batches stop; support hears about it from merchants

Error trackers see nothing — no process, no exception. Uptime checks against the API still pass. Only “expected ping missing” catches this class of bug.

Failure mode: started, never finished

#!/usr/bin/env bash
set -euo pipefail
lockfile=/var/lock/etl.lock
exec 9>"$lockfile"
flock -n 9 || exit 0   # previous run still holds lock — quiet no-op
run_etl
curl -fsS https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success

If run_etl hangs while holding the lock, every subsequent invocation exits 0 immediately without pinging — or never reaches the ping. Depending on where you place the check-in, you either see a miss (good) or a false sense of progress (bad). Place the ping after real work completes, and treat “skipped because locked” as a condition you either alert on or monitor with a separate stale-lock check.

Failure mode: finished with a lie

def renew_certs():
    try:
        return perform_renewal()
    except Exception as exc:
        log.warning("renewal failed: %s", exc)
        return True  # caller treats truthy as success

The job “ran.” Certificates still expire. A dead man’s switch does not replace correctness checks, but pairing /success with post-conditions (expiry date moved forward, object exists, row count > 0) makes the switch mean something.

Scenario: certificate renewal that almost looked fine

A weekly Certbot wrapper runs via cron. After a base-image change, the container lacks network permission to the ACME endpoint. The wrapper catches the error, prints a warning to stdout that nobody reads, and exits 0. TLS expiry is 40 days out. On day 41, browsers scream.

With a dead man’s switch that only pings after a successful renewal and a verified new expiry, the miss alerts within one grace window — not after the outage.

Scenario: “successful” backup with empty dumps

Disk quotas change. pg_dump writes a near-empty file, compression succeeds, upload succeeds. Exit 0. Object storage has a fresh timestamp and a useless blob. Heartbeat after upload alone is insufficient; heartbeat after a size or restore-smoke check is better. See also backup failed silently.

Dead man’s switch vs error tracking vs uptime vs logs

Error tracking is necessary for application defects. It is not a schedule guarantee.

Uptime monitoring asks whether an endpoint answers. Your ETL job is not an endpoint.

Logs answer “what happened during a run?” after you already suspect a window. They are a poor primary pager for “did Tuesday’s job exist?”

Dead man’s switch expects a ping. If neither /success nor /fail arrives in time, you page. That is the definition of missed cron detection.

Seiri intentionally does not try to be all four. It is the switch.

Common approaches teams try first

1. Grep cron logs on a schedule

# Fragile: assumes syslog format, host affinity, and that CMD line == success
if ! grep -q "backup.sh" /var/log/syslog; then
  mail -s "backup missing?" oncall@example.com
fi

Problems: multi-host fleets, containerized jobs with ephemeral logs, timezone mismatches, and the fact that CMD appearance ≠ completion.

2. Prometheus time() - job_last_success_timestamp

This is a dead man’s switch implemented in PromQL. It works well when you already run Prometheus and instrument every job. It fails organizationally when:

• Half the jobs are on VMs without the exporter
• Nobody owns the alert rule after the author leaves
• Kubernetes Jobs need different labels than systemd timers

A dedicated monitor per job is often clearer than one giant rules file.

3. “Email yourself from cron”

0 2 * * * /usr/local/bin/backup.sh && echo OK | mail -s backup-ok you@company.com

That is a human-parsed dead man’s switch. It does not scale, has no grace period logic, and trains people to ignore mail. Better: send machine-checkable pings to a system that pages on silence — including email heartbeats addressed to YOUR_REF@ping.seiri.app when HTTP is unavailable.

4. Proper push check-ins

Define schedule + grace, ping on completion, alert on miss. That is the rest of this article.

How to wire a dead man’s switch with Seiri

1. Create a monitor with an expected interval or cron expression
2. Set a grace period for normal runtime variance
3. Ping on success from the job; optionally ping /fail on known errors
4. Route alerts to Slack, email, webhook, or your on-call path

Canonical URL shapes:

https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF
https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success
https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/fail

Email: YOUR_REF@ping.seiri.app

Linux cron

*/15 * * * * /app/bin/process-payments && \
  curl -fsS -m 10 --retry 3 \
  https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF

If process-payments hangs or the crontab disappears, Seiri never gets the ping and alerts after the grace period.

Success/fail variant:

*/15 * * * * \
  /app/bin/process-payments \
  && curl -fsS -m 10 --retry 3 \
       https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success \
  || curl -fsS -m 10 --retry 3 \
       https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/fail

Python worker heartbeat (alive loop)

For long-lived workers (not only cron), ping on an interval from the main loop so a dead process stops checking in:

import threading
import time
import requests

PING = "https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF"
INTERVAL = 60  # seconds; align monitor expectation + grace


def heartbeat_loop(stop: threading.Event) -> None:
    while not stop.wait(interval):
        try:
            requests.get(PING, timeout=10).raise_for_status()
        except Exception:
            # Local log; Seiri will also notice if pings stop entirely
            pass


def main() -> None:
    stop = threading.Event()
    t = threading.Thread(target=heartbeat_loop, args=(stop,), daemon=True)
    t.start()
    try:
        run_worker_forever()
    finally:
        stop.set()

Use a short interval for “process must stay up.” Use schedule-aligned pings for “batch must finish nightly.” Do not confuse the two monitors.

Shell with explicit verification

#!/usr/bin/env bash
set -euo pipefail

PING="https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF"
OUT="/var/backups/db-$(date -u +%F).sql.gz"

pg_dump "$DATABASE_URL" | gzip > "$OUT"
BYTES=$(wc -c < "$OUT")
# Refuse to claim success on empty/tiny dumps
if [ "$BYTES" -lt 1024 ]; then
  curl -fsS -m 10 --retry 3 "${PING}/fail" || true
  exit 1
fi

aws s3 cp "$OUT" "s3://backups/$(basename "$OUT")"
curl -fsS -m 10 --retry 3 "${PING}/success"

The switch is only as trustworthy as the conditions you put before /success.

Kubernetes CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-etl
spec:
  schedule: "0 */6 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      activeDeadlineSeconds: 7200
      backoffLimit: 1
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: etl
              image: etl:stable
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  /app/run-etl.sh
                  curl -fsS -m 10 --retry 3 \
                    https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success                  

For operator-style checks without embedding curl in every image, use seiri-kube-agent (Helm HealthCheck operator): chart docs, product guide /kubernetes-agent, and the deep dive Kubernetes CronJob monitoring.

Grace periods: when the switch should wait

A dead man’s switch that is too eager becomes noise. One that is too patient becomes theater.

Grace period = time after the expected check-in moment before alerting.

Guidelines:

• Base grace on observed duration distribution, not the happy-path demo run alone
• For high-frequency jobs (every 5–15 minutes), grace of one extra interval is often enough
• For long ETL, grace may be tens of minutes to a few hours — still shorter than the business pain window
• Explicit /fail can alert immediately; silence waits for grace
• Revisit grace after you change data volume, instance size, or network path

Example timeline:

Schedule:        02:00 daily
Typical finish:  02:08–02:18
Monitor expect:  check-in by ~02:20
Grace:           30 minutes → alert ~02:50 if silent
Business need:   backup must exist before 06:00 trading prep

You still have room to fix before the business deadline. A 12-hour grace would not.

Schedule drift and the dead man’s switch

Clocks lie in distributed systems. DST skips an hour. A node’s CMOS battery dies. A CronJob gains timeZone: America/New_York while the monitor still assumes UTC. Someone edits the schedule from 0 2 * * * to 0 3 * * * and forgets the monitor.

Effects:

• False alerts (job is fine, expectation wrong)
• Missed alerts (expectation too loose relative to new schedule)
• Confusing postmortems (“it ran at 3 but we paged at 2:45”)

Mitigations:

1. Document the schedule in one place; change monitor and scheduler together
2. Prefer UTC in both systems when business rules allow
3. Use interval-based monitors for “every N minutes” workers so absolute clock faces matter less
4. Keep grace wide enough to survive DST edges on daily jobs without hiding multi-hour outages
5. After migrations, require evidence of a fresh ping before calling cutover done

Drift is also why wall-clock-only scripts (“if hour is 2, check process list”) rot. The switch cares about check-ins, not about agreeing with every clock in the fleet forever.

When to use a dead man’s switch

Use it when:

Prefer other tools when:

• You need to know if https://app.example.com is up from multiple networks → uptime product
• You need span-level latency across microservices → APM
• You need full-text search over application logs → log platform

Those tools answer different questions. The dead man’s switch answers Did it run?

Operational practices that keep the switch honest

1. One monitor per critical job — shared monitors blur ownership
2. Ping after verification — size checks, row counts, object head requests
3. Include /fail on known error paths — faster than waiting for grace when the job can speak
4. Alert where humans are — on-call and chat beat a shared inbox nobody reads
5. Treat monitor updates as part of deploys — crontab/Helm/CronJob changes ship with ping tests
6. Link runbooks — the alert should say what the job is and who owns it
7. Test the negative path — temporarily pause the job in staging and confirm the page fires

Related reading for operators chasing symptoms: cron job not running, cron job failed silently, scheduled job reliability.

Building the switch into existing systems

You rarely greenfield every scheduler. Dead man’s switches land best as a thin layer on top of what already runs.

Crontab estates

Inventory with crontab -l per user and /etc/cron.d. For each line that moves money, data, or certificates, add a monitor and a ping. Leave low-risk noise jobs alone until you have capacity — coverage of the top ten critical jobs beats superficial coverage of fifty.

A shared wrapper keeps diffs small:

# /usr/local/bin/guarded-job
set -euo pipefail
REF_URL="$1"; shift
if "$@"; then
  curl -fsS -m 10 --retry 3 "${REF_URL}/success"
else
  rc=$?
  curl -fsS -m 10 --retry 3 "${REF_URL}/fail" || true
  exit "$rc"
fi

Application schedulers

In Sidekiq, Celery Beat, Laravel, Hangfire, and similar systems, the recurring definition lives in code or Redis — not in crontab. The switch still belongs at the end of the job class/method after domain verification. If the recurring entry is deleted but workers stay up, only the missing ping reveals the gap. Framework-specific wiring: Sidekiq, Celery, Laravel.

CI-triggered “schedules”

GitHub Actions on.schedule and GitLab pipeline schedules fail like cron: the workflow file changes, the cron expression breaks, or the runner pool is empty. Add a final step that pings Seiri on success. Treat Actions as another scheduler, not as “monitoring already handled because GitHub emails you” — those emails are easy to miss and often disabled.

# GitHub Actions sketch
- name: Heartbeat
  if: success()
  run: |
    curl -fsS -m 10 --retry 3 \
      https://ping.seiri.app/webhook/${{ secrets.SEIRI_ORG }}:${{ secrets.SEIRI_REF }}/success    
- name: Heartbeat fail
  if: failure()
  run: |
    curl -fsS -m 10 --retry 3 \
      https://ping.seiri.app/webhook/${{ secrets.SEIRI_ORG }}:${{ secrets.SEIRI_REF }}/fail || true    

Devices and edge scripts

Kiosks, stores, and appliances often cannot open inbound ports and sometimes cannot use arbitrary HTTPS. Email to YOUR_REF@ping.seiri.app keeps the dead man’s switch model when email heartbeats are the only reliable egress.

False positives and false negatives

A switch that cries wolf gets muted. A switch that never cries gets ignored until disaster.

False positives (alert, job was fine):

• Grace too tight for month-end volume
• Monitor cron still on old schedule after a deliberate one-hour shift
• DST edge without enough slack
• Ping succeeded but from a retry of an old run while you expected the new window (label runs clearly in logs)

False negatives (no alert, job was not fine):

• Ping at start instead of after verification
• Unconditional ; curl after a failing script
• Shared monitor across jobs so one healthy job masks another’s silence
• /success after upload without checking object size or checksum
• Alert routed to an abandoned inbox

Review monitors after any incident that “monitoring should have caught.” Update verification, grace, or ownership — do not only write a postmortem paragraph.

Runbooks that belong next to the switch

When the page fires, responders need more than a red badge.

Minimum runbook contents:

1. What the job is supposed to produce (path, bucket, table, email)
2. Which scheduler definition to inspect (crontab path, CronJob name, DAG id)
3. How to run it safely by hand without double-writing
4. How to verify the artifact after a manual run
5. Who owns the data if the job cannot be fixed in-shift
6. Whether a missed window needs backfill or only “catch next”

Attach the runbook URL to the Seiri alert description or your paging tool’s payload. The dead man’s switch buys you time; the runbook spends it well.

Team vocabulary

Agree on words so handoffs do not confuse uptime with schedules:

• Heartbeat / dead man’s switch — expect check-in; silence pages
• Uptime check — probe a URL; failure pages
• Job Success in Kubernetes — container exit 0; not necessarily business success
• Grace — allowed lateness before silence becomes an incident

When someone says “we monitor the backup,” ask: Do we get paged if it never checks in? If the answer is no, you do not have a dead man’s switch yet — you have hope. Product hub: /dead-mans-switch. Deep dive on the synonym: heartbeat monitoring explained.

Worked example: payment batch every fifteen minutes

#!/usr/bin/env bash
# /app/bin/process-payments
set -euo pipefail

PING="https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF"
LOCK=/var/lock/process-payments.lock
exec 9>"$LOCK"
flock -n 9 || {
  # Previous run still active — do not pretend this window succeeded
  curl -fsS -m 10 --retry 3 "${PING}/fail" || true
  exit 1
}

/app/bin/process-payments-inner
# Inner verifies gateway acknowledgements before returning 0
curl -fsS -m 10 --retry 3 "${PING}/success"

*/15 * * * * /app/bin/process-payments >>/var/log/payments.log 2>&1

Monitor settings that match reality:

• Expectation: every 15 minutes
• Grace: enough for a slow gateway burst (often one extra interval, not four)
• Alert: page payments on-call on silence or /fail
• Runbook: how to inspect the lock, how to drain safely, whether to replay a window

Failure modes this catches:

• Crontab removed during an Ansible refactor
• Inner binary hung on DB pool exhaustion (no success ping; grace fires)
• Overlap storms that hit the lock and /fail immediately instead of silent no-ops

Failure modes this does not catch alone:

• Inner returns 0 after “ack” that the gateway later reverses — needs domain reconciliation
• Wrong merchant account configured but still “successful” — needs business checks

The switch narrows the search space. It does not replace payment reconciliation.

Worked example: weekly cert renewal

import subprocess
import requests

PING = "https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF"


def expiry_days(domain: str) -> int:
    # Prefer your real cert inventory API; openssl sketch for illustration
    out = subprocess.check_output(
        ["openssl", "x509", "-enddate", "-noout", "-in", f"/etc/certs/{domain}.pem"],
        text=True,
    )
    # parse notAfter=... → days remaining in your codebase
    return parse_days_remaining(out)


def main() -> None:
    before = expiry_days("api.example.com")
    subprocess.check_call(["certbot", "renew", "--quiet"])
    after = expiry_days("api.example.com")
    if after <= before:
        requests.get(f"{PING}/fail", timeout=10)
        raise SystemExit("renewal did not extend certificate")
    requests.get(f"{PING}/success", timeout=10)


if __name__ == "__main__":
    main()

Weekly cron plus a Seiri monitor with multi-day grace still beats discovering expiry in the browser. Pair with a calendar reminder for human audit if you want defense in depth — the automatic page is for the week the cron disappears.

Choosing interval vs cron expectations

Interval monitors (“must check in at least every N minutes”) fit workers and high-frequency batches. They degrade gracefully when clocks disagree slightly.

Cron expression monitors fit civil-time jobs (“weekdays at 06:30”). They need correct timezone alignment and thoughtful grace around DST.

Do not force every job into one shape. A Sidekiq worker that must stay alive wants interval heartbeats from its main loop. A nightly warehouse load wants cron-shaped expectation plus finish ping. Mixing those without documenting which is which confuses on-call.

See also scheduled job reliability for batch-focused patterns and Kubernetes CronJob monitoring when the scheduler is a cluster controller.

Conclusion

A dead man’s switch for engineers is not a metaphor exercise. It is a concrete contract: the job checks in; silence pages you. That contract catches the failures that error trackers and uptime probes systematically miss — deleted schedules, hung processes, suspended CronJobs, and workers that die between deploys.

Implement it with push URLs (https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF), optional /success and /fail, email when needed (YOUR_REF@ping.seiri.app), and grace periods tuned to real runtimes. Use heartbeat monitoring vocabulary with your team so “we have a switch on backups” means the same thing to everyone.

Seiri hosts that switch for cron, workers, and Kubernetes — free forever at cloud.seiri.app. See also /pricing, the complete cron guide, and heartbeat monitoring explained.