Heartbeat Monitoring Explained: Detecting Silent Failures

Your backup script exits 0. The worker process is still listed in ps. Dashboards are green. And the work never happened.

Heartbeat monitoring answers the question traditional tools skip: Did it run? Your job sends a check-in when it finishes. If that check-in never arrives past a grace period, you get an alert.

This is not website uptime monitoring, APM, or log management. Uptime tools poll a URL to see if a server responds. Heartbeat (push) monitoring works the other way: the job pings Seiri, and silence is the incident. That model is the foundation of cron job monitoring, scheduled job monitoring, and dead man’s switch patterns.

What is heartbeat monitoring?

Heartbeat monitoring expects a periodic signal from a job, worker, device, or script.

1. You define an expected schedule — every 5 minutes, nightly at 02:00, or a cron expression.
2. The workload sends a check-in (HTTP ping or email) when it completes successfully — or reports /fail when it knows it failed.
3. If no check-in arrives before the deadline plus grace period, the monitor alerts.

Also called push monitoring or dead man’s switch monitoring, the idea is simple: absence of a signal is itself a signal.

That catches failure modes that leave almost no error trail:

• The crontab line was deleted or commented out during a deploy
• The process hung before it could log or throw
• The script returned success without doing useful work (empty dump, skipped upload)
• A Kubernetes CronJob stopped creating Jobs after a schedule or suspend change
• A queue worker crashed and nobody restarted it

If you only watch CPU, HTTP status codes, or exception trackers, those cases stay invisible until a human notices missing backups, stale invoices, or empty reports.

Why silent failures happen

Scheduled and background work fails differently from user-facing APIs. Nobody is refreshing a page. There is no 500 in the browser. Cron, systemd timers, Kubernetes CronJobs, and batch schedulers are designed to run quietly — and quiet is exactly what hides problems.

The job never started

# Yesterday's crontab (worked)
0 2 * * * /usr/local/bin/backup.sh

# After a "cleanup" deploy — line gone, cron still healthy
# syslog may show other jobs; this one simply never appears

The cron daemon is fine. Other jobs run. Your monitoring that checks “is cron up?” stays green. Nothing scheduled your backup.

The job started and hung

#!/usr/bin/env bash
set -euo pipefail
pg_dump "$DATABASE_URL" | gzip > /tmp/backup.sql.gz
# Network stall on the next line — process waits forever
aws s3 cp /tmp/backup.sql.gz "s3://backups/$(date +%F).sql.gz"

No exception. No exit code. Logs stop mid-run. A process list still shows the script. Metrics may show a long-lived process and call it “busy.” Without a completion heartbeat, you only learn when restore time comes and the object is missing.

The job “succeeded” without doing the work

def run_etl():
    rows = fetch_source()
    if not rows:
        # Empty is treated as success — but finance expected data
        return 0
    transform_and_load(rows)
    return 0

Exit code 0. Scheduler marks success. Downstream dashboards render yesterday’s numbers until someone compares dates. Heartbeats do not magically validate business logic, but they do force you to decide when “done” means “safe to check in” — and they catch the never-ran and never-finished cases that dominate silent incidents.

Real failure scenarios worth designing for

Nightly backup after credential rotation. The dump still writes locally. The upload fails. Without set -e (or equivalent), the script continues and exits 0. Disk fills with local dumps; object storage stays stale.

Payment batch every 15 minutes. The DB pool is exhausted. The process blocks on acquire(). Cron starts a new instance every interval (concurrencyPolicy equivalent: allow). You get a pile of hung processes and zero completed batches.

Report generator after host migration. The new VM never got the crontab. Ops verifies the app is up with an uptime check. Reports stop for two weeks. Customers do not open a ticket about a PDF they never knew to expect.

Kubernetes CronJob with suspend: true left on. Someone paused a Job for a maintenance window. The cluster is healthy. Prometheus shows the last successful Job from before the pause. No new pods. No new pings.

These are the problems cron job monitoring and scheduled job monitoring exist to catch — and heartbeats are the primitive that makes detection reliable.

Heartbeat monitoring vs uptime, APM, and logs

Uptime probes are excellent for public HTTP services. They are the wrong tool for a nightly pg_dump, an air-gapped appliance, or a Sidekiq recurring job with no inbound port.

APM helps when code runs and emits spans. It does not create a span for a crontab line that was deleted.

Logs help when you know which host and time window to search — and when the job wrote something. Rotated logs and “never started” both leave you empty-handed.

Metrics can approximate heartbeats if you increment job_last_success_timestamp and alert on staleness. That is effectively building a dead man’s switch yourself. Many teams do it for one or two critical jobs and stop. A dedicated heartbeat monitor is the same idea with less custom PromQL and clearer ownership.

For a deeper comparison of the safety pattern name, see What is a dead man’s switch?. For the marketing hub, see heartbeat monitoring.

Common monitoring approaches (and gaps)

1. Rely on logs

grep CRON /var/log/syslog | tail
# Nov 15 02:00:01 box CRON[12345]: (root) CMD (/usr/local/bin/backup.sh)

That line means cron started the command. It does not mean the script finished, uploaded, or produced a usable artifact. Searching logs after an incident is forensics, not monitoring.

2. Alert on error rates

Exception trackers fire when code throws. They stay quiet when:

• The process never starts
• The process hangs without throwing
• Failures are swallowed (|| true, bare except:)

Useful for application bugs. Incomplete for schedule reliability.

3. Alert on infrastructure metrics

CPU, memory, disk, and “process count” answer “is the machine sick?” They do not answer “did invoice-sync finish by 06:30?”

4. Heartbeat / push check-ins

Require the job to report completion. Alert on silence. That is the approach Seiri is built for — not multi-region website probing, not log search, not distributed tracing.

You can combine approaches: keep APM for request latency, keep logs for debugging, and add heartbeats for Did it run?

Implementation: a minimal heartbeat with Seiri

Create a monitor in Seiri with an expected interval or cron schedule. You get a ping URL shaped like:

https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF
https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success
https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/fail

Email check-ins use YOUR_REF@ping.seiri.app — useful when HTTP egress is blocked. See email heartbeat monitoring.

Bash / crontab

Ping only after success:

# Nightly backup — check in only if backup.sh exits 0
0 2 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 \
  https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF

Report success and failure explicitly:

0 2 * * * \
  /usr/local/bin/backup.sh \
  && curl -fsS -m 10 --retry 3 \
       https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success \
  || curl -fsS -m 10 --retry 3 \
       https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/fail

Notes that matter in production:

• -f fails on HTTP errors so a 5xx does not look like a successful ping
• -m 10 caps hang time so a stuck curl does not strand the shell
• --retry 3 absorbs brief network blips without inventing a full client
• Put the ping after the work you care about, not at the start (unless you intentionally monitor “started” separately)

Wrapper script pattern

#!/usr/bin/env bash
set -euo pipefail

PING="https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF"
JOB="/usr/local/bin/backup.sh"

cleanup_fail() {
  curl -fsS -m 10 --retry 3 "${PING}/fail" || true
}
trap cleanup_fail ERR

"$JOB"
curl -fsS -m 10 --retry 3 "${PING}/success"

Wire the wrapper in crontab so every critical job has the same check-in contract.

Python

import sys
import requests

PING = "https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF"
TIMEOUT = 10


def ping(path: str = "") -> None:
    url = f"{PING}{path}"
    requests.get(url, timeout=TIMEOUT).raise_for_status()


def run_backup() -> None:
    # ... dump, compress, upload, verify object exists ...
    pass


def main() -> int:
    try:
        run_backup()
        ping("/success")
        return 0
    except Exception:
        try:
            ping("/fail")
        except Exception:
            pass
        raise


if __name__ == "__main__":
    sys.exit(main())

Ping after verification steps (object exists in S3, row counts non-zero), not merely after “the function returned.”

systemd timer + oneshot service

# /etc/systemd/system/nightly-backup.service
[Unit]
Description=Nightly database backup with Seiri heartbeat

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=/usr/bin/curl -fsS -m 10 --retry 3 https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success

ExecStartPost runs only if ExecStart succeeds. Pair with a .timer unit for the schedule. If the timer is disabled, Seiri still alerts on the missed expected check-in — which is the point.

Kubernetes CronJob (HTTP from the Job)

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      activeDeadlineSeconds: 3600
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: backup:latest
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  /backup.sh
                  curl -fsS -m 10 --retry 3 \
                    https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success                  

If the CronJob never creates a Job, the ping never arrives. For in-cluster HealthCheck CRs and the Helm operator, see Kubernetes CronJob monitoring and the seiri-kube-agent chart, plus the product page at /kubernetes-agent.

Grace periods: absorbing real runtime variance

Jobs do not finish at the same second every night. Backups grow. Network latency varies. Lock waits spike. A monitor that expects a ping at exactly 02:05 and alerts at 02:06 will train your team to ignore pages.

A grace period is extra time after the expected check-in window before alerting.

Practical guidance:

• Measure a few normal runs (or use historical duration if you have it)
• Set grace to cover typical variance plus a buffer — often on the order of “expected duration × 1.5–2” for batch work, not milliseconds of jitter
• Keep grace shorter than the business impact window. A 24-hour grace on a 15-minute payment job is too late
• Prefer separate monitors for jobs with different SLAs rather than one mega-grace for everything

Example: a job usually finishes in 8–12 minutes after a 02:00 start. Expect the check-in by ~02:15 with a 20–30 minute grace. If nothing arrives by then, page. If the same job sometimes runs 90 minutes during month-end, either raise grace for that monitor or use a different schedule/monitor for month-end runs.

Grace periods interact with /fail: an explicit fail can alert immediately, while silence waits for the deadline. That split lets you distinguish “we know it broke” from “we never heard anything.”

Schedule drift: clocks, timezones, and DST

Heartbeat monitoring cares that the job checked in — not that your wall clock and the scheduler’s wall clock agree forever.

Sources of drift and confusion:

• Timezone changes on the host (CRON_TZ, container TZ, Kubernetes timeZone on CronJob)
• Daylight saving transitions that skip or repeat an hour
• Clock skew between nodes and the monitoring service
• Manual schedule edits that shift a job by an hour without updating the monitor expectation
• Catch-up behavior in some schedulers after downtime (burst of runs, then quiet)

Mitigations:

1. Store schedules in UTC when possible; document local business time separately
2. Align the Seiri monitor’s cron/interval with the scheduler’s actual expression
3. Use grace periods large enough for DST edges on daily jobs
4. Prefer “expected interval since last success” thinking for frequent jobs (every 5–15 minutes) — a missed window is obvious without arguing about absolute clock faces
5. After any infra migration, treat “first successful ping received” as part of the cutover checklist

Drift is also why “alert if local hour == 2 and process not running” home-grown scripts rot. The heartbeat contract is simpler: check in when done; alert if late.

For related troubleshooting when cron itself is the suspect, see cron job not running and cron job failed silently.

When to use a heartbeat monitor

Use heartbeats when:

• Work is scheduled or recurring and silence is bad (backups, ETL, reports, cleanups, renewals)
• Failure modes include never started or never finished, not only thrown exceptions
• There is no useful public URL to probe
• You need a clear ownership signal: one monitor per critical job
• Egress is limited to email or a single outbound HTTPS call

Skip or deprioritize heartbeats when:

• You only care about HTTP availability of a public site (use an uptime tool)
• You need deep request tracing and dependency maps (use APM)
• You are debugging a single failed run and need the log body (use logs — after the heartbeat already told you when)

Heartbeats complement the rest of the stack. They do not replace log search when you are root-causing a /fail ping. They tell you to start looking.

Workloads that benefit most

Putting it together: a reliability checklist

1. List jobs where “missed once” hurts (data loss, money, compliance, customer trust)
2. Create one Seiri monitor per job with a realistic schedule
3. Ping /success only after verification, not after “script started”
4. Ping /fail (or trap) on known failure paths
5. Set grace from measured runtime, not guesswork alone
6. Route alerts to a channel humans see (on-call, Slack) — not only a ignored inbox
7. After deploys that touch crontab, Helm values, or CronJob YAML, confirm a ping still arrives
8. Link runbooks to the monitor so the page includes “what this job is”

Seiri is push/heartbeat monitoring for cron, workers, and Kubernetes — free forever to start at cloud.seiri.app. Pricing details live at /pricing.

Designing check-ins that mean something

A heartbeat is only as useful as the definition of “done.” Teams often wire the ping too early and recreate silent failure with better branding.

Anti-pattern: ping at start

# BAD — proves the shell started, not that work finished
0 2 * * * curl -fsS https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF; /usr/local/bin/backup.sh

If backup.sh hangs or fails after the curl, Seiri is satisfied and you are not.

Anti-pattern: ping unconditionally

# BAD — always checks in even when backup fails
0 2 * * * /usr/local/bin/backup.sh; curl -fsS https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF

The ; runs the ping regardless of exit status. Prefer && for success-only, or an explicit /success / /fail branch.

Better: verify then ping

For backups, verification might mean:

• Compressed file size above a floor
• aws s3api head-object succeeds on the new key
• A restore smoke test against a throwaway instance (heavier, for critical data)

For ETL:

• Row counts in the destination within an expected band
• Watermark / max(updated_at) advanced past the previous run
• No rows left in a “poison” or DLQ table beyond threshold

Encode those checks in the job, then ping. The monitor’s job is time; your job’s job is truth.

Start vs finish monitors

Some teams want two signals: “started on time” and “finished on time.” That can help when runs are long and you want early warning of a missing start. Use two monitors and two refs — do not overload one URL with ambiguous meaning. Most teams should start with finish-only until they feel a real need for start detection.

Alerting and on-call hygiene

Heartbeats fail in production when alerts are noisy or invisible.

Route by severity. A missed nightly backup may page. A missed “vacuum optional analytics table” job may open a ticket. If everything pages, nothing pages.

Include context. The alert should name the job, the expected schedule, the last successful check-in time if available, and a link to the runbook. “Monitor X is down” without X’s meaning wastes the first ten minutes of response.

Avoid email-only for critical paths. Email is fine as a secondary channel. Primary pages belong where the on-call already lives (phone, SMS, PagerDuty/Opsgenie-style routing, or a watched Slack channel with escalation).

Test deliberately. In staging, pause the crontab or set suspend: true and confirm Seiri alerts within grace. Untested monitors are décor.

After false alerts, fix the model — not the mute button. Wrong grace, wrong cron expectation, or DST mismatch should change the monitor config. Muting teaches the team to distrust the system.

Heartbeats across environments

Promote the pattern the same way you promote code:

1. Create monitors for staging with non-paging channels
2. Prove /success, /fail, and missed-ping paths
3. Create production monitors with paging
4. Keep refs and org slugs out of public git if your threat model requires it (Secrets, CI variables)
5. On teardown of an environment, disable or delete monitors so stale expectations do not page forever

Config-as-code for monitors is nice when you have it; a spreadsheet of “job → monitor → owner” is better than tribal knowledge. The operational invariant is: every critical schedule has a named human and a named monitor.

Integrating with existing cron estates

You do not need to rewrite every job on day one.

Phase 1 — money and data. Backups, billing batches, ledger exports, certificate renewals.

Phase 2 — customer-visible batch. Reports, notification digests, search index rebuilds.

Phase 3 — hygiene jobs. Log rotation, temp cleanup, vacuum — only if missing them causes incidents.

Wrap legacy scripts without invasive refactors:

#!/usr/bin/env bash
# /usr/local/bin/with-seiri-ping
set -euo pipefail
PING="$1"
shift
if "$@"; then
  curl -fsS -m 10 --retry 3 "${PING}/success"
else
  status=$?
  curl -fsS -m 10 --retry 3 "${PING}/fail" || true
  exit "$status"
fi

0 2 * * * /usr/local/bin/with-seiri-ping \
  https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF \
  /usr/local/bin/backup.sh

Same wrapper works for many binaries. Language-specific SDKs are unnecessary for most shells and cron estates — HTTP GET is the integration.

For Kubernetes estates, combine Job pings with seiri-kube-agent where images cannot change. For framework schedulers, see the Sidekiq, Celery, and Laravel posts linked from the blog index.

Conclusion

Heartbeat monitoring flips the default assumption of ops tooling. Instead of asking “is something responding when I poke it?”, you ask “did the work check in when it was supposed to?”

That single question — Did it run? — covers deleted crontabs, hung uploads, suspended CronJobs, and workers that died between deploys. Logs, metrics, and APM remain useful for other questions. They are incomplete substitutes for a push check-in with a grace period.

Implement the smallest version today: one critical backup, one curl after success, one alert channel. Expand to ETL, reports, and Kubernetes CronJobs once the pattern is habit.

Further reading:

• Pillar: Heartbeat monitoring
• Pillar: Cron job monitoring
• Dead man’s switch for engineers
• Scheduled job reliability
• Complete cron monitoring guide
• Backup failed silently