Scheduled Job Reliability: Preventing Silent Failures

Scheduled job monitoring is the practice of verifying that recurring work — batch jobs, ETL, reports, cleanups, renewals — actually ran and finished on time.

It overlaps with cron job monitoring and heartbeat monitoring, but the intent is broader than Linux cron: any scheduler can go quiet. Kubernetes CronJobs, Airflow, Sidekiq, Celery, Laravel’s scheduler, Windows Task Scheduler, and homegrown loops all share the same failure shape.

The question to ask every night: Did it run?

Seiri is push/heartbeat monitoring for that question — not website uptime, not APM, not log management. Free forever at cloud.seiri.app. Pillar: scheduled job monitoring.

Short answer

Give each critical schedule its own monitor, expected cadence, and grace period. Ping Seiri when the job finishes successfully (https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF or /success). Alert when the ping is late. Optionally ping /fail when the job knows it broke. That dead man’s switch pattern catches missed and hung runs that logs and uptime probes miss.

Why scheduled jobs fail silently

User-facing APIs fail loudly. Scheduled work fails in conference rooms three weeks later.

The scheduler is healthy; the job definition is not

Airflow UI:       green
DAG file on disk: deleted in a bad deploy
Result:           no runs; dashboards still render cached data

Or the crontab user changed, the systemd timer was disabled, the Laravel schedule stopped being invoked from cron, or a Sidekiq Enterprise recurring job was removed from Redis. Infrastructure health checks stay green because they never knew your job’s name.

The job starts and never finishes

def nightly_etl():
    extract()          # OK
    transform()        # blocks forever on a lock
    load()             # never reached
    ping_seiri()       # never reached

Process lists show “busy.” Queue depths look plausible. Nobody gets an exception. Morning reports are empty.

The job finishes with the wrong meaning of success

#!/usr/bin/env bash
# "Success" = script did not crash — not "S3 object is restore-tested"
pg_dump "$URL" | gzip > /tmp/b.sql.gz
aws s3 cp /tmp/b.sql.gz s3://backups/latest.sql.gz || true
exit 0

Upload failures become local-only dumps. Timestamps in object storage go stale. See backup failed silently.

Downstream still looks fine

BI tools show last week’s numbers with no banner that says “pipeline stopped.” Finance opens a CSV that was regenerated from cache. Customers do not file tickets about invoices they never knew to expect. Silence has no ticket ID.

Scenario: six-hour ETL that skipped a window

An ETL job runs at 00:00, 06:00, 12:00, 18:00. During a cloud provider blip, the 06:00 run hangs until 11:40. With overlapping runs allowed, 12:00 starts too. Locks escalate. Both fail or one no-ops. Without a heartbeat per window, you discover the gap when a KPI dips at noon — and you lack a clear “which window missed?” signal.

Scenario: report job after a host migration

App servers move to new VMs. HTTP uptime checks are updated. The report crontab on the old box is not. Executives stop getting the 06:00 PDF. Support volume does not spike. Two weeks later someone asks where the email went.

These are the same class of incidents as cron job failed silently and cron job not running — generalized across schedulers.

Common monitoring approaches

Logs

tail -f /var/log/etl.log
# or: kubectl logs, CloudWatch, Loki, etc.

Logs are forensics. They require knowing which host, which stream, and which time range. They do not page you when no log line appears because the scheduler never started the job. Retention and volume filters make “absence” hard to alert on correctly.

Metrics

Counters like etl_runs_total and gauges like etl_last_success_unixtime can power staleness alerts. That is a hand-rolled dead man’s switch. It works when every job is instrumented and every alert rule is owned. It falters when half the estate is “just a cron on a box” without an exporter.

Error / exception alerts

Necessary for thrown failures. Blind to never-started and many hung cases. Blind to swallowed errors (except: pass, || true).

Uptime probes

Wrong tool for batch schedules. There is usually no URL that means “last night’s ledger export completed.”

Heartbeat / push check-ins

The job announces completion. Silence announces trouble. This is the primary approach for scheduled job reliability with Seiri — aligned with heartbeat monitoring and cron job monitoring.

Implementation patterns

Canonical Seiri URLs:

https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF
https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success
https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/fail

Email when HTTP egress is blocked: YOUR_REF@ping.seiri.app — see email heartbeat monitoring.

One monitor per job

Do not share a single monitor across unrelated schedules. Ownership, grace, and alert routing differ. Name monitors after the business job (ledger-export-daily), not after the host.

Linux cron / bash

# ETL every 6 hours — ping only after success
0 */6 * * * /opt/etl/run.sh && \
  curl -fsS -m 10 --retry 3 \
  https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF

With explicit success/fail:

0 */6 * * * \
  /opt/etl/run.sh \
  && curl -fsS -m 10 --retry 3 \
       https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success \
  || curl -fsS -m 10 --retry 3 \
       https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/fail

Wrapper with verification

#!/usr/bin/env bash
set -euo pipefail

PING="https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF"
STAGING="/var/data/etl-out"
MIN_ROWS=1

/opt/etl/run.sh
ROWS=$(wc -l < "$STAGING/facts.csv")
if [ "$ROWS" -lt "$MIN_ROWS" ]; then
  curl -fsS -m 10 --retry 3 "${PING}/fail" || true
  echo "etl produced $ROWS rows; refusing success ping" >&2
  exit 1
fi

curl -fsS -m 10 --retry 3 "${PING}/success"

Ping after the artifact check, not after “process started.”

Python batch

import sys
import requests

PING = "https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF"


def ping(suffix: str = "") -> None:
    requests.get(f"{PING}{suffix}", timeout=10).raise_for_status()


def run_etl() -> None:
    rows = extract()
    if not rows:
        raise RuntimeError("extract returned no rows")
    transform_and_load(rows)


def main() -> int:
    try:
        run_etl()
        ping("/success")
        return 0
    except Exception:
        try:
            ping("/fail")
        except Exception:
            pass
        raise


if __name__ == "__main__":
    sys.exit(main())

systemd timer

# etl.service
[Unit]
Description=Six-hour ETL with Seiri heartbeat

[Service]
Type=oneshot
ExecStart=/opt/etl/run.sh
ExecStartPost=/usr/bin/curl -fsS -m 10 --retry 3 https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success

# etl.timer
[Unit]
Description=Run ETL every 6 hours

[Timer]
OnCalendar=*-*-* 00/6:00:00
Persistent=true

[Install]
WantedBy=timers.target

If the timer is disabled, Seiri still alerts on missing check-ins — which is what you want.

Kubernetes CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: ledger-export
spec:
  schedule: "0 6 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      activeDeadlineSeconds: 7200
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: export
              image: ledger-export:stable
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  /export.sh
                  curl -fsS -m 10 --retry 3 \
                    https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success                  

For operator-based checks without embedding curl, use seiri-kube-agent HealthCheck CRs — details in Kubernetes CronJob monitoring, chart docs at seiri-kube-agent, product page /kubernetes-agent.

Framework-shaped jobs (Sidekiq / Celery / Laravel)

The pattern is identical: after the unit of work succeeds (and verifies), HTTP GET the ping URL. Dedicated guides:

• How to monitor Sidekiq
• How to monitor Celery
• How to monitor Laravel scheduler

Pseudo-pattern for any worker:

@scheduled(every=timedelta(hours=6))
def export_ledger():
    path = build_export()
    assert_export_sane(path)
    requests.get(
        "https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success",
        timeout=10,
    )

Grace periods for batch and ETL

Long jobs need patience. Impatient monitors train people to ignore pages.

Grace period = slack after the expected completion window before alerting.

Guidance:

• Measure p50/p95 duration if you can; otherwise time a few production-like runs
• Include scheduler delay (queue lag, Kubernetes pull time)
• Keep grace shorter than the business impact window (e.g. “finance needs the file by 09:00”)
• Use separate monitors for “fast path every 15 minutes” vs “heavy month-end”
• Let /fail alert immediately when the job can report; let silence wait for grace

Example for a six-hour ETL:

Schedule:        00:00, 06:00, 12:00, 18:00
Typical runtime: 35–50 minutes
Grace:           ~90 minutes after expected start
Overlap policy:  forbid concurrent runs

If a run is still going when the next window starts, fix concurrency — do not hide it with a 12-hour grace.

Schedule drift across schedulers

Drift is not only NTP skew. It is any mismatch between when you think work runs and when the monitor expects a ping.

Sources:

• Timezone and DST differences between app config, host TZ, and monitor cron
• “Every 6 hours” meaning wall-clock slots vs “six hours after last finish”
• Catch-up / backfill behavior after downtime (Airflow, some queue schedulers)
• Manual edits in one environment not promoted to others
• Kubernetes timeZone vs UTC-only assumptions

Mitigations:

1. Prefer UTC for technical schedules; document civil-time business meaning separately
2. Change monitor expectations in the same PR as scheduler config
3. For interval workers, alert on max time since last success rather than absolute clock faces
4. Widen grace slightly around DST for daily civil-time jobs without masking multi-hour outages
5. After migrations, require a observed ping before declaring victory

Heartbeats reduce arguments about clocks: either the check-in arrived in the allowed window or it did not.

Batch alerting without drowning in noise

Reliability dies when every monitor is critical and none are trusted.

Practices:

• Severity by blast radius — backups and money movement page; optional cleanup jobs ticket
• Deduplicate — one alert per missed window, not a storm per retry
• Runbooks in the alert — job purpose, owner, where artifacts live, how to re-run safely
• Separate start vs finish only if you need duration awareness; most teams only need finish /success
• Channels humans use — on-call and Slack beat digests nobody opens
• Staging monitors — prove the negative path (pause job → page fires) before production cutover

Keyword mapping (same product story):

When to use a heartbeat monitor

Use heartbeats when:

• Recurring work must happen for data integrity, revenue, compliance, or customer trust
• Failure includes never-started and never-finished, not only exceptions
• There is no meaningful public URL to probe
• Multiple schedulers exist and you want one operational pattern

Use other tools when:

• You need multi-region HTTP availability → uptime product
• You need request traces and dependency maps → APM
• You need to search log bodies for a known incident → log platform

Heartbeats tell you that a window failed. Logs and APM help you learn why after /fail or a missed ping.

Backfill, catch-up, and idempotency

Scheduled job reliability is not only “did the last window fire?” It is also “what happens after a miss?”

Idempotent jobs

If a job can safely re-run for the same business day, responders can fix forward: run by hand, ping /success, move on. Design exports and ETL with deterministic keys, upserts, or partition overwrites so panic re-runs do not duplicate charges or double-send email.

Non-idempotent jobs

Payment capture, one-shot notifications, and irreversible ledger posts need explicit backfill procedures. The heartbeat tells you a window was missed; it does not invent a safe replay. Document:

• Whether to skip the window (accept loss) or replay with a dated parameter
• Which locks or “already processed” tables to consult
• Who approves replays that move money

Catch-up storms

Some schedulers, after downtime, attempt to run every missed tick. That can overload databases and produce confusing ping bursts. Prefer:

• concurrencyPolicy: Forbid (Kubernetes) or equivalent single-flight locks
• Schedulers configured not to backfill blindly
• Monitors that alert on the first miss so humans intervene before a storm

Example single-flight lock in bash:

#!/usr/bin/env bash
set -euo pipefail
LOCK=/var/lock/etl.lock
exec 9>"$LOCK"
if ! flock -n 9; then
  echo "previous run still active" >&2
  exit 0   # optional: ping a different "overlap" monitor instead
fi
/opt/etl/run.sh
curl -fsS -m 10 --retry 3 \
  https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success

If overlapping skips are unacceptable for SLA, do not exit 0 on lock failure — exit non-zero and /fail, or use a dedicated monitor that expects progress every window regardless.

Ownership and the schedule inventory

Silent failures thrive in orphan schedules — jobs nobody claims.

Maintain a living inventory:

Rules of thumb:

• No owner → no production schedule (or accept it as best-effort without pages)
• Owner changes in the same PR as CODEOWNERS / team routing updates
• Decommissioned jobs disable monitors the same day to prevent zombie pages

This inventory is also how you prioritize heartbeat rollout: sort by blast radius, not by how easy the curl is.

Migrating from “email ourselves” to heartbeats

Many teams already have crude switches: cron mails stdout to a shared inbox, or a script posts “OK” to Slack.

Problems with that approach:

• No grace semantics — late and missing look the same as spam
• No structured state for “last success”
• Humans mark as read without action
• Hard to cover Kubernetes and VMs uniformly

Migration path:

1. Keep the email/Slack note temporarily
2. Add Seiri pings alongside
3. Route Seiri to on-call for critical jobs
4. Remove the human-parsed OK messages once the page path is trusted

You are not abandoning communication — you are making absence machine-detectable.

Example: end-to-end daily export

#!/usr/bin/env bash
# daily-ledger-export.sh
set -euo pipefail

PING="https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF"
DAY="${1:-$(date -u +%F)}"
OUT="/var/exports/ledger-${DAY}.csv"

/usr/local/bin/ledger-export --day="$DAY" --out="$OUT"

# Basic sanity: header + at least one data row
LINES=$(wc -l < "$OUT")
if [ "$LINES" -lt 2 ]; then
  curl -fsS -m 10 --retry 3 "${PING}/fail" || true
  echo "export too small: $LINES lines" >&2
  exit 1
fi

aws s3 cp "$OUT" "s3://company-exports/ledger/${DAY}.csv"
aws s3api head-object --bucket company-exports --key "ledger/${DAY}.csv" >/dev/null

curl -fsS -m 10 --retry 3 "${PING}/success"

Crontab:

0 6 * * * /usr/local/bin/daily-ledger-export.sh >>/var/log/ledger-export.log 2>&1

Seiri monitor: daily at 06:00 UTC, grace long enough for export + upload variance, alert to finance on-call. After any miss: check inventory row, run with an explicit --day, confirm S3 object, confirm ping.

That is scheduled job reliability in practice — not a dashboard wallpaper.

Reliability checklist

1. List schedules where one miss hurts
2. Create one Seiri monitor per schedule with matching cadence
3. Ping /success only after verification (rows, object size, checksum, row counts)
4. Ping /fail on known error paths
5. Set grace from real durations + scheduler skew
6. Forbid unsafe overlap for stateful jobs
7. Route to on-call; attach runbooks
8. Include “confirm ping” in deploy/migration checklists
9. Cover Kubernetes with Job pings and/or seiri-kube-agent
10. Review monitors when schedules change — treat config drift as an incident waiting to happen
11. Document backfill/idempotency per job
12. Keep a schedule inventory with owners

Comparing scheduler families (same heartbeat contract)

You do not need five monitoring products for five schedulers. You need one check-in contract and adapters. That is the point of Seiri’s push model and the reason sister posts exist for frameworks and Kubernetes rather than inventing a new philosophy per tool.

SLOs for scheduled work

Translate reliability into language finance and product understand:

• Freshness SLO: “Warehouse facts for day D available by 07:00 UTC”
• Success ratio: “At least N of the last M windows checked in” (use carefully — consecutive misses matter more for backups)
• Mean time to detect: should be bounded by grace, not by “someone opened the dashboard”

Heartbeats primarily improve detection. They do not automatically improve mean time to repair — runbooks and idempotent re-runs do. Measuring only “monitors configured” without paging tests overstates readiness.

Example freshness framing for an export:

Business need:  CSV in s3://exports/ledger/YYYY-MM-DD.csv by 07:00 UTC
Job schedule:   06:00 UTC
Expected ping:  ~06:20–06:40
Grace → page:   07:00 at latest (aligned to SLO, not vanity tightness)

Aligning grace to the SLO prevents both early noise and late discovery.

Security notes for ping URLs

Ping URLs are capabilities: anyone who can GET them can mark success. Treat refs like weak secrets:

• Prefer Secrets / CI variables over committing production refs to public repos
• Rotate refs if they leak
• Do not put ping URLs in client-side JavaScript or mobile apps
• NetworkPolicies that allow only the Job’s ServiceAccount egress reduce accidental exposure from other pods
• /fail is also a capability — abuse can page you; still better than silent miss for most teams, but know the trade-off

Email heartbeats inherit mailbox security — protect SMTP credentials and treat the YOUR_REF@ping.seiri.app address as sensitive.

Conclusion

Scheduled job reliability is not “the scheduler process is up.” It is “each critical window produced a verified result on time.” Logs, metrics, and exception trackers help, but they leave gaps exactly where silent failures live.

A heartbeat — a dead man’s switch on each job — closes that gap. Implement it with Seiri push URLs, optional email check-ins, grace periods tuned to batch reality, and the same discipline on Kubernetes as on VMs.

Further reading:

• /scheduled-job-monitoring
• /heartbeat-monitoring
• /cron-job-monitoring
• Heartbeat monitoring explained
• Kubernetes CronJob monitoring
• Complete cron guide
• /pricing · cloud.seiri.app