How to Monitor Celery: Heartbeats for Periodic Tasks

Flower shows workers online. Prometheus scrapes celery_task_* metrics. Redis (or RabbitMQ) looks healthy. The nightly cleanup that was supposed to run at 02:00 did not.

Celery monitoring in the Seiri sense is not “are workers up?” — that is process / queue telemetry. The question for periodic and critical tasks is: Did it run?

Seiri is push / heartbeat monitoring, not uptime or APM. Your task pings when the work finishes. Missed ping past grace → alert. Same dead man’s switch pattern as cron job monitoring, applied to Celery Beat and workers.

The problem: Beat, broker, and task are three failure domains

A Celery deployment has at least three moving parts:

1. Celery Beat — decides when to enqueue periodic tasks.
2. Broker — Redis, RabbitMQ, SQS, etc. — holds messages.
3. Workers — execute tasks.

Any one can fail while the others look fine:

• Beat dies or loses its schedule → no messages, empty queues, idle workers that still answer pings.
• Broker accepts messages but a routing key typo sends them to a queue nobody consumes.
• Worker runs the task, hits a soft time limit (or never does), and the business side effect never completes.
• Task code returns early on a “nothing to do” branch and Celery records SUCCESS.

Dashboards that only show worker liveness will not page you for a dead Beat. Task success counters will not page you for a no-op success.

Why Celery tasks fail silently

Beat process down or schedule missing

Beat is a single scheduler process (or RedBeat / custom scheduler). If it exits after a deploy, or beat_schedule keys change and drop an entry, periodic work simply stops. Workers keep consuming whatever is left in the queues.

Soft / hard time limits vs hangs

soft_time_limit and time_limit help, but only if you set them. Without them, a blocked requests.get can occupy a pool slot indefinitely. Visibility: busy worker, no FAILURE state, no new SUCCESS for that periodic task name.

Autoretry and acks_late edge cases

acks_late=True and aggressive retries can leave work in a confusing state after worker death. You may eventually get a failure — or you may get a long gap with no completion signal. Heartbeats treat “no completion” as the alertable event.

SUCCESS that means nothing

@shared_task
def sync_partners():
    partners = Partner.active()
    if not partners:
        return  # Celery: SUCCESS — business: maybe a config bug
    for p in partners:
        p.sync()

Empty queryset after a bad migration looks identical to a quiet Sunday. Prefer raising (or pinging /fail) when zero rows is unexpected for that job.

Routing and queue misconfiguration

After renaming queues or changing task_routes, tasks land on queues with no consumers. Flower may show the message briefly; days later the only evidence is stale data.

Common approaches (and gaps)

Use metrics for capacity. Use heartbeats for scheduled job monitoring of the tasks that hurt when silent.

Implementation: heartbeats from Celery tasks

Ping URL

In Seiri, create a monitor and use:

https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF
https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success
https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/fail

Align the monitor’s cron/interval with Beat’s schedule. Give grace for normal runtime (if cleanup usually takes 20 minutes, do not set a 2-minute grace).

Minimal periodic task

import requests
from celery import shared_task

SEIRI_PING = "https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF"

@shared_task(bind=True, max_retries=3, soft_time_limit=600, time_limit=900)
def nightly_cleanup(self):
    do_cleanup()  # raise on failure
    requests.get(SEIRI_PING, timeout=10)

Beat schedule (Django-style example):

# settings.py / celeryconfig
beat_schedule = {
    "nightly-cleanup": {
        "task": "myapp.tasks.nightly_cleanup",
        "schedule": crontab(hour=2, minute=0),
    },
}

If Beat dies, nightly_cleanup never runs, the ping never arrives, Seiri alerts.

Success / fail paths

import requests
from celery import shared_task
from celery.exceptions import SoftTimeLimitExceeded

BASE = "https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF"

def seiri(status: str | None = None) -> None:
    url = f"{BASE}/{status}" if status else BASE
    try:
        requests.get(url, timeout=10)
    except requests.RequestException:
        # log; do not hide the original task error
        pass

@shared_task(bind=True, soft_time_limit=600, time_limit=900)
def invoice_export(self):
    try:
        path = build_export()
        upload(path)
        seiri("success")
    except SoftTimeLimitExceeded:
        seiri("fail")
        raise
    except Exception:
        seiri("fail")
        raise

Silence (no success, no fail) still trips the missed-check-in alert. That covers “task never enqueued” and “worker died before the finally block.”

Optional: Beat liveness vs task completion

Two different monitors:

1. Task completion — ping at end of nightly_cleanup (business signal).
2. Scheduler pulse — a tiny task every 5 minutes that only pings YOUR_ORG:beat-pulse.

If (2) alerts but (1) does not, workers or task code are the problem. If (1) alerts and (2) is fine, the specific schedule entry or task path broke. Start with (1) for the jobs that matter; add (2) when you have been burned by Beat outages.

Signals (advanced)

Celery’s task_success / task_failure signals can centralize pings:

from celery.signals import task_success, task_failure

CRITICAL = {
    "myapp.tasks.nightly_cleanup": "YOUR_ORG:cleanup",
    "myapp.tasks.invoice_export": "YOUR_ORG:invoices",
}

@task_success.connect
def on_success(sender=None, **kwargs):
    ref = CRITICAL.get(sender.name if sender else "")
    if ref:
        requests.get(f"https://ping.seiri.app/webhook/{ref}/success", timeout=10)

@task_failure.connect
def on_failure(sender=None, **kwargs):
    ref = CRITICAL.get(sender.name if sender else "")
    if ref:
        requests.get(f"https://ping.seiri.app/webhook/{ref}/fail", timeout=10)

Only map tasks with a known cadence. Do not heartbeat every apply_async fan-out.

Example with Seiri

Scenario: nightly DB cleanup + partner sync.

1. Create two monitors in Seiri with schedules matching Beat.
2. Ping /success only after cleanup commits and after sync confirms HTTP 2xx from the partner API.
3. Set soft_time_limit / time_limit so hung tasks die and can emit /fail.
4. Send a test nightly_cleanup.delay() and confirm the monitor is healthy.
5. Stop Beat in staging; wait for grace; confirm the alert.

For hosts that cannot open outbound HTTPS, send mail to YOUR_REF@ping.seiri.app instead — see email heartbeat monitoring.

Django, Flask, and plain Celery apps

The heartbeat placement does not change with the web framework. Only the config location does.

Django + django-celery-beat: schedules often live in the database. A row deleted in admin stops enqueues with no code diff. Heartbeats on the task body still catch that — the ping stops even though workers are healthy. Prefer code-reviewed schedules for critical jobs, or alert on both Beat pulse and task completion.

RedBeat / custom schedulers: same rule — monitor task completion first. Scheduler process uptime alone will not tell you a single crontab entry was removed.

Canvas / chords: ping when the business chord finishes (callback task), not on every partial. Otherwise a failed body with a successful header chord confuses the monitor.

from celery import chord
from myapp.tasks import shard_work, finalize_export, seiri

@shared_task
def kick_export():
    header = [shard_work.s(i) for i in range(8)]
    chord(header)(finalize_export.s())

@shared_task
def finalize_export(results):
    upload_combined(results)
    seiri("success")  # one ping for the whole pipeline

Idempotency vs “ping once”

Periodic tasks should be safe to retry. Heartbeats should still mean “this schedule window produced a good outcome.” Patterns that work:

• Ping after the idempotent write commits (upsert, object overwrite with known key).
• If the task no-ops because work was already done by a previous successful run in the same window, you may still ping success — the business invariant holds.
• If the task no-ops because of a bug (wrong day key, empty filter), raise instead of pinging.

@shared_task(bind=True, max_retries=5)
def daily_ledger_close(self, day=None):
    day = day or yesterday()
    if LedgerClose.objects.filter(day=day, status="done").exists():
        seiri("success")  # already closed — invariant OK
        return
    close_ledger(day)  # raises on failure
    seiri("success")

Local verification

# shell / manage.py shell
from myapp.tasks import nightly_cleanup
nightly_cleanup.delay()
# or eager for debugging:
nightly_cleanup.apply()

Confirm the Seiri monitor flips healthy. Then stop the worker (not only Beat) mid-run in staging and confirm you get either /fail or a missed check-in — depending on whether the process could still HTTP out.

Store refs in environment variables:

import os
SEIRI_CLEANUP = os.environ["SEIRI_CLEANUP_REF"]  # YOUR_ORG:cleanup

Best practices

1. Heartbeat the business task, not only “worker process up.”
2. One Seiri monitor per critical periodic task — cleanup ≠ billing ≠ backup.
3. Raise when empty work is unexpected — do not confuse quiet success with a config bug.
4. Set time limits on long tasks; pair with /fail when possible.
5. Keep Flower/Prometheus for capacity; use Seiri for missed runs.
6. Document the schedule in one place — Beat config and Seiri monitor must agree after deploys.
7. Kubernetes — Celery in-cluster can still HTTP ping; for broader workload checks see seiri-kube-agent docs and seiri.app/kubernetes-agent.
8. Do not heartbeat every autonomous apply_async — only cadence-bound work.
9. Treat refs like secrets — rotate if they appear in public logs or tickets.

Conclusion

Celery metrics tell you about workers and queues. They do not reliably tell you that a periodic business task finished. Push a Seiri heartbeat from the success path of each critical task (and optionally a Beat pulse). When the check-in stops, you know before customers do.

Free forever at cloud.seiri.app. Related: how to monitor Sidekiq, heartbeat monitoring explained, cron job monitoring complete guide.