Kubernetes CronJob Monitoring Guide

Kubernetes CronJobs fail in ways that look healthy on a cluster dashboard: the schedule YAML is valid, the last Job succeeded days ago, nodes are Ready, and nobody is paging — until backups are gone or the ledger is stale.

Kubernetes CronJob monitoring means detecting when a CronJob did not run, hung, or completed without doing useful work. That is a heartbeat / dead man’s switch problem — not generic cluster “observability,” not website uptime, and not APM.

The question remains: Did it run?

Seiri approaches CronJobs two complementary ways:

1. HTTP (or email) check-ins from the Job container — the Job pings Seiri when work succeeds
2. seiri-kube-agent — a Helm HealthCheck operator that watches workloads in-cluster and reports health to Seiri

Pillars: Kubernetes CronJob monitoring · Kubernetes agent. Chart docs: seiri-kube-agent. Free forever at cloud.seiri.app.

Why CronJobs fail silently

A CronJob is a controller that creates Jobs on a schedule. Many failure modes never produce a Failed Job object — so “last Job succeeded” is a weak signal.

1. Schedule never creates a Job

spec:
  suspend: true   # left on after maintenance
  schedule: "0 2 * * *"

Or the schedule expression changes, timeZone is misunderstood, or the CronJob is deleted while a Deployment with the same name still looks “fine” in a messy overview UI. The cluster is green. No new Job. No pods. No logs.

2. Job created, Pod pending forever

Image pull secret missing. Resource requests that never fit. Node taints. The Job exists; the container never runs. Controllers may retry depending on backoff. From a business perspective the backup still did not happen.

3. Pod runs and hangs

No activeDeadlineSeconds. Process blocked on NFS, DB lock, or cloud API. kubectl get jobs shows Active. Metrics show CPU near zero. Without a completion signal, you wait until a human notices.

4. Exit 0, empty work

#!/bin/sh
set -e
ROWS=$(psql "$DATABASE_URL" -tAc "select count(*) from staging")
if [ "$ROWS" -eq 0 ]; then
  echo "nothing to do"
  exit 0
fi
run_pipeline

Kubernetes records Success. Downstream is wrong. Heartbeats should fire only after verification — or you deliberately ping /fail when “nothing to do” violates an SLA that expects data nightly.

5. Node drain / eviction mid-run

Depending on restartPolicy, backoffLimit, and whether the Job is still within deadline, you may get retries, a Failed Job, or confusion during incidents. A push check-in on successful completion still tells Seiri whether a good run finished in the expected window.

Scenario: the paused backup nobody unmuted

Platform engineering suspends nightly-backup during a Postgres upgrade. The upgrade succeeds. The suspend flag stays. Three weeks later a node dies and restore tooling finds the last object from before the upgrade. Cluster metrics never lied — they simply never answered Did it run?

Scenario: resource pressure only at 02:00

The CronJob works in staging and in ad-hoc kubectl create job --from=cronjob/... tests at noon. At 02:00 the batch node pool is full of other Jobs. Pods pend until the next day. A heartbeat monitor pages when the morning check-in is missing — before traders open dashboards on empty data.

Common monitoring approaches (and gaps)

Logs

kubectl logs job/nightly-backup-28439210

Useful after you know which Job name to inspect. Useless when no Job was created. Ephemeral; gone when the Job is deleted by history limits (successfulJobsHistoryLimit).

Metrics

kube-state-metrics and custom exporters can expose CronJob schedules and Job statuses. You can alert on “no successful Job in N hours.” That is effectively a dead man’s switch with more YAML. It still does not know whether the container’s exit 0 meant “backup uploaded and verified.”

Generic alerting on Failed Jobs

Necessary, but incomplete. Many of the worst incidents never produce Failed — they produce nothing, or Success with empty artifacts.

Heartbeat / push monitoring

The Job (or an in-cluster operator) reports completion to Seiri. Missed expected check-in → alert. This matches cron job monitoring on VMs, with Kubernetes-specific wiring below.

Option A: HTTP ping from the Job

Create a monitor in Seiri, then ping at the end of a successful run:

https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF
https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success
https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/fail

Minimal CronJob example

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
  namespace: batch
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      activeDeadlineSeconds: 3600
      backoffLimit: 1
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: ghcr.io/example/backup:1.4.2
              env:
                - name: SEIRI_PING
                  value: "https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF"
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  /backup.sh
                  curl -fsS -m 10 --retry 3 "${SEIRI_PING}/success"                  

If the CronJob never starts a Job, the ping never arrives — Seiri alerts after the grace period.

Success and fail paths

command:
  - /bin/sh
  - -c
  - |
    set +e
    /backup.sh
    status=$?
    if [ "$status" -eq 0 ]; then
      curl -fsS -m 10 --retry 3 "${SEIRI_PING}/success"
      exit 0
    fi
    curl -fsS -m 10 --retry 3 "${SEIRI_PING}/fail" || true
    exit "$status"    

Init container vs main container

Prefer pinging from the same container that performed the work (or a follow-up step in the same script) so you do not mark success when only an init probe ran. Avoid a second container that always curls success regardless of the sibling’s exit code unless you use a shared volume flag pattern carefully.

NetworkPolicy and egress

If the namespace denies egress, either:

• Allow HTTPS to ping.seiri.app, or
• Use email heartbeats to YOUR_REF@ping.seiri.app if outbound SMTP/API email is available, or
• Rely more on seiri-kube-agent (Option B) which reports from the operator’s network path

Curl in distroless images

Distroless or scratch images may lack curl. Options:

• Multi-stage build that copies a static curl
• Ping from a language HTTP client already in the app
• Sidecar pattern only if exit signaling is correct (easy to get wrong)
• HealthCheck CR via the agent so the app image stays minimal

Python example inside the Job:

import os
import sys
import requests

PING = os.environ["SEIRI_PING"]


def main() -> int:
    try:
        run_backup()
        requests.get(f"{PING}/success", timeout=10).raise_for_status()
        return 0
    except Exception:
        try:
            requests.get(f"{PING}/fail", timeout=10)
        except Exception:
            pass
        return 1


if __name__ == "__main__":
    sys.exit(main())

Option B: seiri-kube-agent (HealthCheck operator)

For Deployments, Pods, Jobs, StatefulSets, DaemonSets — and CronJobs — install the agent and declare HealthCheck custom resources. The operator watches cluster state and reports to Seiri without requiring every image to embed curl.

Install via Helm repo

helm repo add seiri https://seiri-app.github.io/seiri-kube-agent
helm repo update
helm install seiri-agent seiri/seiri-kube-agent \
  --namespace seiri-system \
  --create-namespace \
  --set seiri.authToken="<YOUR_AUTH_TOKEN>" \
  --set seiri.clusterId="<YOUR_CLUSTER_ID>"

Install via OCI (GHCR)

helm install seiri-agent oci://ghcr.io/seiri-app/charts/seiri-kube-agent \
  --namespace seiri-system \
  --create-namespace \
  --set seiri.authToken="<YOUR_AUTH_TOKEN>" \
  --set seiri.clusterId="<YOUR_CLUSTER_ID>"

Docs: https://seiri-app.github.io/seiri-kube-agent/. Product guide: /kubernetes-agent.

HealthCheck CR for a CronJob

apiVersion: monitoring.seiri.app/v1alpha1
kind: HealthCheck
metadata:
  name: check-backup-cron
  namespace: batch
spec:
  targetRef:
    apiVersion: batch/v1
    kind: CronJob
    name: nightly-backup
  interval: 30s
  checkID: "chk-nightly-backup"

The CR ties Kubernetes workload identity to a Seiri check. When the CronJob / Job health diverges from expectations, Seiri can alert without waiting for an application-level curl — useful for pending pods, missing schedules, and workloads you do not want to modify.

When to use both ping and agent

Many teams use agent for cluster truth and ping for business truth on the few jobs where empty success is dangerous (backups, money movement, compliance exports).

Grace periods for Kubernetes schedules

Kubernetes adds scheduling delay on top of application runtime: queueing, image pulls, volume attaches, Admission webhooks.

Set Seiri grace to cover:

• Typical Job duration
• Occasional image pull latency on new nodes
• Cluster autoscaler spin-up if the Job needs a scaled node
• DST / timezone behavior if you use spec.timeZone

Example:

CronJob schedule:     02:00
App runtime:          10–20 minutes
Pull + schedule skew: 0–10 minutes
Grace:                ~45–60 minutes before alert
SLA:                  backup usable by 06:00

Also set activeDeadlineSeconds on the Job so hung runs die instead of holding locks until the next Forbid window.

jobTemplate:
  spec:
    activeDeadlineSeconds: 3600
    template:
      spec:
        restartPolicy: OnFailure

Pair deadline (cluster kills hung Job) with heartbeat grace (Seiri pages if no success). Neither replaces the other.

Schedule drift on CronJobs

Kubernetes 1.27+ supports spec.timeZone on CronJobs. Misalignment with Seiri’s expected cron is a common source of false pages.

spec:
  schedule: "0 2 * * *"
  timeZone: "UTC"

Practices:

1. Prefer UTC in CronJob + monitor unless product owners demand local civil time
2. When using civil time, document DST behavior and widen grace on daily jobs
3. Change Seiri schedule in the same PR as CronJob schedule / timeZone edits
4. Remember startingDeadlineSeconds — if the controller misses a window by too much, it may skip; your heartbeat should still alert on the missed check-in
5. After control-plane upgrades, verify a known CronJob still creates Jobs and still pings

Drift also appears when GitOps reverts a hotfix schedule, or when two environments share manifests but different cluster local times via node config. Heartbeats make the mismatch visible.

Kubernetes-specific best practices

1. concurrencyPolicy: Forbid for backups and financial jobs — overlapping runs corrupt state
2. activeDeadlineSeconds — bound hung Jobs
3. History limits — keep enough Failed/Success Jobs to debug, not infinite etcd growth
4. restartPolicy + backoffLimit — understand whether retries help or hide poison messages
5. Resource requests — pending forever is a silent failure mode; alert via agent or missed ping
6. Secrets for ping URLs — prefer Secrets/CSI over plain env in git if refs are sensitive in your threat model
7. One Seiri monitor per CronJob — clearer ownership than one mega-check
8. Test --from=cronjob after changes — then confirm the Seiri UI received a ping
9. Do not rely only on kube-state-metrics Success — verify business outcome before /success
10. Link runbooks in the alert — include namespace, CronJob name, and restore steps

When to use heartbeat monitoring for CronJobs

Use heartbeats when:

• Missing a run causes data loss, stale analytics, missed billing, or compliance gaps
• You need the same operational model as VM cron (complete guide)
• You want alerts even when no Failed Job object exists

Lean on the agent when:

• You cannot modify the container image
• You need broader workload coverage (Deployments, DaemonSets, etc.)
• Pending / scheduling failures are as important as app-level completion

Use neither as a substitute for:

• Multi-region website probes (wrong product category)
• Full APM traces across microservices
• Centralized log search for deep forensics

Seiri’s niche is push/heartbeat reliability for scheduled and background work — including Kubernetes. See also scheduled job monitoring and cron job failed silently.

Debugging a missed CronJob check-in

When Seiri pages “no ping,” work the cluster systematically. The goal is to separate “never scheduled,” “never ran,” “ran and failed,” and “ran and lied.”

1. Did the CronJob create a Job?

kubectl get cronjob nightly-backup -n batch -o wide
kubectl get jobs -n batch -l job-name --sort-by=.metadata.creationTimestamp
# or list jobs by owner:
kubectl get jobs -n batch | grep nightly-backup

Check suspend, LAST SCHEDULE, and ACTIVE. If suspend is true, unmute it and treat the missing ping as working as designed.

2. If a Job exists, what is its status?

kubectl describe job nightly-backup-<id> -n batch
kubectl get pods -n batch -l job-name=nightly-backup-<id>

Pending pods: events will show pull errors, taints, or insufficient CPU/memory. Failed pods: read logs. Running pods past expectation: check for locks, DNS, or cloud API stalls; activeDeadlineSeconds should eventually stop them.

3. If the Job succeeded, why no ping?

kubectl logs job/nightly-backup-<id> -n batch

Look for curl errors (DNS, NetworkPolicy, TLS). Confirm the image includes curl or your HTTP client. Confirm the ping runs only on the success path and that verification did not exit before the curl. Manually:

kubectl run curl-test -n batch --rm -it --restart=Never \
  --image=curlimages/curl:8.5.0 -- \
  curl -fsS -m 10 -o /dev/null -w "%{http_code}\n" \
  https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF

If manual curl works from the namespace but the Job does not ping, the bug is in the Job script. If manual curl fails, fix egress or switch to email / agent.

4. Confirm the Seiri side

Verify the monitor’s cron expression and timezone match spec.schedule / spec.timeZone. Verify grace. Verify you are looking at the correct org and ref. A successful ad-hoc ping should clear or update the monitor depending on your configuration — use that as a connectivity proof during incidents.

GitOps and change control

CronJobs in Git are easy to break with “small” PRs: a wrong indent on suspend, a schedule typo, a missing secret volume. Require:

1. Review of schedule, timezone, concurrency, and deadline fields
2. Staging apply + forced Job + observed Seiri check-in before production merge
3. Same PR updates Seiri monitor expectations when the schedule changes
4. CODEOWNERS or review from the data/job owner, not only platform

Helm values that template CronJobs deserve the same discipline — a default suspend: true copied from a chart example has caused real outages.

Example of keeping the ping URL in a Secret:

apiVersion: v1
kind: Secret
metadata:
  name: seiri-backup-ping
  namespace: batch
type: Opaque
stringData:
  url: "https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF"
---
# reference in CronJob env
# - name: SEIRI_PING
#   valueFrom:
#     secretKeyRef:
#       name: seiri-backup-ping
#       key: url

Combining kube-state-metrics with heartbeats

If you already scrape kube-state-metrics, keep Failed Job alerts — they catch crash loops quickly. Add heartbeats for:

• No Job created
• Success with bad artifacts
• Cross-cluster consistency with the same operational model as VM cron

Do not delete Prometheus rules the day you add Seiri. Overlap is fine; silence on both systems is the problem. Document which signal pages humans so you do not double-page for the same Failed Job plus /fail ping unless you want that during early rollout.

Multi-cluster and multi-tenant notes

• Use distinct Seiri monitors (and preferably naming that includes cluster) so nightly-backup in prod-eu and prod-us do not share fate
• seiri-kube-agent clusterId should identify the cluster clearly in the Seiri UI
• NetworkPolicies differ per cluster — test egress in each
• Do not assume a chart default for HealthCheck interval fits every workload; tune to how fast you need scheduling anomaly detection versus API load

Implementation checklist

1. Inventory CronJobs where a missed run hurts
2. Create Seiri monitors with cron expectations matching spec.schedule + timezone
3. Add /success ping after verification or install seiri-kube-agent and HealthCheck CRs (or both)
4. Set grace from measured Job duration + scheduling skew
5. Set activeDeadlineSeconds and concurrencyPolicy
6. Allow egress or use email / agent paths
7. Page a human channel; attach runbook links
8. On every CronJob manifest change, require evidence of a fresh check-in in staging
9. Practice the missed-ping debug path in staging once
10. Align GitOps reviews so schedule edits update monitors in the same change

Production patterns worth copying

Pattern: verify object then ping

command:
  - /bin/sh
  - -c
  - |
    set -e
    DAY=$(date -u +%F)
    /backup.sh --out "/tmp/${DAY}.sql.gz"
    # Fail closed on tiny dumps
    BYTES=$(wc -c < "/tmp/${DAY}.sql.gz")
    test "$BYTES" -gt 1048576
    aws s3 cp "/tmp/${DAY}.sql.gz" "s3://backups/${DAY}.sql.gz"
    aws s3api head-object --bucket backups --key "${DAY}.sql.gz" >/dev/null
    curl -fsS -m 10 --retry 3 "${SEIRI_PING}/success"    

Kubernetes Success now correlates with “object exists and is large enough,” not merely “shell exited.”

Pattern: HealthCheck CR plus app ping

Keep the HealthCheck CR for scheduling/pending detection, and keep the app ping for business verification. When only the agent alerts, look at cluster events. When only the ping is missing but Jobs Succeeded, look at the script’s verification and curl. When both fire, start with suspend/schedule/controller issues.

Pattern: forced Job in CI

# after applying CronJob manifests in staging
kubectl create job --from=cronjob/nightly-backup "nightly-backup-ci-${CI_COMMIT_SHORT_SHA}" -n batch
kubectl wait --for=condition=complete "job/nightly-backup-ci-${CI_COMMIT_SHORT_SHA}" -n batch --timeout=30m
# assert Seiri received a ping via API or wait condition you own

Wire this into the pipeline that merges CronJob changes. It is the Kubernetes analogue of “run the cron once after editing crontab.”

Pattern: email fallback

When a namespace’s NetworkPolicy cannot allow ping.seiri.app quickly, ship a short-term email heartbeat from a sidecar or from an in-cluster mail relay to YOUR_REF@ping.seiri.app, and track an issue to restore HTTPS. Document the fallback so the next engineer does not assume curl is present.

What “green cluster” still misses

Cluster dashboards answer:

• Are nodes Ready?
• Are control-plane components healthy?
• Did the last recorded Job have status Succeeded?

They do not answer:

• Was last night’s business window covered?
• Is suspend still true?
• Did Success mean empty work?
• Did prod-us and prod-eu both check in?

Heartbeat monitoring exists to ask those questions explicitly. Seiri is not a replacement for kube-prometheus stacks; it is the missing schedule contract. Full product positioning: /kubernetes-cronjob-monitoring, agent: /kubernetes-agent.

When you roll this out, start with the CronJobs whose miss would force a restore, a refund, or a compliance conversation. Add Job-level /success pings where you can change images; add HealthCheck CRs where you cannot. Expand coverage only after the first pages have proven useful in staging and the runbooks match real cluster namespaces.

Conclusion

Kubernetes makes scheduling declarative — not reliable by default. Controllers, suspend flags, pending pods, and exit-0 empty work all produce “quiet” failures that cluster greenness hides.

Monitor CronJobs by answering Did it run? with push check-ins from the Job, in-cluster HealthChecks via seiri-kube-agent, or both. Tune grace for pull and schedule delay. Align timezones. Bound hung Jobs.

Further reading:

• /kubernetes-cronjob-monitoring
• /kubernetes-agent
• Heartbeat monitoring explained
• Dead man’s switch for engineers
• Cron job not running
• /pricing · cloud.seiri.app