How to Monitor Sidekiq: Heartbeats for Background Jobs

Sidekiq Web is green. Redis is up. Queue latency looks fine. And the nightly report that should have landed in finance’s inbox never ran.

That gap is the real Sidekiq monitoring problem. Most teams instrument Redis and worker processes — uptime and APM questions. What you actually need for critical jobs is a different question: Did it run?

Seiri is push / heartbeat monitoring, not uptime or APM. Your job pings when the work finishes. If the ping never arrives, you get alerted. Same dead man’s switch model you would use for shell cron — applied to Sidekiq workers and scheduled jobs.

The problem: queue health ≠ job success

Sidekiq sits between your app and Redis. A healthy stack can still fail business outcomes:

• The Sidekiq process is alive, but a specific worker class stopped getting enqueued after a deploy.
• A job is “busy” for hours because it hung on a remote API — Sidekiq shows a busy thread, not a failed job.
• Retries exhaust; the last failure sits in the Dead set until someone opens the UI.
• A scheduled job (sidekiq-cron, sidekiq-scheduler, or Solid Queue–style cron) silently stops after a config reload.
• The job returns without raising — Sidekiq marks it successful — while the upload, email, or ledger write never happened.

Process monitors and Redis dashboards answer “is the machinery up?” Heartbeats answer “did this critical unit of work complete?”

Why Sidekiq jobs fail silently

1. Scheduling stops, workers keep running

sidekiq-cron and sidekiq-scheduler store schedules in Redis or YAML. A bad deploy, missing gem in the production bundle, or a Redis flush can drop schedules while worker processes keep processing the ordinary queue. From the outside, Sidekiq “looks fine.”

2. Enqueue never happens

Application code that should call NightlyReportWorker.perform_async gets refactored, gated behind a feature flag, or wrapped in a rescue that swallows the error. No job, no Sidekiq failure, no Dead set entry.

3. Success without outcome

Classic Ruby trap:

def perform
  result = ExternalApi.sync
  return unless result.ok?  # silent no-op — Sidekiq still succeeds
  persist!(result)
end

Sidekiq only cares about exceptions. Guard clauses and swallowed errors look like success.

4. Hung jobs with no deadline

Without sidekiq_options lock: / timeouts / middleware that kills long runners, a stuck HTTP call can occupy a thread indefinitely. Queue depth may stay low; the next scheduled run never completes either if you use unique locks poorly.

5. Ops UI is not an alert

The Dead set and retries tab are excellent for debugging after you know something is wrong. They are not paging. Silent failure is the interval between “job stopped” and “human noticed.”

Common approaches (and what they miss)

You still want Redis and process checks. Add heartbeats for the jobs that hurt when they go quiet — the same pattern as scheduled job monitoring and cron job monitoring.

Implementation: push heartbeats from Sidekiq

Seiri ping URL

Create a monitor in Seiri (free forever). You get a check-in URL:

https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF

Optional status paths:

https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/success
https://ping.seiri.app/webhook/YOUR_ORG:YOUR_REF/fail

Set the monitor schedule to match the job (e.g. daily at 02:00 with a grace period that covers normal runtime variance).

Rule: ping only after the real work succeeds. A ping before the upload finishes creates false confidence.

Shared helper

# lib/seiri.rb
require "net/http"
require "uri"

module Seiri
  BASE = "https://ping.seiri.app/webhook"

  def self.ping!(org_ref, status: nil)
    path = [BASE, org_ref, status].compact.join("/")
    uri = URI(path)
    Net::HTTP.start(uri.host, uri.port, use_ssl: true, open_timeout: 5, read_timeout: 10) do |http|
      http.request(Net::HTTP::Get.new(uri))
    end
  end
end

Prefer a short timeout so a slow ping cannot stall the worker. Failures of the ping itself should be logged; they should not undo a successful business write — but you should still alert if pings stop arriving (Seiri handles that).

Worker that pings on success

class NightlyReportWorker
  include Sidekiq::Worker
  sidekiq_options queue: :critical, retry: 5

  SEIRI_REF = "YOUR_ORG:YOUR_REF"

  def perform
    Report.generate!          # raises on failure
    deliver_to_finance!
    Seiri.ping!(SEIRI_REF, status: "success")
  rescue StandardError
    begin
      Seiri.ping!(SEIRI_REF, status: "fail")
    rescue StandardError
      # never mask the original error
    end
    raise
  end
end

If the job never runs, neither /success nor /fail arrives. Seiri still alerts on the missed schedule — that is the point of heartbeat monitoring.

Middleware for many critical jobs

If several workers need the same pattern, Sidekiq server middleware keeps the call site thin:

# config/initializers/sidekiq.rb
class SeiriHeartbeatMiddleware
  REFS = {
    "NightlyReportWorker" => "YOUR_ORG:report-nightly",
    "InvoiceSyncWorker"   => "YOUR_ORG:invoice-sync"
  }.freeze

  def call(worker, _job, _queue)
    yield
    ref = REFS[worker.class.name]
    Seiri.ping!(ref, status: "success") if ref
  rescue StandardError
    ref = REFS[worker.class.name]
    Seiri.ping!(ref, status: "fail") if ref
    raise
  end
end

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add SeiriHeartbeatMiddleware
  end
end

Only list job classes that define a business cadence. Do not attach one monitor to every random perform_async — noisy heartbeats hide the signal.

Scheduled Sidekiq jobs (cron-like)

For sidekiq-cron:

# config/schedule.yml
nightly_report:
  cron: "0 2 * * *"
  class: "NightlyReportWorker"
  queue: critical

The schedule only enqueues. The heartbeat must live in the worker (or middleware), not in the YAML. If cron stops enqueuing, the worker never pings — Seiri alerts.

Same idea for sidekiq-scheduler / Rufus cron entries: monitor the completion of the worker, not “is the scheduler process up?” alone. If you also want a signal that the scheduler process itself is alive, run a tiny dedicated job every N minutes whose only job is to ping a separate Seiri monitor — two monitors, two failure modes.

Example: end-to-end with Seiri

1. Create monitor sidekiq-nightly-report with schedule 0 2 * * * and grace (e.g. 45 minutes if the report usually finishes in 15).
2. Put YOUR_ORG:YOUR_REF into the worker or middleware map.
3. Deploy. Trigger a manual NightlyReportWorker.perform_async in staging and confirm the monitor flips healthy.
4. Route Seiri alerts to Slack or PagerDuty — the same place you already page for production incidents.
5. Intentionally pause the cron entry in staging; confirm you get a missed-check-in alert after grace.

That drill is cheap and proves the path better than another dashboard panel.

Unique jobs, locks, and “success by skip”

Sidekiq uniqueness gems and sidekiq-unique-jobs (or custom locks) prevent duplicate work. They also create a silent-skip mode: the second enqueue is dropped, the worker never runs, and no ping fires.

That is correct if the first run still completes and pings. It is a false missed-check-in if the first run died while holding the lock and uniqueness rejects retries until TTL expires. Mitigations:

• Keep lock TTLs aligned with worst-case runtime.
• On lock conflict, log explicitly; do not treat skip as success in business metrics.
• Prefer Seiri grace wide enough that a single delayed retry still checks in; if skips can last longer than grace, fix the lock — do not widen grace forever.

class InvoiceSyncWorker
  include Sidekiq::Worker
  sidekiq_options queue: :critical, retry: 8, lock: :until_executed

  def perform(day = Date.yesterday.to_s)
    scope = Invoice.where(business_date: day, synced_at: nil)
    raise "unexpected empty invoice set for #{day}" if scope.none? && Invoice.where(business_date: day).none?

    scope.find_each { |inv| inv.sync_to_ledger! }
    Seiri.ping!("YOUR_ORG:invoice-sync", status: "success")
  end
end

Raising on impossible empty sets turns a config bug into a Sidekiq failure and a /fail or missed success ping — better than a quiet green day.

Testing the monitor without waiting for production night

1. Point staging workers at a staging Seiri monitor (separate YOUR_REF).
2. NightlyReportWorker.perform_async and confirm healthy within a minute.
3. Block egress or comment out Seiri.ping! and confirm missed-check-in after grace (shorten grace in staging).
4. Raise inside perform and confirm /fail if you use status paths.
5. Only then wire production refs via credentials (ENV), not committed secrets.

# config/initializers or credentials
ENV.fetch("SEIRI_NIGHTLY_REPORT_REF") # "YOUR_ORG:report-nightly"

What not to monitor with Seiri

• Every perform_async fan-out for thumbnail generation — use queue latency / error tracking.
• Redis CPU or Sidekiq process restarts — keep Prometheus/Datadog for that.
• HTTP uptime of your customer API — wrong tool; Seiri is not an uptime checker.

Use Seiri where silence is the bug: nightly money movement, compliance exports, data retention, partner syncs, anything you would wake up for if it skipped a day.

Best practices

1. One monitor per critical job class — not one monitor for “all of Sidekiq.” A busy image-resize queue should not mask a missed billing sync.
2. Ping after the side effect — after S3 upload, email send, or DB commit that defines “done.”
3. Match schedule + grace to reality — DST, long GC pauses, and dependency lag all need headroom.
4. Keep Redis/process monitors — heartbeats complement them; they do not replace infra checks.
5. Fail closed in application logic — raise when the outcome is incomplete; let Sidekiq retry and let Seiri see /fail or silence.
6. Air-gapped / no egress — if workers cannot HTTPS out, use email heartbeats (YOUR_REF@ping.seiri.app) via your internal MTA.
7. Kubernetes workers — if Sidekiq runs in-cluster, you can still ping HTTP from the pod; for workload health beyond jobs, see seiri-kube-agent and Kubernetes agent.
8. Store refs in ENV/credentials — treat webhook refs like secrets; rotate if leaked in logs.
9. Alert where humans already are — Slack/PagerDuty beats another unread email folder.

Conclusion

Sidekiq monitoring that only watches Redis and process liveness will miss the failures that matter: jobs that never enqueue, schedules that vanish, and “successful” workers that did no useful work.

Wire a Seiri heartbeat into the success path of each critical worker. Ask Did it run? — and get paged when the answer is silence.

Start free at cloud.seiri.app. Related reading: cron job monitoring complete guide, how to monitor Celery, background job monitoring.