Worker Concurrency & Recovery

Cron-driven workers (EmailProcessor, UserDeletionWorker) poll their queue tables every 30 seconds (emails) or every minute (deletions). The third cron worker, SessionCleanupWorker (src/auth/session-cleanup.worker.ts, hourly), claims nothing: it is a TTL sweep of expired sessions, tokens, magic-link attempts and aged billing audit rows, and deleting the same expired row twice is harmless. To stay safe across multiple replicas without a Redis or BullMQ dependency, every queue worker uses a per-row claim with TTL recovery pattern:

The claim is atomic. Workers issue UPDATE ... WHERE id IN (SELECT id ... FOR UPDATE SKIP LOCKED) RETURNING *, which lets exactly one worker win each row.
The claim filter is two-arm: it picks up pending (or retryable failed) rows OR rows in processing whose claimed_at is older than STALE_MS. This recovers a row whose worker died holding the claim, without a separate orphan reaper.
STALE_MS defaults to 5 minutes. Override with the WORKER_CLAIM_STALE_MS env var (milliseconds) if you need shorter / longer recovery windows.
On per-item failure, workers mark the row status='failed', drop the claim, record last_error, and set next_retry_at to an exponential backoff (30s, 2min, 8min, 32min). Both queues cap retries: email_jobs on its per-row max_attempts, the deletion queue on USER_DELETION_MAX_ATTEMPTS (5, in users.service.ts). The two caps are enforced differently, and the difference matters if you copy one: email_jobs counts in the catch and gates only the retry arm, while the deletion queue counts on the claim and gates both arms. Counting on the claim is what makes the cap hold for an item that kills the process before any catch runs; gating both arms is what stops the stale-claim recovery arm from re-picking it forever.

A deletion that exhausts its cap is abandoned, and what that leaves behind is not always the same half of the job. admin_get_abandoned_deletions on the admin MCP server lists those rows; Billing has the triage and the requeue SQL. The template ships no alerting, so wire the worker's give-up log line into whatever you use.

This rules out double-sends across replicas during normal operation, and bounds the worst-case "stuck claim" window to STALE_MS. It does not eliminate an at-least-once double-process if a worker dies and its claimed row's TTL expires before the work actually completed at SMTP. At-least-once is the explicit contract here.

The claim is atomic. Workers issue UPDATE ... WHERE id IN (SELECT id ... FOR UPDATE SKIP LOCKED) RETURNING *, which lets exactly one worker win each row.

The claim filter is two-arm: it picks up pending (or retryable failed) rows OR rows in processing whose claimed_at is older than STALE_MS. This recovers a row whose worker died holding the claim, without a separate orphan reaper.

STALE_MS defaults to 5 minutes. Override with the WORKER_CLAIM_STALE_MS env var (milliseconds) if you need shorter / longer recovery windows.

On per-item failure, workers mark the row status='failed', drop the claim, record last_error, and set next_retry_at to an exponential backoff (30s, 2min, 8min, 32min). Both queues cap retries: email_jobs on its per-row max_attempts, the deletion queue on USER_DELETION_MAX_ATTEMPTS (5, in users.service.ts). The two caps are enforced differently, and the difference matters if you copy one: email_jobs counts in the catch and gates only the retry arm, while the deletion queue counts on the claim and gates both arms. Counting on the claim is what makes the cap hold for an item that kills the process before any catch runs; gating both arms is what stops the stale-claim recovery arm from re-picking it forever.