Rule

Every background job must produce the same outcome when replayed with the same input.

Why

Queue processors guarantee at-least-once delivery. Without idempotency, receiver-side retries corrupt data.

Must

• Assign a stable job ID to every enqueued task.
• Check whether the job has already succeeded before executing its side effects.
• Record job completion atomically with its side effects where possible (database transaction).
• Mark jobs as failed with reason after max retry attempts.

Should

• Use a dead-letter queue for jobs that exhaust retries.
• Alert on dead-letter queue depth.

Anti-patterns

• Job that sends an email without checking if it was already sent.
• Unbounded retries with no backoff.

Test Cases

• Replaying a completed job produces no additional side effects.
• Job exhausting retries moves to dead-letter queue.

Telemetry

• job_started
• job_completed
• job_skipped_already_processed
• job_dead_lettered