--- term: 'Background Jobs & Task Schedulers' seoTitle: 'Background Jobs & Task Schedulers: Queues, Workers, Retries' headline: 'What are Background Jobs & Task Schedulers?' slug: background-jobs-task-schedulers category: backend-compute shortDefinition: 'A background job is a task that runs outside the request cycle — enqueued by the app, executed by workers, retried on failure.' relatedTerms: - scheduled-cloud-code-cron-jobs - cloud-code-serverless-functions - pub-sub-pattern - webhooks contrastsWith: - scheduled-cloud-code-cron-jobs aboutTerms: - 'Job Queue' - 'Worker' - 'Dead-Letter Queue' - 'Cron' faq: - question: 'What is a background job?' answer: 'A task your application runs outside the request-response cycle: the server enqueues the work, answers the user immediately, and a separate worker executes it asynchronously — with retries if it fails. Anything slow, retryable, or unneeded for the response belongs there.' - question: 'Why not just do the work in the request handler?' answer: 'Because slow work blocks the response, ties up server capacity, and hits gateway timeouts — and if the work errors mid-request, the user gets a failure even though part of it happened. The canonical example: sending the welcome email during signup, where a slow mail server turns account creation into a spinning wheel.' - question: 'How does a job queue work?' answer: 'It is a durable list between producers and workers: the app pushes a job — a type plus a small payload — the queue persists it, and workers pull, execute, and acknowledge. Unacknowledged jobs return to the queue, which is where the reliability comes from.' - question: 'What is the difference between a job queue and a message queue?' answer: 'Purpose. A message queue moves data between services — delivery is the goal, and a message may fan out to many consumers. A job queue executes work — it adds retries, scheduling, priorities, and status on top, and each job goes to exactly one worker.' - question: 'What is the difference between cron jobs and background jobs?' answer: 'The trigger. Cron is time-driven — "every night at 2 a.m."; background jobs are event-driven — "when a user uploads a file." The patterns compose: a common design has cron enqueue the nightly batch while workers execute it with full retry semantics.' - question: 'How do job retries work?' answer: 'Failed jobs re-run automatically with exponential backoff — one second, then two, four, eight — plus random jitter so a thousand failures don''t all retry in the same instant, capped at a maximum attempt count. Backoff bridges transient failures; the cap keeps permanent ones from retrying forever.' - question: 'Why must background jobs be idempotent?' answer: 'Because queues promise at-least-once execution: a worker can crash after doing the work but before acknowledging it, and the job runs again. Running twice must not charge twice — idempotency keys, unique constraints, and upserts turn duplicates into no-ops.' - question: 'What is a dead-letter queue?' answer: 'Where jobs go after exhausting their retries — held for inspection, alerting, and manual replay instead of retrying forever or vanishing silently. Malformed jobs that can never succeed should skip the retries and go straight there.' - question: 'What are delayed and scheduled jobs?' answer: 'The same queue, shifted in time: a delayed job is enqueued now but eligible later ("send the reminder in 24 hours"); a recurring job re-enqueues on a cron-style schedule. Delay is also the mechanism retries use — a failed job is just a job delayed by its backoff.' - question: 'How do you monitor background jobs?' answer: 'Failures are silent by default — nothing renders an error page. Watch queue depth (is work accumulating?), job age from enqueue to completion (not just processing time), failure rate, dead-letter depth, and missed schedule runs — the alarm nothing else raises, because nothing ran.' codeLanguages: [javascript, dart, swift, kotlin] externalAuthorities: - name: 'Sidekiq Best Practices' url: 'https://github.com/sidekiq/sidekiq/wiki/Best-Practices' - name: 'Celery — Distributed Task Queue' url: 'https://docs.celeryq.dev/en/stable/getting-started/introduction.html' - name: 'crontab(5) — Linux manual page' url: 'https://man7.org/linux/man-pages/man5/crontab.5.html' - name: 'Scaling Slack''s Job Queue — Slack Engineering' url: 'https://slack.engineering/scaling-slacks-job-queue/' - name: 'Job scheduler — Wikipedia' url: 'https://en.wikipedia.org/wiki/Job_scheduler' cta: title: 'Jobs without the queue to run' text: 'Define a Cloud Job on Back4app and run it on demand or on a schedule from the dashboard — status, logs, and recurrence included, with no broker, worker fleet, or queue infrastructure to operate.' linkText: 'Start building free' linkUrl: 'https://www.back4app.com/signup' author: 'Back4app Engineering' publishedDate: '2026-07-30' translationKey: background-jobs-task-schedulers --- **A background job is a task that runs outside the request cycle — enqueued by the app, executed by workers, retried on failure.** The dividing line is time: a response should feel instant (a few hundred milliseconds) and *must* beat the gateway timeout (typically ~30 seconds), while real work — emails over SMTP, image processing, report generation, third-party APIs — takes seconds to minutes and deserves retries. Everything on the wrong side of that line gets enqueued, acknowledged, and done elsewhere; at scale this is core infrastructure, not plumbing — one large messaging company processes over a billion jobs a day through exactly this machinery. ## Key takeaways | Question | Answer | | --- | --- | | The architecture | Producer enqueues → queue persists → worker executes and acks | | The dividing line | Response budget ~300 ms; anything slow or retryable becomes a job | | The three schedules | Immediate · delayed ("in 24 h") · recurring (cron) | | The reliability contract | At-least-once + idempotent jobs + backoff with jitter + dead letters | | The silent failure | Nothing errors when a scheduled job doesn't run — monitor for absence | ## The anti-pattern that teaches everything ```js // ✗ The signup that hostages itself to a mail server app.post('/signup', async (req, res) => { const user = await createUser(req.body); await sendWelcomeEmail(user); // SMTP slow? The user stares at a spinner. res.json(user); // SMTP down? Signup 500s — but the }); // account EXISTS. Worst of all worlds. // ✔ Enqueue and answer app.post('/signup', async (req, res) => { const user = await createUser(req.body); await jobs.enqueue('welcomeEmail', { userId: user.id }); // milliseconds res.json(user); // email sends, retries, and succeeds }); // on its own schedule — invisibly ``` The same principle from both sides of a real backend — a job over the data, and a client that enqueues instead of waiting: **JavaScript:** ```javascript // JavaScript — Cloud Code (cloud/main.js) // A background job: heavy work outside the request cycle Parse.Cloud.job('sendWeeklyDigest', async (request) => { const query = new Parse.Query(Parse.User).equalTo('digestOptIn', true); await query.eachBatch(async (users) => { for (const user of users) await sendDigest(user); // per-record, resumable }, { useMasterKey: true }); request.message('Digest run complete'); // status visible in the dashboard }); // Run on demand or on a schedule from the dashboard — no queue to operate ``` **Flutter:** ```dart // Flutter / Dart — Back4app Flutter SDK // The client's rule: never wait on heavy work — enqueue and move on final export = ParseObject('ExportRequest') ..set('user', currentUser) ..set('status', 'queued'); // an afterSave trigger starts the work await export.save(); // returns in milliseconds // Watch the job's progress like any other data: final sub = await LiveQuery().client.subscribe( QueryBuilder(ParseObject('ExportRequest')) ..whereEqualTo('objectId', export.objectId)); sub.on(LiveQueryEvent.update, (job) { if (job.get('status') == 'done') openReport(job.get('fileUrl')); }); ``` **Swift:** ```swift // iOS / Swift — Back4app Swift SDK // The client's rule: never wait on heavy work — enqueue and move on var export = ExportRequest() export.status = "queued" // an afterSave trigger starts the work let saved = try await export.save() // returns in milliseconds // Watch the job's progress like any other data: let sub = try await ExportRequest.query("objectId" == saved.id).subscribe() sub.handleEvent { _, event in if case .updated(let job) = event, job.status == "done" { openReport(job.fileUrl) } } ``` **Kotlin:** ```kotlin // Android / Kotlin — Back4app Android SDK // The client's rule: never wait on heavy work — enqueue and move on val export = ParseObject("ExportRequest") export.put("user", ParseUser.getCurrentUser()) export.put("status", "queued") // an afterSave trigger starts the work export.save() // returns in milliseconds // Watch the job's progress like any other data: val q = ParseQuery.getQuery("ExportRequest") q.whereEqualTo("objectId", export.objectId) val sub = ParseLiveQueryClient.Factory.getClient().subscribe(q) sub.handleEvent(SubscriptionHandling.Event.UPDATE) { _, job -> if (job.getString("status") == "done") openReport(job.getString("fileUrl")) } ``` ## Producer, queue, worker ```mermaid flowchart LR accTitle: Producer, queue, and worker architecture with retries and dead letters accDescr: The application enqueues jobs into a durable queue and responds to users immediately. Workers pull jobs, execute them, and acknowledge on success. Failed jobs are re-enqueued with exponential backoff, and jobs that exhaust their retries move to a dead-letter queue for inspection and replay. A["App (producer)
enqueue + respond fast"] --> Q[("Queue
durable, ordered-ish")] S["Scheduler
cron: re-enqueue on time"] --> Q Q --> W["Workers
pull · execute · ack"] W -->|"success"| OK["Done"] W -.->|"failure"| B["Backoff + jitter
delay, then re-enqueue"] -.-> Q B -.->|"attempts exhausted"| DL["Dead-letter queue
inspect · alert · replay"] ``` Three roles, one contract. The **producer** — usually a request handler — creates a job: a type name and a small payload. The **queue** persists it; durability is the point, since work that exists only in a dying process's memory dies with it. **Workers** pull, execute, and *acknowledge*; a job is only gone once acked, which is how a crashed worker's job survives to run again. The vocabulary that rides along: enqueue/dequeue, acks, visibility timeouts, and the distinction worth policing — a *message* queue moves data between services (and may fan out to [many subscribers](/glossary/pub-sub-pattern/)); a *job* queue executes work, exactly once per job per worker, with retries and status built in. ## The three schedules — and a cron primer Every job system implements the same three time modes: **immediate** (enqueue now, run as soon as a worker frees), **delayed** (enqueue now, eligible at a future moment — reminders, trial expirations, and the retry mechanism itself), and **recurring** (a scheduler re-enqueues on a calendar). Recurring schedules are almost always written in [cron's](https://man7.org/linux/man-pages/man5/crontab.5.html) five fields: ```text ┌ minute (0-59) ┌ hour (0-23) ┌ day of month ┌ month ┌ day of week 0 2 * * * → every day at 02:00 */15 * * * * → every 15 minutes 0 9 * * 1 → Mondays at 09:00 Two scheduler landmines: DST transitions (02:30 vanishes or happens twice — schedule in UTC), and overlap (a run outlasting its interval needs a lock, or two copies process the same data). ``` ## Job queues vs. message queues vs. schedulers | | Job queue | Message queue | Scheduler | | --- | --- | --- | --- | | Unit | A job to execute | A message to deliver | A time to fire | | Consumers | Exactly one worker | One or [many subscribers](/glossary/pub-sub-pattern/) | The jobs it enqueues | | Built-ins | Retries, status, priorities, delay | Durability, routing, fan-out | Calendars, recurrence | | Open-source names | Sidekiq, Celery, BullMQ | RabbitMQ, Kafka, Redis | cron, Quartz | | Composes as | Executes what events demand | Transports between systems | Feeds the queue on time | The composition is the answer to most "which one?" debates: schedulers decide *when*, queues hold *what*, workers do the *doing* — and a nightly batch is cron enqueueing jobs that workers execute with full retry semantics. ## The reliability contract The pieces are usually taught separately; they are one contract. **The queue promises at-least-once** — a worker that crashes after the work but before the ack means the job runs again; this is unavoidable, not sloppy. **You promise idempotency** in return: dedupe on a job ID, guard with unique constraints, write with upserts, so the second run is a no-op instead of a second invoice. **Retries with exponential backoff and jitter** bridge transient failures — 1 s, 2 s, 4 s, with randomness so a thousand jobs failing together don't retry together, a courtesy [rate-limited third parties](/glossary/api-rate-limiting-throttling/) will enforce if you don't extend it. **The dead-letter queue catches the rest**: after the attempt cap, jobs park where humans can inspect, fix, and replay them — and permanently malformed jobs should skip the retry theater entirely and go straight there. ## Designing jobs well The rules the queue frameworks agree on, assembled: **pass IDs, not objects** — a job payload holding `userId: "u-8fk2"` refetches fresh state at run time, while a serialized user object is stale the moment it's enqueued; **keep payloads small** and JSON-simple, never secrets; **one job per record** — a per-user job that fails retries one user, a mega-job retries everyone; **make long work resumable** — chunk it, checkpoint progress, and exit gracefully on shutdown signals mid-chunk; and **assume concurrency** — two workers will someday process adjacent jobs touching the same row, which is a locking question you answer at design time or in an incident. ## Watching the queue Background failure is silent — no user sees an error page when the digest job dies. The four gauges that replace the error page: **queue depth** (accumulating backlog means workers are losing), **job age** measured enqueue-to-completion — a job processed in 200 ms after waiting 40 minutes is a 40-minute job to the user; **failure and dead-letter rates** with alerts on the dead-letter queue, because parked jobs someone forgot are data quietly not processed; and **missed runs** for scheduled work — the uniquely sneaky one, since a schedule that never fired produced no error, no log line, and no job at all. Alarm on the absence. ## Common use cases - **Email and notifications** — the canonical deferral: enqueue at signup, deliver with retries. - **Media processing** — resizes, transcodes, thumbnails: minutes of CPU no request should wait on. - **Reports and exports** — generate in the background, notify when the file is ready. - **Third-party synchronization** — CRM syncs and [webhook](/glossary/webhooks/) deliveries, with backoff against flaky remotes. - **Cleanup and maintenance** — TTL sweeps, orphan pruning, digest builds on the nightly cron. ## Which pattern do you need? A decision matrix | Work looks like | Reach for | | --- | --- | | Triggered by user actions, variable volume | Job queue + workers | | Fixed calendar, bounded, predictable | Scheduled job (cron) | | Nightly batch over many records | Cron enqueues; workers execute per-record | | Services telling each other things | [Message queue / pub-sub](/glossary/pub-sub-pattern/) | | Slow work inside a request handler today | The refactor above — enqueue and answer | | Must survive crashes and retry safely | Any of these — plus the reliability contract | ## Limitations and trade-offs - **Eventual, not instant.** Enqueued work happens *later* — UIs need pending states, and "later" needs a bound someone chose on purpose. - **State moves out of band.** Results arrive via status fields, callbacks, or notifications — the request-response simplicity is spent, budget the plumbing. - **Infrastructure is real.** Brokers, workers, and schedulers are services with their own failure modes — or a platform's problem, which is the managed-jobs argument. - **Duplicates are guaranteed eventually.** At-least-once is the contract; every non-idempotent job is an incident on a timer. - **Queues hide overload gracefully — too gracefully.** A growing backlog looks calm right up until job age explodes; depth and age alarms are the honesty mechanism. ## Background jobs on Back4app Back4app is an open-source Backend-as-a-Service (BaaS) platform that combines a managed database, auto-generated REST and GraphQL APIs, authentication, file storage, and Cloud Code serverless functions. Jobs here are [Cloud Code](/glossary/cloud-code-serverless-functions/) with a schedule: define `Parse.Cloud.job` — the code tabs show a digest job batching over users with `eachBatch`, per-record and resumable — and run it on demand or on a cron-style recurrence from the dashboard, with status and logs in the Jobs panel; no broker to provision, no worker fleet to scale, no queue service to keep alive. The event-driven half composes from the same primitives: a request handler writes a row with `status: "queued"`, an `afterSave` trigger starts the work, and clients watch progress over [Live Queries](/glossary/real-time-live-queries/) instead of polling — the producer/worker pattern expressed as data, with the reliability contract (IDs in payloads, idempotent handlers, status you can alarm on) as your design discipline rather than your infrastructure.