What are Background Jobs & Task Schedulers?

A background job is a task that runs outside the request cycle — enqueued by the app, executed by workers, retried on failure. The dividing line is time: a response should feel instant (a few hundred milliseconds) and must beat the gateway timeout (typically ~30 seconds), while real work — emails over SMTP, image processing, report generation, third-party APIs — takes seconds to minutes and deserves retries. Everything on the wrong side of that line gets enqueued, acknowledged, and done elsewhere; at scale this is core infrastructure, not plumbing — one large messaging company processes over a billion jobs a day through exactly this machinery.

Key takeaways

The anti-pattern that teaches everything

// ✗ The signup that hostages itself to a mail server
app.post('/signup', async (req, res) => {
  const user = await createUser(req.body);
  await sendWelcomeEmail(user);        // SMTP slow? The user stares at a spinner.
  res.json(user);                      // SMTP down? Signup 500s — but the
});                                    // account EXISTS. Worst of all worlds.

// ✔ Enqueue and answer
app.post('/signup', async (req, res) => {
  const user = await createUser(req.body);
  await jobs.enqueue('welcomeEmail', { userId: user.id }); // milliseconds
  res.json(user);                      // email sends, retries, and succeeds
});                                    // on its own schedule — invisibly

The same principle from both sides of a real backend — a job over the data, and a client that enqueues instead of waiting:

// JavaScript — Cloud Code (cloud/main.js)
// A background job: heavy work outside the request cycle
Parse.Cloud.job('sendWeeklyDigest', async (request) => {
  const query = new Parse.Query(Parse.User).equalTo('digestOptIn', true);
  await query.eachBatch(async (users) => {
    for (const user of users) await sendDigest(user); // per-record, resumable
  }, { useMasterKey: true });
  request.message('Digest run complete'); // status visible in the dashboard
});
// Run on demand or on a schedule from the dashboard — no queue to operate

// Flutter / Dart — Back4app Flutter SDK
// The client's rule: never wait on heavy work — enqueue and move on
final export = ParseObject('ExportRequest')
  ..set('user', currentUser)
  ..set('status', 'queued'); // an afterSave trigger starts the work
await export.save(); // returns in milliseconds

// Watch the job's progress like any other data:
final sub = await LiveQuery().client.subscribe(
    QueryBuilder<ParseObject>(ParseObject('ExportRequest'))
      ..whereEqualTo('objectId', export.objectId));
sub.on(LiveQueryEvent.update, (job) {
  if (job.get<String>('status') == 'done') openReport(job.get('fileUrl'));
});

// iOS / Swift — Back4app Swift SDK
// The client's rule: never wait on heavy work — enqueue and move on
var export = ExportRequest()
export.status = "queued" // an afterSave trigger starts the work
let saved = try await export.save() // returns in milliseconds

// Watch the job's progress like any other data:
let sub = try await ExportRequest.query("objectId" == saved.id).subscribe()
sub.handleEvent { _, event in
    if case .updated(let job) = event, job.status == "done" {
        openReport(job.fileUrl)
    }
}

// Android / Kotlin — Back4app Android SDK
// The client's rule: never wait on heavy work — enqueue and move on
val export = ParseObject("ExportRequest")
export.put("user", ParseUser.getCurrentUser())
export.put("status", "queued") // an afterSave trigger starts the work
export.save() // returns in milliseconds

// Watch the job's progress like any other data:
val q = ParseQuery.getQuery<ParseObject>("ExportRequest")
q.whereEqualTo("objectId", export.objectId)
val sub = ParseLiveQueryClient.Factory.getClient().subscribe(q)
sub.handleEvent(SubscriptionHandling.Event.UPDATE) { _, job ->
    if (job.getString("status") == "done") openReport(job.getString("fileUrl"))
}

Producer, queue, worker

Three roles, one contract. The producer — usually a request handler — creates a job: a type name and a small payload. The queue persists it; durability is the point, since work that exists only in a dying process’s memory dies with it. Workers pull, execute, and acknowledge; a job is only gone once acked, which is how a crashed worker’s job survives to run again. The vocabulary that rides along: enqueue/dequeue, acks, visibility timeouts, and the distinction worth policing — a message queue moves data between services (and may fan out to many subscribers); a job queue executes work, exactly once per job per worker, with retries and status built in.

The three schedules — and a cron primer

Every job system implements the same three time modes: immediate (enqueue now, run as soon as a worker frees), delayed (enqueue now, eligible at a future moment — reminders, trial expirations, and the retry mechanism itself), and recurring (a scheduler re-enqueues on a calendar). Recurring schedules are almost always written in cron’s five fields:

┌ minute (0-59)  ┌ hour (0-23)  ┌ day of month  ┌ month  ┌ day of week
0 2 * * *      → every day at 02:00
*/15 * * * *   → every 15 minutes
0 9 * * 1     → Mondays at 09:00

Two scheduler landmines: DST transitions (02:30 vanishes or happens twice —
schedule in UTC), and overlap (a run outlasting its interval needs a lock,
or two copies process the same data).

Job queues vs. message queues vs. schedulers

The composition is the answer to most “which one?” debates: schedulers decide when, queues hold what, workers do the doing — and a nightly batch is cron enqueueing jobs that workers execute with full retry semantics.

The reliability contract

The pieces are usually taught separately; they are one contract. The queue promises at-least-once — a worker that crashes after the work but before the ack means the job runs again; this is unavoidable, not sloppy. You promise idempotency in return: dedupe on a job ID, guard with unique constraints, write with upserts, so the second run is a no-op instead of a second invoice. Retries with exponential backoff and jitter bridge transient failures — 1 s, 2 s, 4 s, with randomness so a thousand jobs failing together don’t retry together, a courtesy rate-limited third parties will enforce if you don’t extend it. The dead-letter queue catches the rest: after the attempt cap, jobs park where humans can inspect, fix, and replay them — and permanently malformed jobs should skip the retry theater entirely and go straight there.

Designing jobs well

The rules the queue frameworks agree on, assembled: pass IDs, not objects — a job payload holding userId: "u-8fk2" refetches fresh state at run time, while a serialized user object is stale the moment it’s enqueued; keep payloads small and JSON-simple, never secrets; one job per record — a per-user job that fails retries one user, a mega-job retries everyone; make long work resumable — chunk it, checkpoint progress, and exit gracefully on shutdown signals mid-chunk; and assume concurrency — two workers will someday process adjacent jobs touching the same row, which is a locking question you answer at design time or in an incident.

Watching the queue

Background failure is silent — no user sees an error page when the digest job dies. The four gauges that replace the error page: queue depth (accumulating backlog means workers are losing), job age measured enqueue-to-completion — a job processed in 200 ms after waiting 40 minutes is a 40-minute job to the user; failure and dead-letter rates with alerts on the dead-letter queue, because parked jobs someone forgot are data quietly not processed; and missed runs for scheduled work — the uniquely sneaky one, since a schedule that never fired produced no error, no log line, and no job at all. Alarm on the absence.

Common use cases

• Email and notifications — the canonical deferral: enqueue at signup, deliver with retries.
• Media processing — resizes, transcodes, thumbnails: minutes of CPU no request should wait on.
• Reports and exports — generate in the background, notify when the file is ready.
• Third-party synchronization — CRM syncs and webhook deliveries, with backoff against flaky remotes.
• Cleanup and maintenance — TTL sweeps, orphan pruning, digest builds on the nightly cron.

Which pattern do you need? A decision matrix

Limitations and trade-offs

• Eventual, not instant. Enqueued work happens later — UIs need pending states, and “later” needs a bound someone chose on purpose.
• State moves out of band. Results arrive via status fields, callbacks, or notifications — the request-response simplicity is spent, budget the plumbing.
• Infrastructure is real. Brokers, workers, and schedulers are services with their own failure modes — or a platform’s problem, which is the managed-jobs argument.
• Duplicates are guaranteed eventually. At-least-once is the contract; every non-idempotent job is an incident on a timer.
• Queues hide overload gracefully — too gracefully. A growing backlog looks calm right up until job age explodes; depth and age alarms are the honesty mechanism.

Background jobs on Back4app

Back4app is an open-source Backend-as-a-Service (BaaS) platform that combines a managed database, auto-generated REST and GraphQL APIs, authentication, file storage, and Cloud Code serverless functions. Jobs here are Cloud Code with a schedule: define Parse.Cloud.job — the code tabs show a digest job batching over users with eachBatch, per-record and resumable — and run it on demand or on a cron-style recurrence from the dashboard, with status and logs in the Jobs panel; no broker to provision, no worker fleet to scale, no queue service to keep alive. The event-driven half composes from the same primitives: a request handler writes a row with status: "queued", an afterSave trigger starts the work, and clients watch progress over Live Queries instead of polling — the producer/worker pattern expressed as data, with the reliability contract (IDs in payloads, idempotent handlers, status you can alarm on) as your design discipline rather than your infrastructure.