Skip to main content

Background Jobs Infrastructure

Two distinct GCP mechanisms back the platform's async work, serving different shapes of problem — don't reach for one where the other fits better.

  • Cloud Tasks — for work that's fundamentally a queue of individual, retryable units with a known target (batch vendor-metrics updates, scheduled reminders, the mock webhook simulation covered in Testing Bank Transfers Locally). Each task is an HTTP call to one of this app's own internal/cron endpoints.
  • Pub/Sub — for event-driven fan-out where one event needs to reach multiple independent consumers (RSVP notifications, broadcast delivery, points issuance covered in Points).

Pub/Sub setup​

yarn setup:pubsub   # creates all topics, subscriptions, and dead-letter queues
yarn pubsub:list     # lists what currently exists in the configured GCP project

Both run scripts/setup-pubsub.ts against whatever GOOGLE_CLOUD_PROJECT/GOOGLE_APPLICATION_CREDENTIALS resolve to — point them at the right project before running either. The setup script is idempotent; re-running it against existing topics doesn't modify them.

Topics generally follow a {domain}.{action}.{variant} or {domain}-{action} naming convention with a matching -dlq (dead-letter queue) sibling — e.g. RSVP notification topics fan out into immediate vs. batched variants per channel (email/push/SMS), each with retry limits before falling to its DLQ. Check src/shared/config/pubsub-topics.config.ts for the authoritative current list rather than assuming any one domain guide's topic names are exhaustive or unchanged.

Verifying it's actually running​

yarn pubsub:list

Then start the app and watch for per-topic worker startup logs — the exact count and phrasing may have drifted, but a healthy boot logs one success line per registered worker. A missing worker for a topic that's supposed to be active is the first thing to check when messages seem to vanish rather than process.

Cloud Tasks setup (example: vendor metrics batch job)​

gcloud services enable cloudtasks.googleapis.com

gcloud tasks queues create vendor-metrics-queue \
  --location=<region> \
  --max-dispatches-per-second=10 \
  --max-concurrent-dispatches=50 \
  --max-attempts=5 \
  --max-retry-duration=3600s

The service account the app runs as needs roles/cloudtasks.enqueuer and roles/cloudtasks.taskRunner on the project. Required env vars: GOOGLE_CLOUD_PROJECT, GOOGLE_CLOUD_LOCATION, SERVICE_URL (must be the instance's actual reachable address — Cloud Tasks calls back into it), plus an optional CLOUD_TASKS_SECRET_TOKEN as a second check layered on top of Cloud Tasks' own request headers.

Monitoring & emergency stop​

gcloud tasks queues describe vendor-metrics-queue --location=<region>
gcloud tasks list --queue=vendor-metrics-queue --location=<region>

# Emergency: drain a runaway or misbehaving queue
gcloud tasks queues purge vendor-metrics-queue --location=<region>

Key business rules & edge cases​

  • A SERVICE_URL that doesn't match reality is the single most common cause of "background jobs silently don't run" — both Cloud Tasks and Cloud Scheduler need to reach back into this exact address; a stale value left over from an ngrok tunnel or an old hostname fails quietly rather than with an obvious error at startup.
  • Purging a Cloud Tasks queue is immediate and un-reversible for in-flight tasks — treat it as a genuine emergency-stop, not a routine debugging step.
  • Pub/Sub dead-letter queues exist specifically so a permanently-failing message doesn't retry forever — a growing DLQ backlog is a signal worth alerting on, not something to silently accumulate.
  • scripts/setup-pubsub.ts
  • src/shared/config/pubsub-topics.config.ts
  • src/modules/vendorMetrics/ — the Cloud Tasks batch-processing example above (see Vendors)