Cloud Storage + Cloud Tasks for async webhook processing on Cloud Run - best practice

I've been looking around for an answer to this, but struggling to find something definitive. My apologies if I've overlooked something obvious.

I'm processing webhooks on Cloud Run (Django) that need async handling because processing takes 30+ seconds but the webhook provider times out at 30s.

Since Cloud Run is stateless and spins up per-request (no persistent background workers like Celery), I'm using this pattern:

# 1. Webhook endpoint
def receive_webhook(request):
    blob_name = f"webhooks/{uuid.uuid4()}.json"
    bucket.blob(blob_name).upload_from_string(json.dumps(request.data))
    
    webhook = WebhookPayload.objects.create(gcs_path=blob_name)
    create_cloud_task(payload_id=webhook.id)
    
    return Response(status=200)  # Fast response

And then our cloud task calls the following endpoint with the unique path to the cloud storage url passed from the original webhook endpoint:

def process_webhook(request):
    webhook = WebhookPayload.objects.get(id=request.data['payload_id'])
    payload = json.loads(bucket.blob(webhook.gcs_path).download_as_text())
    
    process_data(payload)  # 30+ seconds
    bucket.blob(webhook.gcs_path).delete()

  1. Is GCS + Cloud Tasks the right pattern for Cloud Run's stateless model, or is storing JSON directly temporarily in a django model fine since Cloud Tasks handles the queueing?

  2. Does temporary storage in GCS rather than in Postgres provide meaningful benefits?

  3. Should I be using Pub/Sub instead? Seems more for event broadcasting; I just need to invoke one endpoint.

Thanks for any advice that comes my way.

Вернуться на верх