GCP Professional Data Engineer Practice Question

Your team built a DAG in Cloud Composer that, on every Pub/Sub message, launches a Dataflow Flex Template, updates a Firestore document, and then calls a partner HTTP API. Even with aggressive DAG scheduling the end-to-end orchestration latency is still several minutes, which violates an SLA requiring a response within 2 seconds from message arrival. You want to redesign only the orchestration layer while keeping the existing services unchanged. What should you do?

Keep Cloud Composer but disable DAG catchup and schedule the DAG every 30 seconds to reduce delay.
Move the orchestration logic into Cloud Tasks, placing one task per Pub/Sub message and executing it with a pull worker in Cloud Functions.
Convert the DAG into a Cloud Run job and invoke it every minute with Cloud Scheduler to orchestrate the steps.
Replace Cloud Composer with a Workflows definition triggered directly by Pub/Sub that sequentially invokes the Dataflow template, updates Firestore, and calls the partner HTTP API.

Report Issue

Answer Description

Workflows is a fully managed, serverless orchestrator that can be invoked directly from Pub/Sub (for example, through Eventarc or an HTTPS trigger) and can call Google Cloud APIs, Cloud Functions or Cloud Run services, and external HTTP endpoints in sequence. Because it does not rely on a periodic scheduler and has no worker pods to spin up, execution starts almost immediately after the triggering event, making sub-second latencies realistic. Cloud Composer relies on Airflow's scheduler, which polls at fixed intervals and introduces startup overhead, so even an aggressively tuned DAG seldom meets a 2-second SLA. Cloud Run or Cloud Functions could hold custom orchestration code, but you would need to implement retries, branching, and state management yourself, recreating capabilities already available in Workflows. Cloud Tasks is intended for individual, independent task dispatch rather than multi-step, stateful workflows.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.