GCP Professional Data Engineer Practice Question

Your analytics team is building a batch pipeline that: (1) triggers when 1000+ CSV files land in a Cloud Storage bucket, (2) launches a Dataflow job to cleanse and join the files, (3) waits until the job finishes, (4) runs a series of BigQuery SQL transformations, and (5) emails stakeholders if any step misses its SLA. The solution must capture task lineage, allow retry policies, enable backfills, and support future integration with on-prem Hadoop jobs. Which Cloud service should you adopt to orchestrate this pipeline?

App Engine cron jobs invoking Cloud Functions for each step
Cloud Composer (managed Apache Airflow)
Dataflow Flex templates orchestrated through Pub/Sub notifications
Workflows triggered by Cloud Scheduler

Report Issue

Answer Description

Cloud Composer is a managed Apache Airflow service. Airflow DAGs let you sequence heterogeneous tasks-including Cloud Storage sensors, Dataflow operators, and BigQuery operators-while providing retries, SLA monitoring, e-mail alerting, and historical backfill execution. Composer therefore covers every listed requirement and can later incorporate additional operators (for example, SSH or Spark on-prem) without re-architecting. Workflows excels at orchestrating short-lived API calls but lacks built-in scheduling, dependency-aware backfills, and the rich operator ecosystem needed for Dataflow or Hadoop integration. Dataflow Flex templates and Cloud Functions can run isolated jobs but do not coordinate multi-stage dependencies or provide scheduling and lineage out-of-the-box. App Engine's cron service can invoke functions on a schedule but offers no workflow graph, retries, or task-level monitoring.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.