GCP Professional Data Engineer Practice Question

Your analytics team must orchestrate a daily data pipeline that: triggers a Cloud Storage Transfer job, runs custom Python data-quality scripts on Cloud Run, loads cleansed data into BigQuery, and finally calls a Vertex AI prediction endpoint. The workflow needs conditional branching, cross-task retries, SSH connections to an on-premises host, and a graphical DAG that operators can monitor. To satisfy these requirements while avoiding heavy infrastructure management and allowing reuse of existing Airflow DAGs, which Google Cloud service should you use?

Workflows
Cloud Scheduler triggers invoking Pub/Sub topics and Cloud Functions
Cloud Composer (managed Apache Airflow)
Dataflow Flex Templates with pipeline options

Report Issue

Answer Description

Cloud Composer is Google Cloud's managed Apache Airflow service. Airflow expresses complex, multi-step workflows as DAGs that support conditional branches, retries, and dependencies. Composer lets you install custom Python packages and Airflow plugins, so you can invoke Cloud Run jobs, BigQuery operators, Storage Transfer Service hooks, SSH operators for on-prem hosts, and custom calls to Vertex AI. It also provides the standard Airflow web UI for visual monitoring, while Google manages the underlying infrastructure.

Workflows can orchestrate Google Cloud APIs but offers limited first-class support for SSH connections, custom Python libraries, or direct reuse of existing Airflow DAGs; authoring complex branching pipelines is possible but less ergonomic. Cloud Scheduler with Pub/Sub and Cloud Functions would require substantial custom code to replicate DAG-level dependency management and monitoring. Dataflow Flex Templates address data processing, not cross-service orchestration. Therefore, Cloud Composer best meets all stated requirements.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.