GCP Professional Data Engineer Practice Question

Your analytics platform runs several regional Dataflow streaming jobs in three separate Google Cloud projects. The governance team needs a single orchestration layer that will (1) kick off a nightly back-fill DAG with six dependent steps, (2) support dynamic branching and automatic retries, (3) invoke REST endpoints hosted in another cloud, and (4) raise SLA-based e-mail alerts that appear on Cloud Monitoring dashboards-all without requiring you to patch or scale servers. What should you implement to satisfy these requirements with minimal operational overhead?

Set up Cloud Scheduler to trigger Cloud Functions that call Dataflow REST APIs, implement branching logic in code, and use custom logging for monitoring.
Deploy a dedicated Google Kubernetes Engine cluster and use Kubernetes CronJobs with custom controllers to orchestrate Dataflow templates and external HTTP calls while exporting Prometheus metrics to Cloud Monitoring.
Define a Google Cloud Workflows specification that sequentially invokes Dataflow templates and external APIs, schedule it with Cloud Scheduler, and add scripts to push status logs into Cloud Monitoring for alerting.
Provision a Cloud Composer environment and author an Apache Airflow DAG that launches the Dataflow jobs, performs external REST calls, configures retries and SLA alerts, and relies on Composer's native Cloud Monitoring integration.

Report Issue

Answer Description

A managed Cloud Composer environment meets all stated requirements because it provides Apache Airflow as a service. Airflow's DAG model handles complex multi-step dependencies, dynamic branching, and task-level retries out of the box. Airflow operators and hooks can invoke Google Cloud services as well as arbitrary HTTP endpoints in other clouds. Composer automatically integrates Airflow metrics and logs with Cloud Monitoring and Cloud Logging, enabling SLA alerts that can trigger e-mail or pager notifications. Google manages the underlying GKE cluster, so you are not responsible for patching or scaling infrastructure.

Using Cloud Scheduler and Cloud Functions or Workflows would require custom code for dependency management, retries, and monitoring integration, increasing operational effort. Running Kubernetes CronJobs on a dedicated GKE cluster shifts full cluster management and monitoring to your team, which contradicts the requirement to avoid server maintenance.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.