Your data engineering team maintains a daily batch workflow that 1) executes a Dataflow job, 2) conditionally calls an external REST API, 3) waits for a BigQuery transformation to finish, and 4) sends a status email on completion. The workflow must offer dependency management, configurable retries and back-off for each step, a graphical DAG view for operators, and access to a large ecosystem of community-maintained connectors. Operations also requires centralized Cloud Monitoring and Logging without having to manage Airflow themselves. Which Google Cloud service should you recommend to orchestrate this workflow?
Workflows, expressed in YAML, invoking Cloud Functions and Cloud Run for each step.
Cloud Composer, the managed Apache Airflow service, defining the process as a DAG with task-level retries and native monitoring.
A single Dataflow Flex Template that embeds the orchestration logic inside the pipeline itself.
Cloud Scheduler jobs that publish Pub/Sub messages to trigger individual Cloud Functions sequentially.
The described requirements map directly to Cloud Composer, Google Cloud's fully managed Apache Airflow service. Composer lets you define complex, dependency-aware DAGs in Python, supports per-task retries and back-off policies, exposes a rich web UI that visualizes DAG execution, and inherits Airflow's extensive library of community plugins and operators (e.g., for HTTP calls, BigQuery, email). Because Composer is managed by Google Cloud, the platform handles Airflow upgrades and infrastructure while automatically integrating with Cloud Logging and Cloud Monitoring for centralized observability.
Workflows can orchestrate Google Cloud APIs but lacks Airflow's DAG UI and broad plugin ecosystem. Cloud Scheduler triggers jobs on a schedule but offers no built-in task dependencies or DAG management. Embedding orchestration in a single Dataflow pipeline limits flexibility and forfeits the cross-service coordination, UI, and retry semantics required. Therefore, Cloud Composer is the most suitable choice.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is a DAG in Cloud Composer?
Open an interactive chat with Bash
What are the benefits of using Cloud Composer over Workflows?
Open an interactive chat with Bash
What is the role of centralized Cloud Monitoring and Logging in Cloud Composer?
Open an interactive chat with Bash
GCP Professional Data Engineer
Designing data processing systems
Your Score:
Report Issue
Bash, the Crucial Exams Chat Bot
AI Bot
Loading...
Loading...
Loading...
Pass with Confidence.
IT & Cybersecurity Package
You have hit the limits of our free tier, become a Premium Member today for unlimited access.
Military, Healthcare worker, Gov. employee or Teacher? See if you qualify for a Community Discount.
Monthly
$19.99
$19.99/mo
Billed monthly, Cancel any time.
3 Month Pass
$44.99
$14.99/mo
One time purchase of $44.99, Does not auto-renew.
MOST POPULAR
Annual Pass
$119.99
$9.99/mo
One time purchase of $119.99, Does not auto-renew.
BEST DEAL
Lifetime Pass
$189.99
One time purchase, Good for life.
What You Get
All IT & Cybersecurity Package plans include the following perks and exams .