Your Cloud Composer DAG has three sequential tasks: create a Dataproc cluster, run a PySpark job on it, and delete the cluster. In testing, when the PySpark job fails, the deletion step is skipped, leaving the cluster running and incurring cost. To ensure the DataprocDeleteClusterOperator always runs after the PySpark task finishes-regardless of success, failure, or skip-which Airflow trigger_rule should you set on the delete-cluster task?
A task's trigger_rule defines when it executes relative to its upstream tasks. The default rule (all_success) requires every upstream task to succeed, so a failed PySpark job prevents the deletion step. Setting trigger_rule to "all_done" (TriggerRule.ALL_DONE) makes the delete task run after all upstream tasks finish in any state-success, failure, or skipped-guaranteeing that the Dataproc cluster is always removed. "one_success" will not run when every upstream task fails, "all_failed" requires all failures, and keeping "all_success" continues the original problem.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is a Trigger Rule in Apache Airflow?
Open an interactive chat with Bash
How does 'all_done' differ from 'all_success' in Airflow?
Open an interactive chat with Bash
Why is setting the correct trigger_rule important in workflows like Cloud Composer?
Open an interactive chat with Bash
What is a trigger_rule in Airflow?
Open an interactive chat with Bash
How does the 'all_done' trigger_rule ensure cluster deletion?
Open an interactive chat with Bash
What is a DataprocDeleteClusterOperator in Airflow?
Open an interactive chat with Bash
GCP Professional Data Engineer
Maintaining and automating data workloads
Your Score:
Report Issue
Bash, the Crucial Exams Chat Bot
AI Bot
Loading...
Loading...
Loading...
Pass with Confidence.
IT & Cybersecurity Package
You have hit the limits of our free tier, become a Premium Member today for unlimited access.
Military, Healthcare worker, Gov. employee or Teacher? See if you qualify for a Community Discount.
Monthly
$19.99 $11.99
$11.99/mo
Billed monthly, Cancel any time.
$19.99 after promotion ends
3 Month Pass
$44.99 $26.99
$8.99/mo
One time purchase of $26.99, Does not auto-renew.
$44.99 after promotion ends
Save $18!
MOST POPULAR
Annual Pass
$119.99 $71.99
$5.99/mo
One time purchase of $71.99, Does not auto-renew.
$119.99 after promotion ends
Save $48!
BEST DEAL
Lifetime Pass
$189.99 $113.99
One time purchase, Good for life.
Save $76!
What You Get
All IT & Cybersecurity Package plans include the following perks and exams .