Your team must automate a nightly data pipeline that 1) launches a parameterized Dataflow Flex template, 2) executes a BigQuery stored procedure on completion, and 3) calls an external Jira REST endpoint to update a ticket. The run order must change if the Dataflow job fails, and business analysts want to reuse existing Python libraries for custom retry logic. All resources will run in a private VPC with no public IPs. Which orchestration approach best meets these requirements?
Schedule the Dataflow Flex template with Cloud Scheduler, embed BigQuery execution and Jira calls inside the pipeline, and rely on Dataflow's built-in retries for error handling.
Chain Cloud Functions with Pub/Sub topics: the first function launches the Dataflow template, a second polls Dataflow and invokes the BigQuery procedure, and a third updates Jira.
Create a private-IP Cloud Composer environment and build a Python DAG that uses built-in Dataflow and BigQuery operators plus a PythonOperator that posts to the Jira REST API.
Use Cloud Workflows triggered nightly by Cloud Scheduler; call Dataflow, BigQuery, and Jira REST endpoints with HTTP steps and handle failures with conditional states.
Cloud Composer is a managed Apache Airflow service, so DAGs can use the Cloud Dataflow and BigQuery operators to invoke jobs with runtime parameters and to branch based on success or failure. Airflow's Python environment lets the team import existing libraries and write custom retry/branching code, while the Composer environment can be created with private IP networking and Cloud NAT for restricted egress. Workflows is serverless but offers limited Python support and branching logic; chaining Pub/Sub-triggered Cloud Functions or relying solely on Cloud Scheduler would require significantly more custom code and lacks the rich dependency management, monitoring, and Python operator ecosystem required.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is Cloud Composer and how does it support orchestration in private VPCs?
Open an interactive chat with Bash
How does Apache Airflow handle task failures and branching logic?
Open an interactive chat with Bash
Why are PythonOperators useful in Cloud Composer environments?
Open an interactive chat with Bash
What is a private-IP Cloud Composer environment?
Open an interactive chat with Bash
What are DAGs in Apache Airflow?
Open an interactive chat with Bash
How does Cloud NAT support private-IP environments?
Open an interactive chat with Bash
GCP Professional Data Engineer
Ingesting and processing the data
Your Score:
Report Issue
Bash, the Crucial Exams Chat Bot
AI Bot
Loading...
Loading...
Loading...
Pass with Confidence.
IT & Cybersecurity Package
You have hit the limits of our free tier, become a Premium Member today for unlimited access.
Military, Healthcare worker, Gov. employee or Teacher? See if you qualify for a Community Discount.
Monthly
$19.99 $11.99
$11.99/mo
Billed monthly, Cancel any time.
$19.99 after promotion ends
3 Month Pass
$44.99 $26.99
$8.99/mo
One time purchase of $26.99, Does not auto-renew.
$44.99 after promotion ends
Save $18!
MOST POPULAR
Annual Pass
$119.99 $71.99
$5.99/mo
One time purchase of $71.99, Does not auto-renew.
$119.99 after promotion ends
Save $48!
BEST DEAL
Lifetime Pass
$189.99 $113.99
One time purchase, Good for life.
Save $76!
What You Get
All IT & Cybersecurity Package plans include the following perks and exams .