Your organization runs a streaming Apache Beam pipeline on Cloud Dataflow. The job is launched from Cloud Composer with the parameters --zone=us-central1-a and processes Pub/Sub messages into a BigQuery table. Twice last quarter, Google maintenance in us-central1-a caused the worker pool to become unavailable and the pipeline stopped until it was manually restarted. You must redesign the deployment so that the job automatically survives a zonal outage, requires no code changes, and adds as little operational overhead as possible. What should you do?
Deploy an identical standby Dataflow job in us-central1-b and let Pub/Sub round-robin messages between the two jobs.
Start the pipeline specifying only --region=us-central1; omit any zone flags so Dataflow can distribute and, if needed, relocate workers across multiple zones in the region.
Port the pipeline to Dataproc and create a regional cluster with three master nodes spread across zones, then schedule the Spark Streaming job there.
Move the target BigQuery table to a multi-regional location so queries remain available even if a zone fails.
When a Dataflow job is started with only --region=us-central1 (and no --zone or --worker_zone flag), the service is free to place and, if necessary, move workers across any zone in that region. If the zone originally selected becomes unavailable, Dataflow transparently restarts workers in another healthy zone, allowing the streaming pipeline to continue without intervention. Creating duplicate jobs, migrating to Dataproc, or relocating the BigQuery dataset either introduce significant complexity or do not address the root problem of a single-zone worker pool.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is Cloud Dataflow, and how does it work?
Open an interactive chat with Bash
What is the difference between specifying `--region` and `--zone` in Dataflow?
Open an interactive chat with Bash
How does Pub/Sub ensure message delivery to Dataflow during downtime?
Open an interactive chat with Bash
What is Cloud Dataflow?
Open an interactive chat with Bash
How does specifying only `--region` help with zonal outages?
Open an interactive chat with Bash
What is the role of Pub/Sub in the streaming pipeline?
Open an interactive chat with Bash
GCP Professional Data Engineer
Maintaining and automating data workloads
Your Score:
Report Issue
Bash, the Crucial Exams Chat Bot
AI Bot
Loading...
Loading...
Loading...
Pass with Confidence.
IT & Cybersecurity Package
You have hit the limits of our free tier, become a Premium Member today for unlimited access.
Military, Healthcare worker, Gov. employee or Teacher? See if you qualify for a Community Discount.
Monthly
$19.99
$19.99/mo
Billed monthly, Cancel any time.
3 Month Pass
$44.99
$14.99/mo
One time purchase of $44.99, Does not auto-renew.
MOST POPULAR
Annual Pass
$119.99
$9.99/mo
One time purchase of $119.99, Does not auto-renew.
BEST DEAL
Lifetime Pass
$189.99
One time purchase, Good for life.
What You Get
All IT & Cybersecurity Package plans include the following perks and exams .