Your team ingests clickstream events into a Pub/Sub topic. Traffic is highly bursty (up to 500 000 events per second). Each record must be parsed, enriched with a lookup table stored in Cloud Storage, placed into one-minute tumbling windows, and then written to BigQuery so near-real-time dashboards stay current. The same transformation logic must also be reused every weekend to reprocess a month of raw data that is stored in Cloud Storage, without modifying code or provisioning extra infrastructure. Which Google Cloud service should you build on to satisfy these requirements while minimizing operational overhead?
BigQuery scheduled queries that read raw files from Cloud Storage and update reporting tables
Dataflow with an Apache Beam pipeline that reads from Pub/Sub or Cloud Storage and writes to BigQuery
Cloud Functions triggered by Pub/Sub that stream each message directly into BigQuery
A Dataproc cluster running Spark Streaming jobs for live data and separate Spark batch jobs for backfills
Dataflow pipelines written with Apache Beam can run in both streaming and batch modes without code changes, letting you process data from Pub/Sub in near real time and reprocess historical files from Cloud Storage on a schedule. Dataflow automatically scales worker instances to handle bursts of traffic, manages infrastructure, and provides native connectors for Pub/Sub, Cloud Storage, and BigQuery. Dataproc requires you to create and manage clusters; Cloud Functions are unlikely to keep pace with 500 000 events per second and lack built-in windowing; BigQuery scheduled queries cannot transform streaming data arriving from Pub/Sub.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is Apache Beam and how does it work in Dataflow?
Open an interactive chat with Bash
How does Dataflow handle bursty traffic without manual intervention?
Open an interactive chat with Bash
What is windowing in Apache Beam and why is it useful for this scenario?
Open an interactive chat with Bash
What is Apache Beam, and why is it useful for Dataflow pipelines?
Open an interactive chat with Bash
How does Dataflow handle bursty traffic and scalability?
Open an interactive chat with Bash
What are tumbling windows, and how do they help in stream processing?
Open an interactive chat with Bash
GCP Associate Cloud Engineer
Planning and implementing a cloud solution
Your Score:
Report Issue
Bash, the Crucial Exams Chat Bot
AI Bot
Loading...
Loading...
Loading...
Pass with Confidence.
IT & Cybersecurity Package
You have hit the limits of our free tier, become a Premium Member today for unlimited access.
Military, Healthcare worker, Gov. employee or Teacher? See if you qualify for a Community Discount.
Monthly
$19.99 $11.99
$11.99/mo
Billed monthly, Cancel any time.
$19.99 after promotion ends
3 Month Pass
$44.99 $26.99
$8.99/mo
One time purchase of $26.99, Does not auto-renew.
$44.99 after promotion ends
Save $18!
MOST POPULAR
Annual Pass
$119.99 $71.99
$5.99/mo
One time purchase of $71.99, Does not auto-renew.
$119.99 after promotion ends
Save $48!
BEST DEAL
Lifetime Pass
$189.99 $113.99
One time purchase, Good for life.
Save $76!
What You Get
All IT & Cybersecurity Package plans include the following perks and exams .