A team operates a streaming Dataflow job that writes event data to BigQuery. When downstream tables start to fall behind real time, analysts complain that dashboards show stale information. You are asked to configure an alert in Cloud Monitoring that notifies the on-call group if the pipeline is more than 5 minutes behind the event time watermark for at least 10 consecutive minutes. Which metric and aggregation should you base the alerting policy on to meet this requirement with the least noise?
Metric: dataflow.googleapis.com/job/total_vcpu_time aggregated with the average across all workers
Metric: dataflow.googleapis.com/job/system_lag aggregated with the maximum across all job workers
Metric: dataflow.googleapis.com/job/elements_processed_count aggregated with the rate over the job
Metric: pubsub.googleapis.com/subscription/num_undelivered_messages aggregated with the sum across all input subscriptions
The metric dataflow.googleapis.com/job/system_lag measures, in seconds, the difference between the current event-time watermark and the present clock time for each running Dataflow job. Alerting on the MAX of this metric across all workers ensures the policy triggers whenever any part of the pipeline lags beyond the 5-minute threshold. Using element counts or processed-bytes metrics does not directly indicate freshness, and Pub/Sub backlog reflects only input subscription state, not end-to-end pipeline latency. Averaging system lag can mask localized hotspots because a few timely workers may dilute a rising lag on others.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is an event-time watermark in Dataflow?
Open an interactive chat with Bash
Why use the MAX aggregation for the system lag metric?
Open an interactive chat with Bash
What distinguishes system lag from Pub/Sub backlog?
Open an interactive chat with Bash
What is the event-time watermark in Dataflow?
Open an interactive chat with Bash
Why is the MAX aggregation used for system lag in Cloud Monitoring alerts?
Open an interactive chat with Bash
How does dataflow.googleapis.com/job/system_lag differ from pubsub.googleapis.com/subscription/num_undelivered_messages?
Open an interactive chat with Bash
GCP Professional Data Engineer
Maintaining and automating data workloads
Your Score:
Report Issue
Bash, the Crucial Exams Chat Bot
AI Bot
Loading...
Loading...
Loading...
Pass with Confidence.
IT & Cybersecurity Package
You have hit the limits of our free tier, become a Premium Member today for unlimited access.
Military, Healthcare worker, Gov. employee or Teacher? See if you qualify for a Community Discount.
Monthly
$19.99 $11.99
$11.99/mo
Billed monthly, Cancel any time.
$19.99 after promotion ends
3 Month Pass
$44.99 $26.99
$8.99/mo
One time purchase of $26.99, Does not auto-renew.
$44.99 after promotion ends
Save $18!
MOST POPULAR
Annual Pass
$119.99 $71.99
$5.99/mo
One time purchase of $71.99, Does not auto-renew.
$119.99 after promotion ends
Save $48!
BEST DEAL
Lifetime Pass
$189.99 $113.99
One time purchase, Good for life.
Save $76!
What You Get
All IT & Cybersecurity Package plans include the following perks and exams .