Your media analytics team runs a streaming Dataflow pipeline that writes to BigQuery. Cloud Monitoring currently alerts only on the Pub/Sub subscription backlog, but last night BigQuery rejected inserts for two hours because the destination table had accidentally been set to expire, and no alert fired. To reduce MTTR you want an alert that triggers whenever streaming inserts to the table fail continuously for at least five minutes, while avoiding noise during brief schema-update glitches. What should you do?
Publish BigQuery job-completion events from the dataset to Pub/Sub and create an alert on the subscription's message backlog exceeding five minutes.
Create a Cloud Monitoring alert on the BigQuery storage/bytes_used metric for the table, firing when the value stops increasing for five minutes.
Enable BigQuery export to Cloud Storage and set up an Object Change Notification; alert if no new file appears in the bucket within five minutes.
Define a logs-based metric that counts BigQuery audit log entries for streaming inserts with a non-zero status code, and configure an alerting policy that triggers when the metric's rate remains above zero for five consecutive minutes.
Streaming inserts that fail generate BigQuery Audit Log entries whose protoPayload.status.code is non-zero. By creating a logs-based metric that counts those error records you can accurately measure sustained insert failures, independent of storage size or job completions. An alerting policy on that metric with a five-minute window fires only when errors persist for that period, eliminating transient false positives. Monitoring table storage size, exporting data to Cloud Storage, or alerting on job-completion Pub/Sub messages would not directly or promptly detect streaming insert errors, and therefore would miss the problem or create unnecessary noise.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is a logs-based metric in Cloud Monitoring?
Open an interactive chat with Bash
What are BigQuery Audit Logs, and what kind of data do they contain?
Open an interactive chat with Bash
How can alerting policies reduce Mean Time to Resolution (MTTR)?
Open an interactive chat with Bash
What are BigQuery Audit Logs?
Open an interactive chat with Bash
How do logs-based metrics work in Cloud Monitoring?
Open an interactive chat with Bash
What is MTTR and why is it important in incident response?
Open an interactive chat with Bash
GCP Professional Data Engineer
Maintaining and automating data workloads
Your Score:
Report Issue
Bash, the Crucial Exams Chat Bot
AI Bot
Loading...
Loading...
Loading...
Pass with Confidence.
IT & Cybersecurity Package
You have hit the limits of our free tier, become a Premium Member today for unlimited access.
Military, Healthcare worker, Gov. employee or Teacher? See if you qualify for a Community Discount.
Monthly
$19.99 $11.99
$11.99/mo
Billed monthly, Cancel any time.
$19.99 after promotion ends
3 Month Pass
$44.99 $26.99
$8.99/mo
One time purchase of $26.99, Does not auto-renew.
$44.99 after promotion ends
Save $18!
MOST POPULAR
Annual Pass
$119.99 $71.99
$5.99/mo
One time purchase of $71.99, Does not auto-renew.
$119.99 after promotion ends
Save $48!
BEST DEAL
Lifetime Pass
$189.99 $113.99
One time purchase, Good for life.
Save $76!
What You Get
All IT & Cybersecurity Package plans include the following perks and exams .