GCP Professional Data Engineer Practice Question

You are the lead data engineer at a media-streaming company. A real-time pipeline ingests Pub/Sub events, processes them through multiple Cloud Dataflow streaming jobs, and writes to BigQuery. Support engineers need a single dashboard that shows each stage's built-in system metrics together with Apache Beam user-defined counters, and they must receive alerts whenever end-to-end latency exceeds two minutes or worker-pool CPU utilization stays above 80 percent for five minutes. The business wants to minimize operational overhead and avoid operating any additional monitoring stack. Which monitoring strategy best meets these requirements?

Enable Cloud Trace for each Dataflow job and configure trace-based alerts; correlate spans from Pub/Sub, Dataflow, and BigQuery in the Cloud Trace console.
Leverage the default export of Dataflow job metrics to Cloud Monitoring, emit Beam counters as custom metrics, and build Cloud Monitoring dashboards and alerting policies for latency and CPU usage.
Create log sinks for Pub/Sub, Dataflow, and BigQuery into Cloud Logging, derive log-based metrics for latency and CPU, and visualize them with Logging charts and alerts.
Deploy a managed Prometheus server on GKE to scrape Dataflow worker logs, store the metrics, and visualize and alert on them using a self-hosted Grafana dashboard.

Report Issue

Answer Description

Cloud Dataflow automatically exports a rich set of built-in metrics (for example, elements-per-second, data freshness, CPU and memory usage) to Cloud Monitoring. Apache Beam user-defined counters can also be published as custom metrics that appear in the same Monitoring workspace. Using these metrics, engineers can build dashboards that combine Dataflow, Pub/Sub, and BigQuery data and can create alerting policies for latency and sustained CPU utilization. This provides a single pane of glass and proactive notifications without deploying or managing extra tooling.

The other options fall short:

Enabling Cloud Trace surfaces trace spans but not CPU, memory, or custom Beam counters.
Log-based metrics would require per-service extraction rules and still miss real-time Dataflow internals.
Running Prometheus and Grafana introduces additional infrastructure and contradicts the low-overhead requirement.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.