GCP Professional Data Engineer Practice Question

A retail company ingests raw clickstream files into a Cloud Storage zone in Dataplex and transforms them into partitioned BigQuery tables for reporting. The compliance team has two new requirements:

Automatically maintain end-to-end lineage between the raw objects and the curated BigQuery tables so auditors can trace every record's origin.
Trigger an on-call alert whenever more than 2 % of rows violate data-quality rules that verify primary-key uniqueness and required column presence in the curated tables.
You must meet both requirements while minimizing custom code and operational toil. What should you do?

Schedule Cloud DLP inspection jobs on the BigQuery tables, send findings to Pub/Sub, and configure an alerting policy on the Pub/Sub topic; document lineage by manually adding Data Catalog tags.
Export BigQuery and Cloud Storage audit logs to BigQuery, join them with Cloud Asset Inventory feeds to infer lineage, and query the logs with scheduled queries that raise alerts when rule violations exceed 2 %.
In Dataplex, create a Data Quality scan on the curated BigQuery asset with the required rules, configure a Cloud Monitoring alert on the failed_rows_ratio metric that the scan exports, and enable the Data Lineage API to let Dataplex capture lineage automatically.
Deploy Apache Atlas on a long-running Dataproc cluster for lineage tracking, and build a Dataflow pipeline that computes data-quality statistics and writes custom metrics to Cloud Monitoring.

Report Issue

Answer Description

Dataplex natively provides both capabilities that are needed without additional custom services. Create a Data Quality scan on the curated BigQuery asset and add rules that check for primary-key uniqueness and non-null columns. The scan automatically exports the metric dataplex.googleapis.com/scan/data_quality/failed_rows_ratio to Cloud Monitoring, where an alerting policy can fire when the value exceeds 0.02. For lineage, enable the Data Lineage API in the same project; Dataplex automatically captures lineage between the Cloud Storage asset and the BigQuery tables whenever BigQuery jobs or Dataproc/Dataflow pipelines move the data, satisfying the audit requirement. The other options rely on manual tagging, custom pipelines, or third-party tooling, which increases operational overhead.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.