Your analytics team uses Cloud Data Fusion to load operational data into a staging dataset in BigQuery each night. SQL transformations are managed in Dataform and publish curated tables to a production dataset feeding dashboards. You need to block the transformation workflow whenever staging tables contain null customer IDs, duplicate order IDs, or negative revenue values. The solution must integrate with the existing Dataform workflow, require little custom code, and surface failures in the run logs. What should you do?
Configure a Cloud Monitoring alert on the nightly load row count; send an incident to the on-call team and rerun the Dataform workflow manually when alerted.
Add NOT NULL and CHECK constraints to the staging tables; rely on BigQuery to reject bad data and make Dataform fail automatically.
Create SQLX assertion files in Dataform that SELECT all rows violating each rule and tag them as tests so the Dataform run fails if any record is returned.
Set up a Cloud Scheduler job that triggers a Cloud Function to run custom validation queries after each load; if any query finds bad data, publish a Pub/Sub message that cancels the Dataform run via Cloud Build.
Dataform supports built-in data quality tests called assertions. An assertion is just a SQLX file whose SELECT statement returns the rows that violate a quality rule; when an assertion returns any rows, the Dataform run fails, downstream operations are skipped, and the failure is shown in the run logs and UI. Creating assertions that check for NULL customer_id values, duplicate order_id values, and negative revenue therefore stops the workflow automatically and keeps the implementation entirely inside the existing Dataform environment with minimal additional code.
Adding BigQuery NOT NULL or CHECK constraints would not stop a batch query in BigQuery today because those constraints are informational only. External Cloud Functions or Monitoring alerts would add custom components and would not natively cancel the Dataform run; they also introduce operational overhead and delayed feedback.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is Dataform and why is it used?
Open an interactive chat with Bash
What are SQLX assertion files in Dataform?
Open an interactive chat with Bash
How do NOT NULL and CHECK constraints work in BigQuery?
Open an interactive chat with Bash
GCP Professional Data Engineer
Designing data processing systems
Your Score:
Report Issue
Bash, the Crucial Exams Chat Bot
AI Bot
Loading...
Loading...
Loading...
Pass with Confidence.
IT & Cybersecurity Package
You have hit the limits of our free tier, become a Premium Member today for unlimited access.
Military, Healthcare worker, Gov. employee or Teacher? See if you qualify for a Community Discount.
Monthly
$19.99 $11.99
$11.99/mo
Billed monthly, Cancel any time.
$19.99 after promotion ends
3 Month Pass
$44.99 $26.99
$8.99/mo
One time purchase of $26.99, Does not auto-renew.
$44.99 after promotion ends
Save $18!
MOST POPULAR
Annual Pass
$119.99 $71.99
$5.99/mo
One time purchase of $71.99, Does not auto-renew.
$119.99 after promotion ends
Save $48!
BEST DEAL
Lifetime Pass
$189.99 $113.99
One time purchase, Good for life.
Save $76!
What You Get
All IT & Cybersecurity Package plans include the following perks and exams .