Your team builds an ELT workflow in Dataform that lands raw click-stream data in BigQuery and publishes cleaned tables for analysts. Compliance requires the nightly job to stop immediately whenever the current load introduces duplicate primary keys or orphaned foreign keys. Which Dataform construct should you use to add these data-quality gates so that the pipeline run automatically fails when the rule-checking query returns rows?
Configure the tables as incremental in Dataform and filter out problematic records with a WHERE clause referencing the latest updated_at timestamp.
Create separate .sqlx files defined with type: "assertion", each containing a query that returns rows when the quality rule is violated.
Rely on BigQuery's built-in NOT NULL and UNIQUE table constraints to reject bad data during the load step.
Attach postOperations blocks to the target tables to delete duplicates and unresolved foreign keys after the load finishes.
Create assertion actions in Dataform. An assertion is authored in a .sqlx file whose config block contains type: "assertion". At runtime Dataform executes the query and expects it to return an empty result set; any rows indicate a data-quality violation, causing the assertion to fail and the entire Dataform run to stop with an error. Post-operation scripts, BigQuery schema constraints, or incremental table filters do not provide the same automatic run-blocking behavior for complex business-rule checks such as duplicate detection or referential-integrity validation.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is Dataform and how does it work with BigQuery?
Open an interactive chat with Bash
What are '.sqlx' files and how are they used for assertions?
Open an interactive chat with Bash
Why are postOperations blocks or schema constraints unsuitable for Dataform's data-quality gating?
Open an interactive chat with Bash
What is an assertion in Dataform?
Open an interactive chat with Bash
How does Dataform stop the pipeline when an assertion fails?
Open an interactive chat with Bash
Why don't postOperations or BigQuery constraints work the same as assertions for data-quality checks?
Open an interactive chat with Bash
GCP Professional Data Engineer
Ingesting and processing the data
Your Score:
Report Issue
Bash, the Crucial Exams Chat Bot
AI Bot
Loading...
Loading...
Loading...
Pass with Confidence.
IT & Cybersecurity Package
You have hit the limits of our free tier, become a Premium Member today for unlimited access.
Military, Healthcare worker, Gov. employee or Teacher? See if you qualify for a Community Discount.
Monthly
$19.99
$19.99/mo
Billed monthly, Cancel any time.
3 Month Pass
$44.99
$14.99/mo
One time purchase of $44.99, Does not auto-renew.
MOST POPULAR
Annual Pass
$119.99
$9.99/mo
One time purchase of $119.99, Does not auto-renew.
BEST DEAL
Lifetime Pass
$189.99
One time purchase, Good for life.
What You Get
All IT & Cybersecurity Package plans include the following perks and exams .