GCP Professional Data Engineer Practice Question

Your team must build a daily batch workflow that cleanses raw sensor files in Cloud Storage with Dataflow, enriches each record with predictions from an existing AutoML classification model, and then loads the results into BigQuery for analytics. The solution must provide pipeline versioning, lineage and execution metadata, allow data scientists to add new transformation or training steps declaratively, and avoid writing custom scheduling or dependency code. Which design best meets these goals?

Implement a Vertex AI Pipeline where a custom container component launches the Dataflow cleansing job, followed by a built-in BatchPredictionJob component and a BigQuery SQL load component.
Combine all cleansing, prediction, and loading logic inside a single Dataflow pipeline that calls the AutoML model from a user-defined function and writes directly to BigQuery.
Build an Apache Airflow DAG in Cloud Composer that triggers a Dataflow job, calls the AutoML prediction API with Python operators, and then loads data into BigQuery.
Use Workflows to chain Cloud Functions that invoke the Dataflow Flex Template, call the AutoML model via REST, and insert the enriched records into BigQuery.

Report Issue

Answer Description

Vertex AI Pipelines natively manages end-to-end ML workflows and automatically records pipeline definitions, run metadata, and lineage in Vertex ML Metadata. You can express the workflow as a series of containerized components: one component invokes the Dataflow cleansing job (for example, with DataflowPythonJobOp), a built-in BatchPredictionJob component performs batch predictions with the AutoML model, and a final component runs a BigQuery SQL load job. The pipeline can be versioned and parameterized through infrastructure-as-code, and Vertex AI handles execution order and retries without custom scheduling logic.

Cloud Composer (Airflow) and Workflows can orchestrate the steps, but they lack integrated ML metadata tracking and require more code to manage dependencies. Embedding all cleansing, prediction, and loading logic in a single Dataflow pipeline does capture some transformation lineage for BigQuery, yet it offers no ML-specific metadata, makes the workflow hard to extend modularly, and ties all stages to one job. Therefore, implementing the workflow as a Vertex AI Pipeline with separate components is the most suitable approach.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.