GCP Professional Data Engineer Practice Question

Your analytics team must orchestrate a nightly ETL workflow in Cloud Composer. The workflow must:

Launch a Dataflow Flex Template that loads JSON files from Cloud Storage into a staging BigQuery table.
Run a BigQuery SQL transformation after the Dataflow job successfully completes.
Delete the processed objects from the Cloud Storage bucket.
Automatically retry the Dataflow step up to three times, using an exponential back-off strategy, if the job fails. You want to minimize custom code and rely on built-in Airflow integrations with Google Cloud services. Which approach best meets these requirements?

Configure Pub/Sub notifications that trigger individual Cloud Functions for each step and connect them to Cloud Composer with TriggerDagRunOperator; use Dead Letter Topics for retries.
Create BashOperator tasks that invoke gcloud dataflow flex-template run, bq query, and gsutil rm commands in sequence, setting retries at the DAG level for failure handling.
Build a DAG that uses DataflowStartFlexTemplateOperator with retries=3 and retry_exponential_backoff enabled, followed by a DataflowJobStatusSensor, a BigQueryInsertJobOperator, and a GoogleCloudStorageDeleteOperator, relying on the Composer environment's default service account for authentication.
Use PythonOperator functions that programmatically call the Dataflow SDK, run the BigQuery API through the Google client library, and delete files, implementing custom retry logic inside the Python code.

Report Issue

Answer Description

Using the dedicated Google-provided Airflow operators keeps the DAG declarative and avoids custom scripting. DataflowStartFlexTemplateOperator submits the Flex Template and exposes standard Airflow retry arguments, so you can set retries=3 with retry_exponential_backoff=True to obtain exponential delays between attempts. Chaining a DataflowJobStatusSensor (or the synchronous option on the start operator) guarantees the downstream task starts only after the job reaches a SUCCESS state. BigQueryInsertJobOperator runs the transformation query natively, and GoogleCloudStorageDeleteOperator removes the processed objects. Because all operators use the Composer environment's service account by default, no additional credential code is required. The other choices rely on Bash or Python ad-hoc code, external triggers, or services that do not satisfy the dependency handling and retry requirements with minimal effort.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.