GCP Professional Data Engineer Practice Question

A fintech company receives a CSV file with millions of transactions every night in the Cloud Storage folder gs://txn-ingress/daily/. The required workflow is:

Detect when the file for a given day arrives.
Launch an existing Dataflow Flex Template that cleanses and enriches the file and writes the results to a BigQuery table. Downstream tasks must wait until this job finishes.
After the Dataflow job succeeds, start a Cloud DLP inspection job on the new BigQuery table.
If the DLP job reports any HIGH-risk findings, invoke a Cloud Function that masks the affected columns; otherwise end the DAG. Non-functional constraints:

Orchestration must be implemented in Cloud Composer with minimal operational overhead.
Transient failures must be retried with exponential back-off.
Only one run of this DAG is allowed per calendar day, regardless of how many files arrive.

Which Cloud Composer design best satisfies all requirements?

Replace Cloud Composer with Cloud Workflows triggered by Cloud Storage events; sequence the Dataflow, DLP, and Cloud Function calls in YAML definitions using built-in retries.
Deploy a Cloud Composer 1 environment, poll Pub/Sub notifications with PubSubPullSensor, launch the Dataflow job via a BashOperator running gcloud commands, and create the DLP job with a BigQueryOperator, while keeping default concurrency settings.
Use Cloud Scheduler to trigger the Dataflow template and have a Cloud Function start the DLP job; only invoke Cloud Composer when masking is required, relying on default retries and allowing multiple overlapping DAG runs.
Define the DAG in a Cloud Composer 2 environment. Add a GoogleCloudStoragePrefixSensor for gs://txn-ingress/daily/, call DataflowStartFlexTemplateOperator with wait_until_finished=True, then CloudDLPCreateJobOperator. Use a BranchPythonOperator to check DLP findings and, if any HIGH-risk results are found, invoke a Cloud Function via GoogleCloudFunctionsOperator; otherwise finish the DAG. Configure max_active_runs=1 and retry_exponential_backoff=True in default_args.

Report Issue

Answer Description

Implement the DAG in Cloud Composer 2, which runs on GKE Autopilot to minimize operational management. Use GoogleCloudStoragePrefixSensor (or GoogleCloudStorageObjectSensor) to wait for the nightly CSV file in gs://txn-ingress/daily/. Launch the Dataflow Flex Template with DataflowStartFlexTemplateOperator and set wait_until_finished=True so that downstream tasks block until the job completes successfully. After completion, create a DLP inspection job with CloudDLPCreateJobOperator. A BranchPythonOperator can examine the DLP findings and determine whether to call the CloudFunctionsInvokeFunctionOperator that triggers the masking Cloud Function or to end the workflow. Setting max_active_runs=1 in the DAG ensures only one run per calendar day, and retry_exponential_backoff=True on tasks enables exponential back-off for retries. Alternative designs that rely on separate services for orchestration or that use Cloud Composer 1 with custom Bash commands either increase operational overhead or fail to meet all requirements.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.