GCP Professional Data Engineer Practice Question

Your media company ingests millions of click events per second into Pub/Sub. Design a pipeline that:

Groups events into user sessions separated by 30 minutes of inactivity and writes the enriched session records to BigQuery with sub-minute latency.
After each UTC day ends, executes a dependent SQL job that aggregates the previous day's sessions into a summary table. You want minimal infrastructure management and a clear separation between the real-time transformation and the daily batch step. Which solution meets these requirements?

Use Cloud Data Fusion in batch mode to read from Pub/Sub, apply sessionization with Wrangler transformations, and load sessions into Cloud SQL; call a Workflows orchestration each night to aggregate into BigQuery.
Invoke Cloud Functions for every Pub/Sub event to write raw clicks to Bigtable; configure a daily BigQuery Data Transfer from Bigtable at 00:00 UTC and rely on materialized views for real-time sessionization.
Create a streaming Apache Beam pipeline on Dataflow that applies 30-minute session windows and writes to BigQuery via the Storage Write API; orchestrate a Cloud Composer DAG that, once the Dataflow watermark passes midnight UTC, triggers a BigQuery SQL job to build the daily summary table.
Run a long-lived Spark Streaming job on Dataproc to perform sessionization and write results to Cloud Storage; use Cloud Scheduler to invoke a Cloud Function at 00:10 UTC that runs a BigQuery aggregation query.

Report Issue

Answer Description

A serverless Apache Beam pipeline on Dataflow can apply 30-minute session windows to streaming Pub/Sub events and write the results to BigQuery through the BigQuery Storage Write API, giving near-real-time availability for analysts. Cloud Composer (managed Airflow) provides a fully managed orchestration layer that can wait for the streaming job's watermark to pass midnight and then launch the dependent BigQuery SQL task to build the daily summary table. This approach satisfies both transformation and orchestration needs with the least operational overhead.

The Dataproc option requires cluster provisioning and maintenance and uses Cloud Scheduler plus Cloud Functions, which lack built-in dependency management. The Cloud Data Fusion batch design fails to meet the sub-minute latency requirement. The Bigtable design cannot perform real-time sessionization in BigQuery and BigQuery Data Transfer Service does not pull data from Bigtable, so it does not satisfy either requirement.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.