GCP Professional Data Engineer Practice Question

Your e-commerce company ingests tens of thousands of click-events per second into Pub/Sub. Data engineers must build a pipeline that consumes the stream in real time, performs sliding-window aggregations, and writes the results to BigQuery. When business logic changes, the same code should be rerun in batch to reprocess a full day of raw event files stored in Cloud Storage. The team wants a fully managed, auto-scaling service that lets them implement the pipeline once in Python without having to create or manage clusters. Which Google Cloud service best satisfies these requirements?

Managed Spark Structured Streaming jobs on Cloud Dataproc clusters
A Cloud Composer DAG orchestrating BigQuery SQL transformation jobs
Cloud Dataflow with the Apache Beam SDK
A Cloud Data Fusion pipeline triggered by Pub/Sub and writing to BigQuery

Report Issue

Answer Description

Cloud Dataflow is a serverless, fully managed execution service for Apache Beam pipelines. Beam's unified model lets developers write a single pipeline in Python (or Java, Go) that can be executed in both streaming and batch modes without code changes. Dataflow automatically provisions and scales worker resources, eliminating the need to create or manage clusters. Dataproc requires you to provision and manage Spark clusters and typically separates streaming and batch jobs. Cloud Data Fusion provides visual ETL but still spins up underlying clusters and does not let you write one Beam Python pipeline for both modes. Cloud Composer is an orchestrator, not a data-processing engine, and would still need another service to run the actual transformations. Therefore, Cloud Dataflow best meets all stated requirements.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.