GCP Professional Data Engineer Practice Question

Your company is designing a streaming data pipeline on Google Cloud for a video-streaming platform. Mobile devices publish about 500 000 events per second. A personalization microservice must query the most recent events with single-digit millisecond latency, while the marketing team needs interactive SQL access to 13 months of raw events scanning terabytes daily. Operations staff want a managed, serverless solution and require that processing VMs have no public IP addresses. Which architecture best satisfies these goals?

Ingest events with Pub/Sub and use a single Dataflow job to write all data to time-partitioned BigQuery tables; have the personalization service query BigQuery through BI Engine; disable public IPs on Dataflow workers.
Use Pub/Sub for ingestion; run Dataflow workers without public IPs to branch the stream, writing concurrently to Cloud Bigtable for the personalization microservice and to partitioned BigQuery tables for marketing analytics.
Deploy Apache Kafka on Compute Engine and process the stream with Spark Streaming on Dataproc, persisting events in Cloud SQL for both personalization and marketing queries.
Ingest with Pub/Sub and have Dataflow write Avro files to Cloud Storage; marketers query the files with Presto on Dataproc, and the personalization service retrieves recent files via Cloud Storage through Cloud CDN.

Report Issue

Answer Description

Cloud Bigtable is optimized for very low-latency, high-throughput point lookups and is therefore suited for the personalization service that needs millisecond access to the most recent events. BigQuery provides serverless, petabyte-scale, ANSI-SQL analytics that fits the marketing team's interactive ad-hoc queries over 13 months of data. A single Dataflow streaming job can read from Pub/Sub and branch the stream to multiple sinks, writing each record both to Bigtable and to a partitioned BigQuery table. Dataflow workers can be configured to run without public IP addresses inside a VPC, meeting the networking requirement. The other options fail one or more requirements: BigQuery alone cannot serve sub-second lookups; Cloud SQL cannot sustain the scale; Cloud Storage plus CDN cannot meet millisecond latency and is not serverless for analytics.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.