GCP Professional Data Engineer Practice Question

Your team must design a storage layout for an e-commerce platform that streams about 50 000 clickstream events per second (≈5 TB/day). The recommendation engine needs single-digit millisecond access to the most recent seven days of events, while data scientists expect to run large ad-hoc SQL analyses on the full event history every morning. You want a fully managed solution that minimizes operational overhead and storage cost. Which approach best satisfies these requirements?

Persist events in Cloud SQL for PostgreSQL with logical replication into BigQuery; query Cloud SQL for the latest seven days and BigQuery for historical analysis.
Ingest events directly into BigQuery partitioned tables and build materialized views on the last seven days for the recommendation engine.
Write all events to Cloud Storage Nearline; define a BigQuery external table for analytics and use Memorystore to cache the most recent events for recommendations.
Stream events into Cloud Bigtable for operational access; use a Dataflow job to write each event to a partitioned BigQuery table and delete data in Bigtable after seven days.

Report Issue

Answer Description

Cloud Bigtable is optimized for very high-throughput, sub-10 ms reads and writes, making it well suited for serving the most recent seven days of clickstream events to the recommendation engine. A streaming Dataflow pipeline can simultaneously persist each event to BigQuery, where partitioned tables support cost-effective, serverless SQL analytics on the full history. Periodically expiring data older than seven days in Bigtable keeps operational storage costs low.

Cloud SQL cannot sustain tens of thousands of writes per second or provide millisecond reads at this scale, and its replication to BigQuery would still leave performance and scalability gaps. Memorystore is an in-memory cache that is not intended to store terabytes of data and lacks persistence. Storing all data only in BigQuery would not meet the sub-10 ms latency requirement for online recommendations. Therefore, the Bigtable plus Dataflow-to-BigQuery pattern is the most appropriate choice.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.