GCP Professional Data Engineer Practice Question

An IoT platform ingests JSON sensor events from 5 million devices every second. Product managers need sub-second dashboards that show per-device metrics for the last 24 hours, while data scientists require SQL access to two years of historical events for ad-hoc analysis at reasonable cost. Events can be eventually consistent beyond the dashboard window. Which storage approach best satisfies both access patterns with the least operational complexity?

Store events in Bigtable and use Bigtable Change Streams to continuously load data into BigQuery for long-term analytics.
Stream events directly into BigQuery using the Storage Write API and use BigQuery for both dashboards and historical analysis.
Batch events into Parquet files on Cloud Storage and query them with BigQuery external tables, while Cloud Functions read the files for dashboard queries.
Persist events to Firestore for real-time reads and schedule nightly exports of collections to BigQuery tables.

Report Issue

Answer Description

Cloud Bigtable delivers single-digit millisecond reads and writes, making it suitable for the low-latency dashboard. By configuring a max-age garbage-collection rule, rows older than 24 hours are automatically purged, keeping the operational dataset small. Bigtable Change Streams can stream each mutation to Dataflow, which writes the data into time-partitioned BigQuery tables. Analysts then query years of history in BigQuery without impacting dashboard latency or maintaining custom export jobs. Using only BigQuery would not guarantee sub-second per-device lookups, Firestore struggles with large-scale analytical scans, and Cloud Storage with Parquet files cannot provide millisecond queries.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.