GCP Professional Data Engineer Practice Question

Your SaaS platform ingests status events from 500 000 customer-deployed sensors every few seconds. Today the only requirement is near-real-time dashboards in BigQuery, but product teams expect to add fraud-detection microservices and a mobile alerting system later this year. You must design an ingestion architecture that meets current needs while allowing additional downstream consumers to process the same events without any changes to the sensor firmware. Which approach best satisfies these goals?

Publish every sensor event to a Cloud Pub/Sub topic; run a streaming Dataflow pipeline that subscribes and writes to BigQuery, and let future services create additional subscriptions to the same topic.
Have each sensor call the BigQuery streaming insert API; dashboards query BigQuery and future services read the data through BigQuery views.
Write events into a regional Cloud Spanner database and use change streams to replicate data into BigQuery; future services query the Spanner instance directly.
Upload batched JSON files from sensors to Cloud Storage using signed URLs; trigger Cloud Functions to load the files into BigQuery, and let future services read the files directly from the bucket.

Report Issue

Answer Description

Publishing events to a Cloud Pub/Sub topic creates a durable, horizontally scalable buffer that cleanly decouples producers from consumers. A streaming Dataflow job can subscribe to the topic and write to BigQuery for the existing dashboards, while future services merely create new subscriptions to the same topic and receive identical messages without impacting the sensors. Streaming directly to BigQuery couples every sensor to a single analytics system and forces future services to query BigQuery rather than consume raw events. Relying on Cloud Storage objects plus Cloud Functions introduces higher latency and file-oriented semantics that are harder to scale for per-message processing. Using Cloud Spanner and change streams adds cost and operational complexity, and it still requires each sensor to target a different endpoint than future services might need.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.