GCP Professional Data Engineer Practice Question

Your company is moving its analytics platform to Google Cloud. You must design ingestion for three distinct data sources: (1) an on-premises MySQL transactional database whose row-level changes must flow into BigQuery within seconds and without pausing the source workload, (2) tens of thousands of retail-store sensors that publish JSON messages over MQTT every second, and (3) 5 TB of historical click-stream files stored in Amazon S3 that must be loaded into BigQuery every night. You want to minimize infrastructure management while meeting latency goals. Which combination of Google Cloud ingestion services should you recommend?

Transfer Appliance for periodic MySQL exports, Cloud Functions HTTP endpoints for sensor ingestion, and BigQuery Data Transfer Service for importing Amazon S3 logs.
Database Migration Service continuous replication to Cloud SQL for MySQL, IoT Core for sensor ingestion, and Storage Transfer Service for importing Amazon S3 logs.
Datastream for MySQL change data capture, Cloud Pub/Sub for sensor ingestion, and BigQuery Data Transfer Service for importing Amazon S3 logs.
Cloud Data Fusion replication for MySQL, Pub/Sub Lite for sensor ingestion, and Storage Transfer Service for importing Amazon S3 logs.

Report Issue

Answer Description

Datastream provides a fully managed change-data-capture (CDC) service that can continuously read MySQL binary logs on-premises and write the changes to BigQuery (directly or through Cloud Storage) with sub-minute latency and no downtime to the source. Cloud Pub/Sub offers a global, serverless message ingestion endpoint that supports MQTT devices through standard bridges and automatically scales to handle high-throughput sensor streams, satisfying the second requirement without capacity planning. BigQuery Data Transfer Service has a native Amazon S3 connector that schedules and executes recurring bulk transfers straight into BigQuery, eliminating the need to manage intermediate infrastructure and meeting the nightly 5 TB load. The alternative options either introduce higher operational overhead (Cloud Data Fusion replication, gsutil scripting), lack native MQTT or global scaling (Pub/Sub Lite, Cloud Functions), or do not deliver data directly into BigQuery (Database Migration Service, Storage Transfer Service). Therefore, the Datastream + Cloud Pub/Sub + BigQuery Data Transfer Service combination best meets all three requirements with minimal operational effort.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.