Your company is building a unified analytics platform on Google Cloud. Factory sensors publish billions of events every second via an MQTT broker, while ten years of compressed CSV telemetry is stored in Amazon S3. A single Dataflow pipeline must apply the same transformations to both the real-time stream and periodic back-fills, then write curated data to BigQuery. To minimize operational effort and ensure Dataflow performs all transformations, which managed Google Cloud services should you use to ingest the streaming and batch data, respectively?
Datastream to capture MQTT sensor messages and scheduled gsutil copy jobs on Compute Engine to pull the S3 archives.
Cloud Pub/Sub for the sensor stream and BigQuery Data Transfer Service to load the S3 archives directly into BigQuery.
Cloud Pub/Sub for the sensor stream and Storage Transfer Service to move the S3 archives into Cloud Storage for batch ingestion.
Cloud Data Fusion's real-time streaming source for sensors and Transfer Appliance to ingest the S3 archives.
Cloud Pub/Sub is Google Cloud's serverless, horizontally scalable messaging service that can ingest MQTT events through the MQTT to Pub/Sub template, allowing a Dataflow streaming pipeline to subscribe directly without extra infrastructure. Storage Transfer Service provides scheduled, fully managed transfers from Amazon S3 into Cloud Storage; Dataflow can then read the files in batch, keeping transformations inside the pipeline. BigQuery Data Transfer Service would load data straight into BigQuery, bypassing Dataflow. Transfer Appliance targets offline bulk moves, and Datastream or gsutil scripts add unnecessary operational overhead.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is Google Cloud Pub/Sub, and how is it used for streaming data?
Open an interactive chat with Bash
How does the Storage Transfer Service work to move data from Amazon S3 to Google Cloud?
Open an interactive chat with Bash
Why is it important to process both real-time and batch data within the same Dataflow pipeline?
Open an interactive chat with Bash
What is Cloud Pub/Sub and how does it work?
Open an interactive chat with Bash
How does Storage Transfer Service work with Amazon S3 archives?
Open an interactive chat with Bash
Why is Dataflow used for both real-time and batch processing?
Open an interactive chat with Bash
GCP Professional Data Engineer
Ingesting and processing the data
Your Score:
Report Issue
Bash, the Crucial Exams Chat Bot
AI Bot
Loading...
Loading...
Loading...
Pass with Confidence.
IT & Cybersecurity Package
You have hit the limits of our free tier, become a Premium Member today for unlimited access.
Military, Healthcare worker, Gov. employee or Teacher? See if you qualify for a Community Discount.
Monthly
$19.99
$19.99/mo
Billed monthly, Cancel any time.
3 Month Pass
$44.99
$14.99/mo
One time purchase of $44.99, Does not auto-renew.
MOST POPULAR
Annual Pass
$119.99
$9.99/mo
One time purchase of $119.99, Does not auto-renew.
BEST DEAL
Lifetime Pass
$189.99
One time purchase, Good for life.
What You Get
All IT & Cybersecurity Package plans include the following perks and exams .