GCP Professional Data Engineer Practice Question

Your retail company operates a 600 TB on-premises Hadoop cluster that stores historical sales logs. The corporate data center connects to Google Cloud over a 200 Mbps dedicated link that also carries other production traffic. After an initial one-time backfill of 200 GB, the cluster produces about 1 TB of new log data each day. All data must become queryable in BigQuery, the existing 200 Mbps link must not be saturated, and ongoing operational effort should be minimal. Which approach should you recommend?

Configure BigQuery Data Transfer Service to connect to the on-prem Hadoop cluster, performing an initial full import and scheduling daily incremental transfers.
Use gsutil rsync over the existing 200 Mbps link to copy the 600 TB and the 1 TB daily increments into a Cloud Storage location that BigQuery reads as an external table.
Provision a 10 Gbps Dedicated Interconnect and run a continuous Dataflow pipeline that streams both historical and daily data directly from the Hadoop cluster into BigQuery.
Ship a Transfer Appliance to move the 600 TB of historical data into Cloud Storage, then schedule daily Storage Transfer Service jobs with on-prem agents to copy new HDFS files to the same bucket and load them into BigQuery.

Report Issue

Answer Description

Copying 600 TB over a 200 Mbps line would take many months and monopolize the WAN, so an offline bulk transfer is preferable. Ordering a Transfer Appliance lets the team load the historical data locally and ship the device to Google, where the data is imported into Cloud Storage without consuming network bandwidth. After the backfill, Storage Transfer Service agents on-prem can run scheduled, bandwidth-throttled jobs that move only the new 1 TB of daily files to the same Cloud Storage bucket. BigQuery can ingest the data through scheduled LOAD jobs or external tables. Provisioning a 10 Gbps Dedicated Interconnect adds cost and operational overhead; gsutil rsync over the 200 Mbps link would still overload the network; and BigQuery Data Transfer Service cannot connect directly to on-prem Hadoop. Combining Transfer Appliance for the initial bulk move with Storage Transfer Service for ongoing increments best meets the requirements.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.