GCP Professional Data Engineer Practice Question

Your company stores 600 TB of click-stream logs in an Amazon S3 bucket and ingests about 3 TB of new objects each day. You must replicate the data to a customer-managed-encrypted Cloud Storage bucket in us-central1 within 1 hour of object creation, automatically retry after network interruptions, and avoid re-copying objects that have not changed. A 10 Gbps private link connects AWS and Google Cloud. Which approach should you take?

Ship 200 TB Transfer Appliance devices to Google each week and ingest the data into the CMEK-protected Cloud Storage bucket on arrival.
Provision a high-memory Compute Engine VM, mount the S3 bucket with s3fs, and schedule a cron job that runs gsutil -m rsync to copy files into the CMEK-protected bucket every 30 minutes.
Set up an S3 event notification to Amazon SNS that invokes a Cloud Function for every new object; have the function stream each object into the CMEK-protected Cloud Storage bucket using signed URLs.
Configure two recurring Storage Transfer Service jobs from the S3 bucket to the CMEK-protected Cloud Storage bucket, each scheduled hourly but offset by 30 minutes. Enable the overwrite-if-newer option, grant the Storage Transfer Service service agent permission to use the Cloud KMS key, and provide AWS credentials for source access.

Report Issue

Answer Description

Storage Transfer Service (STS) is purpose-built for large-scale, online transfers from Amazon S3 to Cloud Storage. A recurring STS job can run as often as every hour, and the "overwrite if newer" option ensures that only objects modified since the previous successful run are copied, saving bandwidth and time. STS automatically retries failed copies and resumes from the last successful point, meeting the resiliency requirement. When the destination bucket is configured with a default customer-managed encryption key (CMEK) and the STS service agent is granted CryptoKey Encrypter/Decrypter access, all incoming objects are encrypted with that key. Approaches based on per-object Cloud Functions or a single gsutil rsync process do not scale for hundreds of terabytes, and Transfer Appliance is intended for offline bulk moves, not sub-hourly synchronization.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.