GCP Professional Data Engineer Practice Question

Your analytics team processes 3 PB of application logs that arrive via Pub/Sub, and a Dataflow pipeline writes the raw events as Avro files. Compliance rules mandate that each file be retained unaltered for seven years, yet engineers access the logs only a few times after the first week. Analysts must be able to run occasional ad-hoc SQL queries from BigQuery without first loading the data into native tables. Which Google Cloud storage approach will minimize total cost while satisfying durability, retention, and query needs?

Write the logs to an AlloyDB for PostgreSQL instance using compressed columnar storage and grant BigQuery federated access.
Keep the Avro files in a Cloud Storage bucket configured with the Archive storage class via lifecycle rules, and define BigQuery external tables to query them in place.
Persist the logs in a regional Cloud Bigtable cluster and export snapshots to Cloud Storage once per quarter.
Load the Avro files into partitioned BigQuery tables and rely on BigQuery's long-term storage pricing to reduce costs.

Report Issue

Answer Description

Placing the Avro files in a Cloud Storage bucket that uses the Archive storage class keeps costs as low as possible for data that must be stored at least 365 days and is rarely accessed. Archive offers the same high durability and sub-second access latency as other Cloud Storage classes but at the lowest per-GB price; lifecycle rules can move objects from Standard (for the initial week of frequent use) to Archive automatically. BigQuery can define an external table over the Avro objects in Cloud Storage, allowing analysts to query the data on demand without ingest costs or duplicating storage. Storing the data inside BigQuery tables, a Bigtable cluster, or an AlloyDB instance would incur significantly higher ongoing storage costs and provide capabilities (high-performance scans or transactions) that the workload does not require.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.