An online marketplace receives approximately 150 000 JSON click-stream events per second (about 10 TB/day). Analysts need sub-second, ad-hoc SQL analytics on two years of data, while data scientists occasionally train TensorFlow models that read the same dataset. The engineering team wants a fully managed, low-ops solution that automatically provides cheaper rates for rarely accessed historical data. Which Google Cloud storage approach best satisfies these requirements?
Write the events to Cloud Bigtable and export snapshots to Cloud Storage when analysts need to query historical data with Dataproc.
Ingest the events into partitioned tables in Cloud SQL for PostgreSQL and connect Data Studio for analyst queries.
Persist the raw JSON files in Cloud Storage Nearline and query them through Dataproc Spark SQL jobs scheduled by Cloud Composer.
Stream the events to BigQuery using the Storage Write API, store them in a time-partitioned and clustered native table, and rely on BigQuery's automatic long-term storage pricing for older partitions.
Streaming the events directly into a partitioned and clustered BigQuery native table gives analysts the interactive SQL experience they expect, and BigQuery's managed storage tier automatically moves partitions older than 90 days to long-term (lower-cost) storage without any operational work. The same table can be read by BigQuery ML or exported to Cloud Storage for TensorFlow training. Bigtable and Cloud SQL are optimized for operational workloads and lack interactive SQL analytics at this scale, while relying only on Cloud Storage would require additional systems (such as Dataproc) to achieve the desired query performance.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is the BigQuery Storage Write API?
Open an interactive chat with Bash
What are native tables in BigQuery, and how do time-partitioning and clustering work?
Open an interactive chat with Bash
What is BigQuery's automatic long-term storage pricing?
Open an interactive chat with Bash
What is BigQuery's Storage Write API?
Open an interactive chat with Bash
What are time-partitioned and clustered tables in BigQuery?
Open an interactive chat with Bash
What is BigQuery's automatic long-term storage pricing?
Open an interactive chat with Bash
GCP Professional Data Engineer
Storing the data
Your Score:
Report Issue
Bash, the Crucial Exams Chat Bot
AI Bot
Loading...
Loading...
Loading...
Pass with Confidence.
IT & Cybersecurity Package
You have hit the limits of our free tier, become a Premium Member today for unlimited access.
Military, Healthcare worker, Gov. employee or Teacher? See if you qualify for a Community Discount.
Monthly
$19.99
$19.99/mo
Billed monthly, Cancel any time.
3 Month Pass
$44.99
$14.99/mo
One time purchase of $44.99, Does not auto-renew.
MOST POPULAR
Annual Pass
$119.99
$9.99/mo
One time purchase of $119.99, Does not auto-renew.
BEST DEAL
Lifetime Pass
$189.99
One time purchase, Good for life.
What You Get
All IT & Cybersecurity Package plans include the following perks and exams .