GCP Professional Data Engineer Practice Question

A retailer is migrating its on-premises Hadoop environment to Google Cloud. The data engineering team will run Spark Structured Streaming jobs that ingest inventory events 24×7 and must keep end-to-end latency under one minute. During the day, analysts connect with Hive to perform interactive, ad-hoc queries against the same datasets. The team needs the flexibility to install custom Hadoop libraries and is comfortable paying for always-on capacity to avoid startup delays. Which Dataproc deployment model best satisfies these requirements?

Maintain a single persistent Dataproc cluster that runs the streaming Spark jobs continuously and allows analysts to submit interactive Hive queries on demand.
Store data in BigQuery and replace Spark streaming with scheduled BigQuery queries, eliminating the need for any Dataproc cluster.
Schedule Cloud Composer to launch Dataproc Serverless Spark batches for both streaming ingestion and analyst queries.
Use a Dataproc workflow template that creates an ephemeral cluster for each Spark streaming job and deletes it when the job finishes.

Report Issue

Answer Description

Persistent Dataproc clusters are designed for long-running, stateful workloads that must remain available, such as continuous Spark or Flink streaming pipelines and interactive SQL engines (Hive, Presto, Spark-SQL). Because the cluster and its YARN or Kubernetes schedulers stay up, streaming jobs maintain low, predictable latency and analysts can connect at any time without the overhead of cluster creation. Ephemeral, job-scoped clusters, serverless Spark batches, or eliminating the cluster entirely favor cost-optimized batch use cases but introduce startup delay, restrict custom library installation, or remove interactive capabilities-none of which meet the stated requirements.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.