AWS Certified Data Engineer Associate DEA-C01 Practice Question

Your team launches a transient Amazon EMR cluster nightly to run Spark jobs that turn 50 TB of raw data in Amazon S3 into optimized Parquet files for Athena. The output must persist in S3. Intermediate shuffle and spill data are not required after the job ends. What is the most cost-effective storage configuration that provides high performance and meets these requirements?

Mount an Amazon EFS file system on the cluster and direct Spark to read, write, and spill to the EFS mount; leave the cluster running for reuse.
Configure EMRFS to read from and write to Amazon S3, and let Spark use instance-store HDFS volumes for intermediate data; terminate the cluster after completion.
Enable Spark DirectOutputCommitter v2 to write shuffle and output files directly to S3, and disable local storage on all nodes.
Attach multiple Amazon EBS volumes to each core node, set HDFS replication to three for all data, and copy results to S3 with DistCp after the job.

Report Issue

Answer Description

Using EMRFS lets the cluster read raw data from and write final Parquet files back to Amazon S3 without additional copy steps. Keeping shuffle and spill files on the instances' local HDFS (backed by ephemeral instance-store disks that are provisioned with the EC2 nodes) offers high-throughput, low-latency storage that Spark needs during execution. Because the cluster is terminated when processing finishes, the transient local storage is discarded, avoiding ongoing cost. Alternatives that replicate data on EBS, store all files on EFS, or force Spark to write shuffle data directly to S3 either add storage charges, add operational steps, or degrade Spark performance, so they are less cost-effective for this workload.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.