AWS Certified Data Engineer Associate DEA-C01 Practice Question

A media company performs a weekly Spark ETL job on 40 TB of log files stored in Amazon S3. Intermediate shuffle data is needed only while the job runs; after the job finishes, the cluster is terminated. The team wants to minimize storage costs yet maintain high I/O throughput for the shuffle phase. Which solution meets these requirements?

Copy the data into an Amazon EFS file system and run the ETL using Spark containers on AWS Fargate.
Launch an Amazon EMR cluster that uses EC2 instances with locally attached NVMe instance store volumes as HDFS storage, and avoid provisioning additional EBS volumes.
Create an Amazon EMR cluster and attach gp3 EBS volumes sized to store the entire 40 TB dataset; enable EMRFS consistent view for metadata operations.
Load the data into an Amazon Redshift RA3 cluster with managed storage and perform the transformation using Redshift Spectrum.

Report Issue

Answer Description

Locally attached instance store volumes are included in the price of the EC2 instances that make up the EMR cluster, so no additional storage cost is incurred. Because instance store is directly attached to the host, it delivers the lowest-latency, highest-throughput storage available to the cluster, which is ideal for temporary Spark shuffle files. EBS, EFS, or Redshift all introduce extra per-GB charges and additional network hops, making them more expensive and potentially slower for this workload. Therefore, configuring the EMR cluster to use only instance store for HDFS provides the required performance at the lowest cost.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.