AWS Certified Data Engineer Associate DEA-C01 Practice Question

A data engineering team runs a persistent Amazon EMR cluster that stores intermediate data in HDFS. Each night, about 50 TB of gzip log files arrive in an Amazon S3 bucket and must be copied into HDFS before downstream MapReduce jobs start. The transfer must maximize throughput, minimize S3 request costs, and run by using only the existing EMR cluster resources. Which solution meets these requirements?

Use AWS DataSync to transfer the objects to volumes on each core node, then import the data into HDFS.
Add an EMR step that uses S3DistCp to copy the objects from Amazon S3 to HDFS in parallel.
Mount the S3 bucket on every core node with s3fs and move the objects to HDFS with the Linux cp command.
From the master node, run the AWS CLI command "aws s3 cp --recursive" to copy the objects into HDFS.

Report Issue

Answer Description

S3DistCp is an Amazon EMR utility built on Apache DistCp that runs as a step on the cluster. It launches multiple mapper tasks that copy objects in parallel, optionally combines small files, and uses the cluster's network bandwidth instead of a single node. This approach delivers the highest throughput while reducing the number of GET requests. Running the command from the master node (aws s3 cp) would be single-threaded, DataSync adds an external service and cannot write directly into HDFS, and mounting the bucket with s3fs provides no parallelism and incurs high per-object overhead.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.