AWS Certified Data Engineer Associate DEA-C01 Practice Question

A data engineering team runs a nightly Spark job on Amazon EMR that reads about 50 million 1-MB JSON files stored under the single prefix s3://logs/2024/05/ in an Amazon S3 bucket. Over the last week the job has started to fail with HTTP 503 SlowDown errors caused by a very high GET request rate on that prefix. The team wants to remove the throttling without changing the processing logic or adding new infrastructure. Which approach will best achieve this goal?

Enable S3 Transfer Acceleration on the bucket and keep the current key structure.
Increase the multipart download part size in the Spark job to 64 MB.
Move the data to the S3 Intelligent-Tiering storage class to gain higher throughput.
Rename the objects to include a two-character random hash prefix (for example, s3://logs/05/2024/05/…) and point the Spark job to the new prefixes.

Report Issue

Answer Description

Amazon S3 automatically shreds its key space into partitions. Each partition supports roughly 5,500 GET requests per second. When many objects share an identical prefix, all requests are routed to the same partition and can exceed its throughput limit, resulting in HTTP 503 SlowDown errors. Adding a short, random hash (for example, two hex characters) in front of every object key distributes the files across many prefixes, letting S3 spread the traffic across multiple partitions. This change requires only renaming the objects and updating the job's input path; no bucket-level settings, storage-class changes, or multipart tuning are needed. Transfer Acceleration optimizes cross-region data transfer speed, not request-rate limits. Switching to S3 Intelligent-Tiering affects cost, not throughput. Adjusting the multipart part size influences upload and download efficiency for large objects, but it does not mitigate per-prefix request throttling for millions of small files.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.