AWS Certified Data Engineer Associate DEA-C01 Practice Question
An e-commerce company stores clickstream logs in Amazon S3 in the structure s3://logs/year=YYYY/month=MM/day=DD/. An Athena table is defined in the AWS Glue Data Catalog. New log files arrive every hour. The engineering team must make each hourly partition queryable in Athena within 5 minutes of arrival while minimizing the runtime and cost of Glue crawlers. Which solution meets these requirements?
Create an AWS Glue workflow that runs a crawler over only the last 24 hours of partitions once every hour.
Configure an AWS Glue crawler in incremental mode that is scheduled to run every 5 minutes over the entire S3 prefix.
Use Amazon S3 Event Notifications to invoke an AWS Lambda function that calls the Glue CreatePartition API to add the new partition when objects land.
Schedule an Athena MSCK REPAIR TABLE command to run every 5 minutes through an AWS Lambda function.
S3 can publish an event each time new objects are written to a prefix. A Lambda function invoked by that event can call the AWS Glue CreatePartition (or BatchCreatePartition) API to register the specific year-month-day partition that just received data. This requires no scan of existing data, makes the partition visible to Athena almost immediately, and avoids the per-DPU costs of frequently running a crawler over the whole dataset or the delays and full scans associated with MSCK REPAIR TABLE. Scheduling crawlers-even in incremental mode-every few minutes still incurs charges and unnecessary processing, so those alternatives are less cost-effective and may not meet the 5-minute SLA.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is the AWS Glue CreatePartition API?
Open an interactive chat with Bash
How do Amazon S3 Event Notifications work?
Open an interactive chat with Bash
Why is using a Glue crawler less cost-effective for this solution?
Open an interactive chat with Bash
What is the AWS Glue CreatePartition API used for?
Open an interactive chat with Bash
How does S3 Event Notifications work with Lambda?
Open an interactive chat with Bash
Why is using incremental crawlers or MSCK REPAIR TABLE less efficient for near real-time partitioning?
Open an interactive chat with Bash
AWS Certified Data Engineer Associate DEA-C01
Data Store Management
Your Score:
Report Issue
Bash, the Crucial Exams Chat Bot
AI Bot
Loading...
Loading...
Loading...
Pass with Confidence.
IT & Cybersecurity Package
You have hit the limits of our free tier, become a Premium Member today for unlimited access.
Military, Healthcare worker, Gov. employee or Teacher? See if you qualify for a Community Discount.
Monthly
$19.99
$19.99/mo
Billed monthly, Cancel any time.
3 Month Pass
$44.99
$14.99/mo
One time purchase of $44.99, Does not auto-renew.
MOST POPULAR
Annual Pass
$119.99
$9.99/mo
One time purchase of $119.99, Does not auto-renew.
BEST DEAL
Lifetime Pass
$189.99
One time purchase, Good for life.
What You Get
All IT & Cybersecurity Package plans include the following perks and exams .