AWS Certified Data Engineer Associate DEA-C01 Practice Question

An AWS data engineering team operates multiple Amazon EMR clusters that generate hundreds of gigabytes of Spark and YARN logs daily. Security analysts must retain these logs for 12 months and run ad-hoc SQL queries against them. The solution should minimize EMR cluster overhead and rely only on fully managed AWS services. Which architecture satisfies these requirements?

Install Filebeat on each EMR core node to ship logs to a self-managed Elasticsearch cluster on Amazon EC2; retain data for a year and analyze it with Kibana.
Enable EMRFS consistent view and write all logs directly to HDFS; periodically run Hive queries on the cluster to analyze the logs and copy results to Amazon S3 for retention.
Send EMR application and system logs to Amazon CloudWatch Logs, use a subscription filter to stream the logs through Amazon Kinesis Data Firehose into an Amazon S3 data lake bucket, catalog the data with AWS Glue, and query with Amazon Athena.
Store logs on local disks and use a cron job to transfer them to an on-premises Hadoop cluster for year-long retention and analysis with Presto.

Report Issue

Answer Description

Streaming the EMR application logs to Amazon CloudWatch Logs removes log processing from the clusters. A CloudWatch Logs subscription can forward the data to Amazon Kinesis Data Firehose, which automatically buffers, compresses, and delivers the logs to an Amazon S3 bucket. After the data is cataloged with AWS Glue, the analysts can issue ad-hoc SQL queries in Amazon Athena. All services used are fully managed, and storage in S3 meets the one-year retention requirement.

Keeping logs in HDFS and querying them on the running clusters increases cluster load and requires the clusters to stay online. Shipping logs to an on-premises Hadoop environment or running a self-managed Elasticsearch deployment on Amazon EC2 introduces additional infrastructure to provision and maintain, violating the requirement to rely only on managed AWS services.

Ask Bash

Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.