AWS Certified Data Engineer Associate Practice Test (DEA-C01)

Amazon Redshift supports native streaming ingestion from Amazon Kinesis Data Streams and Amazon MSK. By creating a materialized view that references the stream with the KINESIS clause and enabling auto-refresh, Redshift consumes records directly and makes them available for queries in seconds. This eliminates the S3 staging layer used by Kinesis Data Firehose, avoids the operational overhead of managing AWS Glue or Lambda jobs, and incurs no additional service charges beyond Redshift and the existing Kinesis stream.

Create an Amazon Macie sensitive-data discovery job for the lake buckets. Configure an Amazon EventBridge rule that triggers an AWS Step Functions state machine whenever Macie publishes a sensitive-data finding. In the workflow, use an AWS SDK task to call the Lake Formation AddLFTagsToResource API and attach an LF-tag such as Classification=Sensitive to the object (or its corresponding catalog columns). Lake Formation tag-based access-control policies then deny the analyst principal and allow the governance principal for resources tagged Classification=Sensitive. This uses only managed integrations (Macie, EventBridge, Step Functions, Lake Formation) and requires minimal code-no bespoke parsing beyond the workflow definition.

The other options either rely on custom parsing inside Lambda, do not use Macie for detection, or cannot apply Lake Formation permissions automatically.

AWS Glue provides a serverless Spark runtime, so no clusters need to be provisioned or maintained. The script can be referenced directly from Amazon S3, and the Jenkins pipeline can start each run by calling the AWS Glue StartJobRun API. Each run automatically exposes the Spark UI and executor logs in the AWS Glue console, which allows engineers to inspect stages, tasks, and metrics for troubleshooting. Using an EMR cluster would meet the functional need but requires provisioning, configuring, and terminating infrastructure. Athena and AWS Batch do not provide the full Spark UI, making them less suitable for the stated troubleshooting requirement.

CPU throttling occurs when a container exhausts the CPU limit defined in its pod specification. For Spark workloads this causes tasks to pause even though idle CPU is still available on the node. Removing the CPU limit (or increasing it well above the request) lets the executor containers borrow spare vCPUs and eliminates cgroup throttling, reducing job runtime without needing additional EC2 capacity. Increasing disk throughput, enabling dynamic allocation, or migrating to a different service may improve performance in some cases but do not directly address the throttling caused by the Kubernetes CPU limit in this scenario.

Creating an AWS Glue JDBC connection to the RDS instance keeps network traffic inside the VPC and removes the need to manage custom drivers or endpoints. A crawler that uses this connection can catalog the PostgreSQL table in the AWS Glue Data Catalog. The Spark job can then read both the Parquet dataset and the cataloged JDBC table through the same catalog, allowing other Glue or EMR jobs to reuse the metadata. Exporting to S3, using Athena federation, or replicating into DynamoDB adds extra components, increases management overhead, or changes the data store, so they do not best satisfy the stated constraints.

Amazon Redshift Serverless eliminates the need to size, scale, or turn on and off a cluster. You can restore the existing cluster snapshot into a serverless workgroup, so the current schema and SQL continue to run unchanged. Because compute capacity automatically spins up only when queries arrive and pauses when idle, the company pays for RPU-seconds consumed instead of for a constantly running cluster. Athena would avoid cluster management but would require moving data to open-format tables and adapting queries that rely on Redshift-specific functions. An EMR on-demand cluster still needs provisioning and administration. Switching to RA3 nodes with concurrency scaling retains fixed baseline node costs, so idle time would still accrue charges. Therefore, Redshift Serverless is the most cost-effective and fully managed choice.

Streaming each EMR cluster's application and step logs to Amazon CloudWatch Logs is fully managed and eliminates node-level agents. A CloudWatch Logs subscription can forward the data to an Amazon OpenSearch Service domain, which automatically indexes the events so engineers can query them and build dashboards in OpenSearch Dashboards. Retention policies on the CloudWatch log groups enforce the 30-day requirement. The other options either require self-managed analytics infrastructure, lack near-real-time search, or add unnecessary data-flow components.

Lake Formation LF-tags let administrators assign business-defined classifications (for example, Public, Internal, PII) to databases, tables, and even individual columns in the AWS Glue Data Catalog. Tag-based access control policies can then be granted to roles or users, and Athena automatically honors those permissions. Custom Glue classifiers only identify file formats, separate catalogs add operational overhead, and Amazon Macie does not provide fine-grained permission enforcement for Athena queries.

AWS Step Functions is a serverless, fully managed workflow engine defined in JSON (the Amazon States Language). It offers visual execution graphs in the console, integrated CloudWatch metrics, and configurable retry and error-handling policies. Step Functions provides direct service integrations to invoke AWS Glue StartJobRun, run Amazon Athena StartQueryExecution, and publish Amazon SNS messages, so no custom code is necessary. EventBridge Scheduler supports only a single target per schedule, so it cannot chain the three required steps. Amazon MWAA could orchestrate the tasks but introduces additional effort to provision and maintain an Airflow environment. AWS Glue Workflows can coordinate Glue jobs and crawlers but cannot directly invoke Athena queries or publish SNS notifications without additional Lambda code. Therefore, Step Functions best meets all stated requirements.

The correct action is to increase the ParallelizationFactor. Lambda polls each Kinesis shard and, by default, invokes only one concurrent function per shard. When IteratorAge is high despite fast function execution, the bottleneck is the rate of processing. Increasing the ParallelizationFactor from the default of 1 (up to 10) allows Lambda to process multiple batches from each shard concurrently, which directly increases throughput and reduces IteratorAge. Reducing the BatchSize would trigger more frequent, smaller invocations but would not increase the number of concurrent executions per shard, so it would not solve the throughput bottleneck. While Lambda can be configured as an enhanced fan-out (EFO) consumer, EFO's main benefit is providing dedicated throughput to each consumer, which is most useful when multiple applications are reading from the same stream. For a single consumer, increasing the ParallelizationFactor is a more direct and simpler solution. Increasing the Lambda function's memory would not help, because the function's 200 ms runtime is already very low, indicating that compute resources are not the limiting factor.

Attribute-based access control (ABAC) allows a single IAM policy to enforce that the Team tag on the calling principal matches the Team tag on the S3 object. A condition such as "s3:ResourceTag/Team == aws:PrincipalTag/Team" dynamically authorizes access whenever matching tags are present, so new teams can be onboarded simply by tagging roles and objects. Creating separate roles or managed policies, maintaining ACLs, or configuring individual S3 Access Points would all require ongoing manual updates and therefore do not meet the stated goal.

Using SSE-KMS with a customer-managed key satisfies the requirement for encryption at rest while avoiding the default aws/s3 key. Setting bucket default encryption to that customer-managed key ensures every object written by the Glue job is encrypted without code changes. The key policy for the customer-managed key can grant decrypt permission to the external AWS accounts, and the bucket policy grants object access, so no per-object ACLs or manual key distribution are needed. SSE-S3 lacks customer control of keys. Client-side encryption adds significant key-management overhead. The aws/s3 managed key is explicitly disallowed by the security team and cannot be shared cross-account directly.

Creating a materialized view lets Amazon Redshift store the pre-computed, aggregated result set on disk. When analysts query the materialized view, Redshift returns the stored result almost immediately instead of re-scanning and joining the 5-TB fact table, yielding a large runtime reduction. Scheduling an automatic refresh immediately after the nightly data load maintains accuracy while requiring minimal ongoing management.

Changing the fact table to an ALL distribution style would duplicate terabytes of data across every node, greatly increasing storage space and load time. Concurrency scaling adds transient clusters to improve throughput when many queries run simultaneously, but it seldom reduces the elapsed time of a single long query. Adjusting WLM queues or enabling short query acceleration allocates resources differently but will not eliminate the heavy table scan and aggregation work that dominates the query's runtime.

Amazon S3 is an object store that delivers 11 nines of durability, scales without user intervention, and is the native storage layer for both Amazon EMR and Amazon Athena. Because objects are infrequently accessed after upload, an S3 Lifecycle rule can transition them to a lower-cost storage class such as S3 Glacier Instant Retrieval to reduce cost while still allowing occasional access.

Amazon EFS provides POSIX file storage but does not integrate directly with Athena and is generally more expensive for large, rarely accessed datasets. Loading binary images into Amazon Redshift is inefficient and costly because Redshift is optimized for structured, columnar data, not large unstructured files. DynamoDB cannot store multi-megabyte images because each item is limited to 400 KB and would still require another service for Athena queries. Therefore, storing the data in Amazon S3 with an appropriate Lifecycle policy is the most suitable choice.

Amazon Kinesis Data Streams is designed for high-throughput, low-latency streaming ingestion. Each shard supports up to 1 MB/s or 1,000 records per second, and records are typically readable by consumers in less than one second. Provisioning 20 shards therefore supplies 20 MB/s of write capacity while meeting the sub-second availability goal.

Amazon Data Firehose (formerly Kinesis Data Firehose) adds a delivery buffer-300 s by default for S3 and 0-60 s for most other destinations. Even with the new zero-buffering mode, AWS states that most deliveries occur within about five seconds, so it cannot guarantee availability within a few hundred milliseconds. Writing individual objects directly to Amazon S3 and reacting with S3 event notifications typically introduces seconds or longer of lag because notifications are usually delivered in seconds and can occasionally take a minute or more. Amazon EventBridge charges $1 per million events, has default PutEvents rate limits of 600-10,000 TPS (Region-dependent), and therefore costs significantly more than a provisioned Kinesis stream sized for this workload. For these reasons, a properly sized Kinesis Data Streams solution is the most cost-effective way to meet both throughput and latency requirements.

AWS Lake Formation integrates with Amazon Athena and supports fine-grained permissions down to the column level. By registering the S3 location in Lake Formation, adding the tables to the AWS Glue Data Catalog, and using LF-Tags to identify PII columns, administrators can grant analysts SELECT access to the table while explicitly denying access to columns tagged as PII. Because the data stays in place and permissions are enforced at query time, no additional copies of the dataset or ongoing ETL jobs are required.

Running Amazon Macie and moving objects to a different bucket protects data but introduces daily jobs and creates multiple data copies. Encrypting PII client-side still exposes ciphertext to analysts, does not prevent access, and requires application changes. Creating a separate table without PII columns duplicates data and adds maintenance overhead. Therefore, using Lake Formation column-level security is the most efficient and compliant solution.

Amazon MWAA exposes the standard Airflow connection named "aws_default". By editing this connection in the MWAA console (or with Airflow CLI) and adding the ARN of an IAM role in the role_arn extra field, Airflow's AWSHook automatically calls AWS STS to assume that role. All built-in AWS operators and any custom code that relies on AWSHook or Boto3 inherit those temporary credentials, so no DAG code changes are needed.

Building a custom container image is not supported by MWAA. Adding static credentials directly in a connection violates security best practices and still leaves the execution role in use for other hooks. Wrapping every operator with Lambda vastly increases code and operational overhead. Therefore, updating the existing aws_default connection with the required role_arn is the simplest and most compliant approach.

AWS Glue job bookmarks record the state of the input data that a job has already processed. When bookmarks are enabled, the job automatically skips files it has successfully loaded in earlier runs, preventing the same records from being appended twice. Overwrite mode could remove duplicates but risks data loss if the job fails midway and is less efficient. Deleting landing files with a lifecycle rule still lets duplicates through if the job re-reads already copied data before deletion, and it provides no guard against partial reruns. Adding a Step Functions task to run an Athena DELETE query introduces extra cost and complexity and only corrects duplicates after they occur rather than preventing them.

An Amazon EventBridge rule can use a cron expression to run on an exact hourly schedule limited to specific days and times. The rule invokes an AWS Lambda function, which is serverless and priced per request. The function can use an IAM role that has cross-account S3 permissions to list and copy only new objects. This pattern requires no provisioned infrastructure and schedule changes are made by simply updating the EventBridge rule.

Running Apache Airflow on Amazon MWAA introduces continuous environment costs and additional operational overhead. Maintaining an EC2 instance with a cron job violates the serverless constraint and adds management responsibilities. Scheduling an AWS Glue crawler and job is possible but is more expensive and unnecessary for a straightforward file copy operation, making it less cost-effective than the EventBridge-and-Lambda combination.

AWS Glue DataBrew includes a built-in HMAC-SHA256 transformation that hashes a column by combining the field value with a secret key. The key can be stored in AWS Secrets Manager and referenced directly from the recipe. The result is a consistent, non-reversible pseudonym that supports deterministic joins. S3 bucket keys, SSE-C, or Lambda code either do not perform deterministic hashing or require custom code and state management, making them less suitable.