AWS Certified Data Engineer Associate Practice Test (DEA-C01)

S3DistCp is an Amazon EMR utility built on Apache DistCp that runs as a step on the cluster. It launches multiple mapper tasks that copy objects in parallel, optionally combines small files, and uses the cluster's network bandwidth instead of a single node. This approach delivers the highest throughput while reducing the number of GET requests. Running the command from the master node (aws s3 cp) would be single-threaded, DataSync adds an external service and cannot write directly into HDFS, and mounting the bucket with s3fs provides no parallelism and incurs high per-object overhead.

Amazon Redshift federated queries let you create an external schema that maps directly to tables in Amazon Aurora PostgreSQL and Amazon RDS. When analysts run a query in Redshift, the service automatically pushes down predicates to the remote database and streams only the required rows, so no periodic ETL jobs or data replication pipelines are needed. Redshift Spectrum is designed for data in Amazon S3, not for live Aurora tables. AWS Glue ETL jobs or AWS DMS change-data-capture tasks would satisfy the requirement but require building and maintaining pipelines that copy data into Redshift, introducing additional latency and operational overhead.

AWS Glue is a serverless ETL service; you pay only while the job runs and do not manage clusters. A Glue crawler can infer the schema of the incoming CSV files and store it in the Data Catalog. A Glue Spark job can then read the CSV data from the source prefix, write compressed Parquet files partitioned by event_date to a separate S3 prefix, and can be placed on an hourly schedule so new files are converted as they arrive. Athena automatically benefits because it can query the partitioned Parquet files with far less data scanned, lowering query costs and improving performance.

Launching an EMR cluster or Redshift cluster introduces persistent infrastructure you must configure and pay for even when idle, making them less cost-effective for this workload. S3 Storage Lens and S3 Lifecycle policies reduce storage cost but do not transform file formats. Writing Parquet directly from the source application eliminates the need for conversion but requires changing the upstream producer, which is outside the stated scope.

Columnar formats such as Apache Parquet store values together by column rather than by row, so Athena can read only the columns referenced in a query instead of every field in every record. Snappy compression further reduces the amount of data stored and scanned without adding excessive CPU overhead. Adding a partition key such as transaction_date lets Athena read only the partitions that match a predicate, which sharply limits the amount of data that must be scanned each day. Compressing CSV, combining files, or simply splitting them into smaller objects still forces Athena to read every column of every row, so they do not significantly reduce scan costs or latency for large structured datasets.

When a Spark job must open and schedule tens of thousands of very small objects, task-startup overhead, network calls, and driver pressure dominate run time even though CPU usage is low. AWS Glue lets you reduce that overhead by grouping files as they are read. Setting the S3 connection options "groupFiles" to "inPartition" and specifying an appropriate "groupSize" causes the Glue library to combine many small objects into larger logical partitions before they reach the executors, decreasing the number of tasks that must be scheduled and allowing each task to perform more useful work. Because this change does not request additional DPUs, cost remains the same.

• Increasing executor memory does not attack the scheduling overhead that is the primary bottleneck, and Glue fixes executor memory per DPU.
• Changing the Parquet compression codec affects the write phase, not the excessive read-side task creation.
• Job bookmarking only helps skip files that were processed in earlier runs; it does not speed up processing of the current data set.

AWS Glue crawlers keep track of the folders they have already processed. Setting the crawler's recrawl policy to CRAWL_NEW_FOLDERS_ONLY turns the crawler into an incremental crawler: on each run it compares the current S3 prefix to its internal state and inspects only folders that appeared since the previous crawl. Existing partitions and their schemas are left untouched, so the crawler finishes quickly while still creating or updating the catalog entry for the newest date partition. The other options either continue to scan all folders, rely on S3 event notifications that are not configured, or require changing the folder naming convention used by the ingestion jobs.

AWS Secrets Manager can create a managed secret for an Amazon Redshift cluster whose password is rotated automatically every 30 days. An AWS Glue connection can reference the secret's ARN, so the job continues to run without code changes; the only additional step is to grant the Glue job role permission to call secretsmanager:GetSecretValue. Systems Manager Parameter Store has no built-in rotation, encrypting environment variables with KMS rotates keys rather than credentials, and AWS Glue connections do not provide automatic credential rotation. Therefore the Secrets Manager approach is the only option that satisfies all stated requirements.

AWS CodePipeline can orchestrate the entire release with almost no custom code. A source stage detects new commits in CodeCommit, a build stage (CodeBuild) packages the script and places the artifact in Amazon S3, a deploy stage updates the development Glue job by applying a CloudFormation stack update, a manual approval stage enforces the manager sign-off, and a final deploy stage updates the production Glue job.

Using EventBridge and Glue workflows would automate job execution rather than deployment and provides no approval gate. Triggering Lambda from S3 demands custom code for packaging, promotion logic, and approvals. CodeDeploy supports EC2, Lambda, and Amazon ECS targets-not AWS Glue-so additional wrappers would be necessary. Therefore, the CodePipeline solution offers the lowest operational overhead and the fewest components to maintain.

Amazon Redshift is a managed, petabyte-scale columnar data warehouse purpose-built for complex analytic SQL workloads. RA3 nodes let you separate compute from storage so hundreds of terabytes can be kept at low cost, while Concurrency Scaling automatically adds transient capacity to serve hundreds of simultaneous users with sub-second latency. The COPY command can stream data into staging tables without blocking queries. Amazon RDS for PostgreSQL is limited to 64 TB and becomes I/O-constrained under heavy analytic concurrency. DynamoDB is a NoSQL key-value store; large joins and ad-hoc SQL analytics are inefficient. Apache Hive on EMR performs well for batch processing, but interactive queries across 100 TB typically take minutes and do not scale to hundreds of concurrent dashboard users.

Specifying an Amazon S3 log URI causes Amazon EMR to copy YARN, Spark, and system logs to the bucket as each step completes. Enabling the persistent application user interfaces feature stores Spark History Server data off-cluster in the same S3 location, so the UI can be opened from the Amazon EMR console even after the cluster terminates. This preserves all troubleshooting information without incurring additional EC2 costs. Leaving the cluster up with termination protection meets the troubleshooting goal but accrues ongoing instance charges. EMRFS Consistent View only improves S3 list consistency and does not archive logs. Enabling CloudTrail data events on the input bucket records object-level API activity, not Spark application logs, so it does not provide the required diagnostics.

The AWS Glue Data Catalog is a fully managed, serverless metastore used by Amazon Athena by default and can also be configured as the Hive metastore for Amazon EMR. Pointing both Athena and EMR to the same Glue Data Catalog gives all services a consistent view of table definitions without running or maintaining additional infrastructure. Storing metadata in DynamoDB would require custom integration logic. External schemas in Amazon Redshift do not act as a central Hive metastore. Running a self-managed Hive metastore on the EMR primary node introduces operational overhead and Athena cannot natively query it without Glue.

AWS Glue interactive sessions let a SageMaker Studio notebook start a temporary, serverless Spark environment by selecting the Glue PySpark kernel. The session starts in seconds, bills by the second for DPUs that are actually used, and shuts down automatically when idle, satisfying the no-persistent-cluster policy. An EMR cluster or self-managed EC2 instance requires manual provisioning and ongoing management. Standard Athena SQL cannot easily perform row-level data cleansing for malformed records, and Athena for Apache Spark notebooks are only available in the Athena console, not from a SageMaker Studio Jupyter environment.

Enabling partition projection on the AWS Glue Data Catalog table lets Athena derive partition values from the S3 prefix pattern at query runtime. No crawler, DDL statement, or Lambda function is required, so new partitions are available immediately with virtually no additional cost. Running a Glue crawler every few minutes or invoking BatchCreatePartition on every object adds unnecessary expense. Requiring analysts to run MSCK REPAIR TABLE introduces manual effort and delays.

An organization event data store in AWS CloudTrail Lake automatically ingests management events from every account when created by the management or delegated administrator account. Event data stores are immutable collections, and the one-year extendable retention option meets the 365-day requirement without additional storage configuration. CloudTrail Lake provides a built-in SQL interface, so analysts can query the data directly-no Object Lock configuration, Glue cataloging, or streaming pipelines are needed. The S3/Object Lock and Security Lake options satisfy immutability but add Glue/Athena setup and data-movement overhead. Streaming logs to CloudWatch Logs and OpenSearch requires per-account trail configuration, subscription filters, and OpenSearch management, increasing operational burden and not guaranteeing immutability.

Amazon Redshift supports native streaming ingestion from Amazon Kinesis Data Streams and Amazon MSK. By creating a materialized view that references the stream with the KINESIS clause and enabling auto-refresh, Redshift consumes records directly and makes them available for queries in seconds. This eliminates the S3 staging layer used by Kinesis Data Firehose, avoids the operational overhead of managing AWS Glue or Lambda jobs, and incurs no additional service charges beyond Redshift and the existing Kinesis stream.

RecrawlPolicy with RecrawlBehavior = CRAWL_EVENT_MODE enables the crawler to use Amazon S3 event notifications so each run lists only the folders mentioned in new PUT or DELETE events, giving fast incremental crawls. Setting SchemaChangePolicy DeleteBehavior = DELETE_FROM_DATABASE tells the crawler to drop the partition from the Glue Data Catalog when an object-removal event indicates that the underlying S3 folder no longer exists. This combination therefore (1) registers new YYYY/MM/DD folders as partitions, (2) removes partitions whose folders are deleted, and (3) avoids rereading data that hasn't changed.

CRAWL_NEW_FOLDERS_ONLY cannot delete partitions because the service forces DeleteBehavior to LOG. CRAWL_EVERYTHING with DELETE_FROM_DATABASE does remove partitions, but it must list the entire dataset on every run, increasing runtime and cost. Running a full crawl nightly or recreating the table each day also rescans all data and is unnecessary.

Athena partition projection lets you define the partition keys (year, month, day) as template variables. Athena then resolves the partitions at query time instead of reading them from the AWS Glue Data Catalog. Once projection is configured, the table no longer needs explicit partition objects, so the daily crawler can be disabled, eliminating both the crawl time and the related cost.

Using an S3 event-driven AWS Glue job or Lambda function would still require authoring and maintaining custom code. Setting the crawler to recrawl only new folders reduces, but does not eliminate, the growing scan cost. Glue partition indexes accelerate certain lookups but do not shorten crawler runtime or remove the need to maintain partitions.

Attaching a custom endpoint policy to the S3 interface endpoint restricts the actions that can be performed through that endpoint only. By allowing s3:GetObject and s3:ListBucket on arn:aws:s3:::dept-finance/raw/* and the bucket respectively, and denying other S3 permissions, the Redshift Spectrum query is limited to the required prefix. Redshift Spectrum requires both ListBucket and GetObject permissions to function. Other workloads that reach S3 through public endpoints or a different gateway are not affected because the endpoint policy is evaluated only when the interface endpoint is used. A bucket policy would impact every caller, changing the behavior for other workloads. Replacing the endpoint is unnecessary and costly. Changing only the Redshift role is a less secure option because the endpoint policy creates a network-level boundary that cannot be bypassed, even by a principal with overly permissive IAM credentials.

A sparse graph with far fewer edges than vertices-squared is most memory-efficient when stored as an adjacency list. Implementing the list as a single dictionary whose keys are task IDs and whose values are Python lists of neighboring task IDs requires O(V+E) space-about 10,000 keys and at most 300,000 integers-well within the 512 MB limit. Depth-first or Kahn topological sorting can traverse this structure in O(V+E) time. An adjacency matrix allocates O(V²) space (≈100 million booleans) and would exceed memory. A list of JSON edge objects or a nested dict-of-dicts adds heavy per-edge object overhead, wasting memory without improving traversal speed.

A JDBC-type AWS Glue connection is required for a crawler that targets an RDS database. By selecting the same VPC, one of the private subnets that already hosts the DB instance, and a security group that permits inbound TCP 3306 traffic from the Glue-managed ENIs, the service can place ENIs inside that subnet and reach the database directly. Storing the credentials in AWS Secrets Manager lets the crawler authenticate without hard-coding passwords. Because the connection remains inside the VPC, no internet gateway or NAT device is needed. The other options either place the database in a public subnet, rely on a NAT gateway, or assume that no connection is required, all of which conflict with best practices for a private RDS instance.