AWS Certified Data Engineer Associate Practice Test (DEA-C01)

AWS Step Functions is purpose-built for orchestrating serverless workflows. It offers native integrations with both AWS Lambda and AWS Glue jobs, records a complete execution history, and supports configurable retry logic or catch paths for each state, allowing failed executions to be re-started at the failed step. An EventBridge rule can invoke the state machine when new S3 objects arrive, so no custom scheduling code is needed. EventBridge Pipes alone is an event routing service; the first Lambda would need to manage Glue and Redshift steps as well as error handling in code. Glue Workflows orchestrate only Glue jobs, crawlers, and triggers; adding Lambda requires workarounds such as Python shell jobs and still lacks cross-service state tracking. MWAA can meet the functional need but introduces continuous environment costs and additional operational overhead, making it less cost-effective than the fully managed Step Functions solution.

The correct query filters on the exact error text, selects only the required fields (message and jobRunId), and then uses count_distinct(jobRunId) to report the number of affected job runs in a single pass. Alternative queries either count every matching event instead of distinct jobRunId values, apply sort or limit before aggregation (which can omit relevant events and add processing overhead), or aggregate the wrong field.

Amazon Redshift is a fully managed, petabyte-scale data warehouse designed for analytic workloads. It stores data in columnar format, applies automatic compression, and distributes processing across nodes, allowing complex SQL queries with joins and aggregations to return in seconds. Amazon RDS is optimized for transactional workloads and can struggle with multi-terabyte analytics at this latency. DynamoDB is a NoSQL key-value store that does not support relational joins. An S3-based data lake managed through AWS Lake Formation provides durable storage but relies on external engines (for example, Amazon Athena) whose performance is typically slower for highly concurrent, complex relational queries compared with a dedicated data warehouse.

Amazon SageMaker ML Lineage Tracking is natively integrated with SageMaker Pipelines. When lineage tracking is enabled, each pipeline execution automatically records artifacts such as datasets, processing jobs, training jobs, models, and endpoints, and registers the relationships among them. These artifacts and their dependencies can be queried through the SageMaker Lineage API, used in automated compliance reports, or visualized in SageMaker Studio. AWS Glue crawlers catalog only data locations and cannot track transformations or model artifacts. CloudTrail logs must be parsed and correlated manually and do not provide semantic lineage. Step Functions with X-Ray trace requests but do not capture domain-specific ML artifacts without extensive custom instrumentation.

AWS Secrets Manager offers a built-in rotation feature for Amazon RDS databases. Enabling a rotation schedule creates an AWS-managed Lambda function that updates the database password and stores the new value in the same secret, eliminating the need for custom scripts. The Lambda functions in the pipeline can fetch the current password at run time by using an execution role that has secretsmanager:GetSecretValue permission.

AWS Systems Manager Parameter Store SecureString cannot rotate credentials automatically; implementing rotation would require a custom rule and script. Storing a KMS-encrypted value in Lambda environment variables still requires a manual update process, and Amazon RDS does not support a keyword such as AWS_ROTATE to trigger automatic password changes. Therefore, using AWS Secrets Manager with built-in rotation is the only option that satisfies the 30-day rotation requirement with the least operational effort.

A Map state natively iterates over a JSON array and can control parallelism with the MaxConcurrency field, ensuring that no more than the specified number of iterations (10) run simultaneously. Inside the Map state's Item processor, you can add a Task state that starts the Glue job, apply a Retry clause that specifies a maximum of two attempts with an exponential backoffRate, and add a Catch clause that matches the custom "DATA_VALIDATION_FAILED" error to fail the workflow immediately. This solution uses only Step Functions features and requires no additional Lambda code or complex branching logic.

Parallel states launch a fixed number of branches and therefore cannot dynamically scale to 200 partitions while still limiting concurrency to 10 without extra logic. Iterating in a Lambda function shifts the retry and concurrency control to application code, adding operational overhead. Express workflows cannot directly throttle concurrent Glue invocations and would still require external coordination.

AWS Glue Data Catalog provides a centralized Hive-compatible metastore that is natively supported by Athena, Redshift Spectrum, and EMR. Creating an AWS Glue crawler on the S3 prefix automatically infers the schema and adds or updates partitions on a schedule, so all three query engines can immediately consume the new data without additional DDL. Defining external tables separately, embedding schemas in application code, or storing inline schema files would require manual updates for every new partition and would not give the services a shared catalog.

An S3 Lifecycle configuration can natively expire both current and noncurrent object versions after a specified number of days. By adding separate expiration actions for current versions and for noncurrent versions set to 2,555 days, and enabling the option to remove expired object delete markers, every copy of the object is removed automatically when it reaches the mandated age. This solution is completely managed, requires no code to maintain, and produces an auditable lifecycle rule that demonstrates compliance.

S3 Object Lock prevents deletion until the retention period ends, but it does not automatically remove objects when the period expires; manual deletion would still be required. AWS Backup retention applies only to backup copies, not to the original S3 objects, so the source data would remain. A scheduled EventBridge-Lambda process could work, but it introduces custom code and ongoing operational overhead, making it less suitable than a native lifecycle policy.

AWS Glue records lineage information every time a supported crawler or ETL job runs. When Glue jobs read objects in S3 and write tables in Amazon Redshift, the Glue Data Catalog is updated with table- and column-level lineage that shows the relationship between the S3 sources and the Redshift targets. This information can be explored in the Glue console or queried through the Glue API, requiring no custom tagging or additional services. Adding custom tags (object metadata) or parsing CloudTrail logs would require manual maintenance and do not provide column-level lineage, while Amazon Inspector does not offer data lineage capabilities.

DynamoDB on-demand capacity mode charges only for the read and write requests that are actually made and can instantly scale to thousands of requests per second, so it handles sudden flash-sale bursts without advance capacity planning. Provisioned capacity with auto scaling can lag several minutes before increasing limits, risking throttling and unnecessary cost if pre-provisioned at peak. DAX accelerates reads, not writes, and DynamoDB TTL affects storage cost, not write throughput. Therefore, creating the table in on-demand capacity mode is the most cost-effective and operationally simple choice for this workload.

Bootstrap actions are executed on every node as the cluster is provisioning, making them ideal for installing additional software such as a JDBC driver before any jobs start. After the cluster is ready, an EMR step can invoke spark-submit to run a PySpark script that resides in Amazon S3. This combination uses built-in EMR scripting features, requires no custom AMI maintenance, and fits well into an automated Step Functions orchestration. Notebooks do not install software on all nodes automatically and are harder to schedule. Custom AMIs achieve the goal but add ongoing image-management overhead. Using Hadoop Streaming for software installation and Spark execution is possible but not intended for this scenario and complicates the workflow.

Using the Kubernetes Cluster Autoscaler with an EKS managed node group that contains a mix of On-Demand and Spot Instances allows the cluster to add capacity when Spark jobs need it and automatically scale in when demand falls. Defining CPU and memory requests for the Spark driver and executor pods gives the autoscaler the information it needs to schedule additional nodes only when required. This improves job throughput at night while avoiding unutilized compute during the day, and Spot Instances lower cost further. Moving all workloads to Fargate would simplify management but is typically more expensive for long-running, compute-intensive Spark jobs. Migrating to ECS changes the platform and offers no inherent cost benefit for Spark. Simply doubling the node group and relying on vertical pod autoscaling removes the performance bottleneck but increases costs and does not reclaim idle capacity.

An Amazon EventBridge rule can invoke an AWS Glue job on a cron schedule without provisioning or managing servers. Glue provides a native Redshift connection that issues a COPY operation under the hood, so the job can efficiently load the CSV file from S3 into the staging table. EventBridge and Glue are both serverless and incur charges only when rules fire or jobs run, meeting the cost-effectiveness and low-ops goals.

Launching an EMR cluster requires cluster provisioning, node management, and longer billing intervals. MWAA introduces additional infrastructure to manage and a higher baseline cost for the environment. Kinesis Data Firehose is optimized for continuous streaming, cannot be "scheduled" to start at a specific time, and would run (and bill) continuously, increasing cost and complexity.

Assuming an IAM role in account B provides temporary AWS Security Token Service (STS) credentials to the Glue job. Because the role is trusted by account A, the job can call STS to obtain short-lived keys whenever it runs, eliminating any need to embed or rotate long-term secrets. This is a role-based authentication mechanism. A pre-signed URL would require someone or some process to generate and inject the URL before each run. Storing permanent access keys in AWS Secrets Manager still relies on long-lived credentials that must be rotated. Amazon S3 does not support mutual TLS with customer-provided X.509 client certificates, so a certificate-based approach is not feasible.

AWS Secrets Manager encrypts secrets at rest with AWS KMS, integrates natively with IAM roles so the EMR instance profile can retrieve the secret with no code modification, and provides built-in automatic rotation through an AWS-managed schedule that invokes a Lambda function. Systems Manager Parameter Store SecureString parameters satisfy encryption and IAM integration but lack native rotation. Storing credentials in Amazon S3 or embedding them in bootstrap scripts requires manual rotation and increases the risk of exposure, so these options do not fulfill the security team's requirements.

Athena partitions are stored as separate folders in Amazon S3. When a query's WHERE clause references the partition columns, Athena prunes the unrelated partitions and reads only the relevant files, which reduces the amount of data scanned and lowers cost. Filtering solely on event_time does not use partition pruning because that column is stored inside the files, not in the partition path. A LIMIT clause, ORDER BY, or a common table expression does not affect how much data is read from S3, so they provide no scan-cost benefit.

Using Amazon API Gateway with an attached usage plan lets the company require an API key for every request. A Lambda function behind the API runs simple SELECT statements by calling the Amazon Redshift Data API, formats the result as JSON, and returns it. All components are serverless, no network endpoints for Redshift are exposed, and API Gateway handles throttling and key management. Directly exposing the Redshift Data API would require the partner to sign requests with AWS credentials. Running a container service behind an Application Load Balancer introduces additional infrastructure to operate. Publishing a daily file to Amazon S3 does not provide on-demand queries and relies on presigned URLs rather than API-key authentication.

An AWS Organizations service control policy (SCP) is evaluated before IAM policies in every member account, so an explicit Deny cannot be overridden. A Deny statement that fires when aws:RequestedRegion is not "us-east-2" blocks any API call aimed at another Region. Adding conditions such as s3:LocationConstraint and denying critical calls like s3:PutBucketReplication, s3:CreateBucket, and backup:StartCopyJob ensures engineers cannot create resources or start copy jobs that would place data outside the permitted Region. Because the SCP is attached to the organizational unit, it automatically applies to new accounts, buckets, and backup plans with no further action.

The Same-Region Replication answer relies on every bucket and backup plan being configured correctly and could be changed by developers. Restricting VPC interface endpoints only limits private network access; S3 replication and AWS Backup can use public endpoints that would still succeed. A Region-specific KMS key controls access to existing backups but does not stop a copy job from storing data in a vault in another Region, even if that data is encrypted.

Amazon DynamoDB delivers predictable single-digit millisecond latency, satisfying the <10 ms lookups required by the personalization microservice. Each record includes a TTL value set to 24 hours; DynamoDB automatically deletes items within the next couple of days after their expiration timestamp, which keeps the hot working set small and low-latency without consuming write capacity. With DynamoDB Streams enabled, every insert, update, and the eventual TTL delete record is sent to an AWS Lambda function that writes the data to Amazon S3. This provides a complete history for Amazon Athena at very low storage cost. The alternative solutions either cannot guarantee sub-10 ms access (Athena over S3), introduce higher operational cost and latency (Redshift), or are optimized for sequential stream consumption rather than on-demand key lookups (MSK).

Amazon DynamoDB delivers consistent single-digit millisecond latency at any scale, making it suitable for real-time lookups of individual sensor readings. The service is fully managed and automatically scales throughput to handle sustained ingestion rates like 50,000 writes per second. DynamoDB Time to Live (TTL) can be enabled on the timestamp attribute so items expire automatically after 30 days, eliminating manual cleanup. Amazon S3 is cost-effective but cannot provide millisecond point-read latency for individual items. Amazon RDS Aurora can achieve sub-second queries but would require careful sharding and manual capacity management to sustain 50,000 TPS, increasing operational effort. Amazon Redshift is optimized for analytic scans, not high-velocity, per-item transactions, and it offers second-level rather than millisecond response times. Therefore, DynamoDB is the most appropriate choice.