AWS Certified Data Engineer Associate DEA-C01 Practice Question

Writing the Spark output directly to an Amazon S3 prefix with the EMRFS connector removes the need for a separate copy step and lets the cluster shut down as soon as the job ends, lowering cost. Because the data is already in S3, an AWS Glue crawler can automatically detect the new date partition without any network access to the EMR cluster, so no inbound ports have to be opened. Reading HDFS through a JDBC connection to Hive would require opening port 9083 on the EMR master node and still leaves the data on-cluster. Using AWS DataSync would require installing and managing agents on the EMR core nodes, which adds operational overhead and keeps the cluster running longer. Running DistCp each night also keeps the cluster running longer and adds operational overhead compared with writing directly to S3 in the first place.