AWS Certified Data Engineer Associate DEA-C01 Practice Question

Using a broadcast join hint copies the small 100 MB reference DataFrame to every executor, so the larger 500 GB DataFrame can be joined locally without shuffling either dataset across the network. Increasing the number of shuffle partitions will not reduce the amount of data shuffled, and persisting the DataFrames adds memory pressure without eliminating the shuffle. Enabling speculative execution only mitigates slow tasks but does not address the fundamental shuffle cost of a standard hash join. Therefore, broadcasting the small table is the most effective solution.