Microsoft Fabric Data Engineer Associate DP-700 Practice Question

Spark can avoid expensive shuffle joins by broadcasting small tables to every executor, turning the operation into a more efficient map-side join. Spark decides to broadcast a table only if its size is below the value specified in the spark.sql.autoBroadcastJoinThreshold configuration setting, whose default is 10 MB. Because each dimension table is about 100 MB, you must raise this threshold-setting it to 134,217,728 bytes (128 MB) will allow the three 100 MB dimension tables to qualify for automatic broadcasting and eliminate the large shuffle. Adjusting spark.sql.shuffle.partitions merely changes the number of shuffle partitions and does not enforce broadcasting. Enabling adaptive query execution can improve some plans but will not override the broadcast threshold by itself. Changing spark.sql.files.maxPartitionBytes affects input file partitioning, not join strategy. Therefore, increasing spark.sql.autoBroadcastJoinThreshold is the correct action.