The code block shown below contains an error. The code block is intended to adjust the number of partitions used in wide transformations like join() to 32. Identify the error.
Code block:
spark.conf.set("spark.default.parallelism", "32")
The code block shown below contains an error. The code block is intended to adjust the number of partitions used in wide transformations like join() to 32. Identify the error.
Code block:
spark.conf.set("spark.default.parallelism", "32")
The code intends to adjust the number of partitions used in wide transformations like join(). The correct Spark configuration parameter for this purpose is spark.sql.shuffle.partitions, not spark.default.parallelism.
should be spark.sql.shuffle.partitions for joins
spark.conf.set("spark.sql.shuffle.partitions", "32")
A ist richtig