Certified Associate Developer for Apache Spark Exam

Exam Certified Associate Developer for Apache Spark All QuestionsBrowse all questions from this exam

Question 87

The code block shown below contains an error. The code block is intended to adjust the number of partitions used in wide transformations like join() to 32. Identify the error.

Code block:

spark.conf.set("spark.default.parallelism", "32")

spark.default.parallelism is not the right Spark configuration parameter – spark.sql.shuffle.partitions should be used instead.

There is no way to adjust the number of partitions used in wide transformations – it defaults to the number of total CPUs in the cluster.

Spark configuration parameters cannot be set in runtime.

Spark configuration parameters are not set with spark.conf.set().

The second argument should not be the string version of "32" – it should be the integer 32.

Correct Answer: A

The code intends to adjust the number of partitions used in wide transformations like join(). The correct Spark configuration parameter for this purpose is spark.sql.shuffle.partitions, not spark.default.parallelism.

Discussion

Ram459Option: A

should be spark.sql.shuffle.partitions for joins

newusernameOption: A

spark.conf.set("spark.sql.shuffle.partitions", "32")

AnweeeOption: A

A ist richtig