Explanation
Correct code block:
spark.conf.set("spark.sql.shuffle.partitions", 100)
The conf interface is part of the SparkSession, so you need to call it through spark and not pyspark. To configure spark, you need to use the set method, not the get method. get reads a property, but does not write it. The correct property to achieve what is outlined in the question is spark.sql.aggregate.partitions, which needs to be passed to set as a string. Properties spark.shuffle.partitions and spark.sql.aggregate.partitions do not exist in Spark.
Static notebook | Dynamic notebook: See test 2