From the answer here, spark.sql.shuffle.partitions
configures the number of partitions that are used when shuffling data for joins or aggregations.
spark.default.parallelism
is the default number of partitions in RDD
s returned by transformations like join
, reduceByKey
, and parallelize
when not set explicitly by the user. Note that spark.default.parallelism
seems to only be working for raw RDD
and is ignored when working with dataframes.
If the task you are performing is not a join or aggregation and you are working with dataframes then setting these will not have any effect. You could, however, set the number of partitions yourself by calling df.repartition(numOfPartitions)
(don't forget to assign it to a new val
) in your code.
To change the settings in your code you can simply do:
sqlContext.setConf("spark.sql.shuffle.partitions", "300")
sqlContext.setConf("spark.default.parallelism", "300")
Alternatively, you can make the change when submitting the job to a cluster with spark-submit
:
./bin/spark-submit --conf spark.sql.shuffle.partitions=300 --conf spark.default.parallelism=300