What is the difference between spark sql shuffle partitions and spark default parallelism

Question

What s the difference between spark sql shuffle partitions and spark default parallelism   I have tried to set both of them in SparkSQL  but the task number of the second stage is always 200

User · Answer

spark default parallelism is the default number of partition set by spark which is by default 200   and if you want to increase the number of partition than you can apply the property spark sql shuffle partitions to set number of partition in the spark configuration or while running spark SQL   Normally this spark sql shuffle partitions it is being used when we have a memory congestion and we see below error  spark error java lang IllegalArgumentException  Size exceeds Integer MAX VALUE  so set your can allocate a partition as 256 MB per partition and that you can use to set for your processes   also If number of partitions is near to 2000 then increase it to more than 2000  As spark applies different logic for partition  lt  2000 and   2000 which will increase your code performance by decreasing the memory footprint as data default is highly compressed if  2000

User · Answer

From the answer here  spark sql shuffle partitions configures the number of partitions that are used when shuffling data for joins or aggregations   spark default parallelism is the default number of partitions in RDDs returned by transformations like join  reduceByKey  and parallelize when not set explicitly by the user  Note that spark default parallelism seems to only be working for raw RDD and is ignored when working with dataframes   If the task you are performing is not a join or aggregation and you are working with dataframes then setting these will not have any effect  You could  however  set the number of partitions yourself by calling df repartition numOfPartitions   don t forget to assign it to a new val  in your code     To change the settings in your code you can simply do   sqlContext setConf  spark sql shuffle partitions    300   sqlContext setConf  spark default parallelism    300     Alternatively  you can make the change when submitting the job to a cluster with spark-submit     bin spark-submit --conf spark sql shuffle partitions 300 --conf spark default parallelism 300

[performance] What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism?

Examples related to performance

Examples related to apache-spark

Examples related to hadoop

Examples related to apache-spark-sql