How to set Apache Spark Executor memory

Question

How can I increase the memory available for Apache spark executor nodes   I have a 2 GB file that is suitable to loading in to Apache Spark  I am running apache spark for the moment on 1 machine  so the driver and executor are on the same machine  The machine has 8 GB of memory    When I try count the lines of the file after setting the file to be cached in memory I get these errors   2014-10-25 22 25 12 WARN  CacheManager 71 - Not enough space to cache partition rdd 1 1 in memory  Free memory is 278099801 bytes    I looked at the documentation here and set spark executor memory to 4g in  SPARK HOME conf spark-defaults conf  The UI shows this variable is set in the Spark Environment  You can find screenshot here  However when I go to the Executor tab the memory limit for my single Executor is  still set to 265 4 MB   I also still get the same error    I tried various things mentioned here but I still get the error and don t have a clear idea where I should change the setting     I am running my code interactively from the spark-shell

User · Answer

you mentioned that you are running yourcode interactivly on spark-shell so, while doing if no proper value is set for driver-memory or executor memory then spark defaultly assign some value to it, which is based on it's properties file(where default value is being mentioned).

I hope you are aware of the fact that there is one driver(master node) and worker-node(where executors are get created and processed), so basically two types of space is required by the spark program,so if you want to set driver memory then when start spark-shell .

spark-shell --driver-memory "your value" and to set executor memory : spark-shell --executor-memory "your value"

then I think you are good to go with the desired value of the memory that you want your spark-shell to use.

User · Answer

As far as i know it wouldn t be possible to change the spark executor memory at run time  If you are running a stand-alone version  with pyspark and graphframes  you can launch the pyspark REPL by executing the following command  pyspark --driver-memory 2g --executor-memory 6g --packages graphframes graphframes 0 7 0-spark2 4-s 2 11  Be sure to change the SPARK VERSION environment variable appropriately regarding the latest released version of Spark

User · Answer

spark-submit      --class org apache spark examples SparkPi      --master yarn      --deploy-mode cluster      can be client for client mode    --executor-memory 2G      --num-executors 5       path to examples jar      1000

User · Answer

You need to increase the driver memory On mac i e when running on local master   the default driver-memory is 1024M   By default  thus 380Mb is allotted to the executor     Upon increasing  --driver-memory 2G   executor memory got increased to  950Mb

User · Answer

In Windows or Linux  you can use this command   spark-shell --driver-memory 2G

User · Answer

You can build command using following example    spark-submit    --jars  usr share java postgresql-jdbc jar    --class com examples WordCount3   home vaquarkhan spark-scala-maven-project-0 0 1-SNAPSHOT jar --jar  --num-executors 3 --driver-memory 10g   --executor-memory 10g   --executor-cores 1  --master local --deploy-mode client  --name wordcount3 --conf  spark app id wordcount

User · Answer

The answer submitted by Grega helped me to solve my issue   I am running Spark locally from a python script inside a Docker container   Initially I was getting a Java out-of-memory error when processing some data in Spark   However  I was able to assign more memory by adding the following line to my script   conf SparkConf   conf set  spark driver memory    4g      Here is a full example of the python script which I use to start Spark   import os import sys import glob  spark home     lt DIRECTORY WHERE SPARK FILES EXIST gt  spark-2 0 0-bin-hadoop2 7   driver home     lt DIRECTORY WHERE DRIVERS EXIST gt    if  SPARK HOME  not in os environ      os environ  SPARK HOME     spark home   SPARK HOME   os environ  SPARK HOME    sys path insert 0 os path join SPARK HOME  python    for lib in glob glob os path join SPARK HOME   python    lib      zip         sys path insert 0 lib    from pyspark import SparkContext from pyspark import SparkConf from pyspark sql import SQLContext  conf SparkConf   conf set  spark executor memory    4g   conf set  spark driver memory    4g   conf set  spark cores max    2   conf set  spark driver extraClassPath       driver home   jdbc postgresql-9 4-1201-jdbc41 jar         driver home   jdbc clickhouse-jdbc-0 1 52 jar         driver home   mongo mongo-spark-connector 2 11-2 2 3 jar         driver home   mongo mongo-java-driver-3 8 0 jar     sc   SparkContext getOrCreate conf   spark   SQLContext sc

User · Answer

Also note  that for local mode you have to set the amount of driver memory before starting jvm   bin spark-submit --driver-memory 2g --class your class here app jar   This will start the JVM with 2G instead of the default 512M  Details here      For local mode you only have one executor  and this executor is your driver  so you need to set the driver s memory instead   That said  in local mode  by the time you run spark-submit  a JVM has already been launched with the default memory settings  so setting  spark driver memory  in your conf won t actually do anything for you  Instead  you need to run spark-submit as follows

User · Answer

Spark executor memory is required for running your spark tasks based on the instructions given by your driver program  Basically  it requires more resources that depends on your submitted job   Executor memory includes memory required for executing the tasks plus overhead memory which should not be greater than the size of JVM and yarn maximum container size    Add the following parameters in spark-defaults conf  spar executor cores 1  spark executor memory 2g   If you using any cluster management tools like cloudera manager or amabari please refresh the cluster configuration for reflecting the latest configs to all nodes in the cluster   Alternatively  we can pass the executor core and memory value as an argument while running spark-submit command along with class and application path   Example   spark-submit      --class org apache spark examples SparkPi      --master yarn      --deploy-mode cluster      can be client for client mode    --executor-memory 2G      --num-executors 5       path to examples jar      1000

User · Answer

create a file called spark-env sh in spark conf directory and  add this line  SPARK EXECUTOR MEMORY 2000m  memory size which you want to allocate for the executor

User · Answer

Apparently  the question never says to run on local mode not on yarn  Somehow I couldnt get spark-default conf change to work  Instead I tried this and it worked for me  bin spark-shell --master yarn --num-executors 6  --driver-memory 5g --executor-memory 7g     couldnt bump executor-memory to 8g there is some restriction from yarn configuration

User · Answer

Since you are running Spark in local mode  setting spark executor memory won t have any effect  as you have noticed  The reason for this is that the Worker  lives  within the driver JVM process that you start when you start spark-shell and the default memory used for that is 512M  You can increase that by setting spark driver memory to something higher  for example 5g  You can do that by either    setting it in the properties file  default is  SPARK HOME conf spark-defaults conf     spark driver memory              5g  or by supplying configuration setting at runtime      bin spark-shell --driver-memory 5g    Note that this cannot be achieved by setting it in the application  because it is already too late by then  the process has already started with some amount of memory   The reason for 265 4 MB is that Spark dedicates spark storage memoryFraction   spark storage safetyFraction to the total amount of storage memory and by default they are 0 6 and 0 9   512 MB   0 6   0 9   265 4 MB   So be aware that not the whole amount of driver memory will be available for RDD storage   But when you ll start running this on a cluster  the spark executor memory setting will take over when calculating the amount to dedicate to Spark s memory cache

[memory] How to set Apache Spark Executor memory

Examples related to memory

Examples related to apache-spark