[windows] How to set up Spark on Windows?

I am trying to setup Apache Spark on Windows.

After searching a bit, I understand that the standalone mode is what I want. Which binaries do I download in order to run Apache spark in windows? I see distributions with hadoop and cdh at the spark download page.

I don't have references in web to this. A step by step guide to this is highly appreciated.

Here is a simple minimum script to run from any python console. It assumes that you have extracted the Spark libraries that you have downloaded into C:\Apache\spark-1.6.1.

This works in Windows without building anything and solves problems where Spark would complain about recursive pickling.

import sys
import os
spark_home = 'C:\Apache\spark-1.6.1'

sys.path.insert(0, os.path.join(spark_home, 'python'))
sys.path.insert(0, os.path.join(spark_home, 'python\lib\pyspark.zip')) 
sys.path.insert(0, os.path.join(spark_home, 'python\lib\py4j-0.9-src.zip')) 

# Start a spark context:
sc = pyspark.SparkContext()

# 
lines = sc.textFile(os.path.join(spark_home, "README.md")
pythonLines = lines.filter(lambda line: "Python" in line)
pythonLines.first()

The guide by Ani Menon (thx!) almost worked for me on windows 10, i just had to get a newer winutils.exe off that git (currently hadoop-2.8.1): https://github.com/steveloughran/winutils

Cloudera and Hortonworks are the best tools to start up with the HDFS in Microsoft Windows. You can also use VMWare or VBox to initiate Virtual Machine to establish build to your HDFS and Spark, Hive, HBase, Pig, Hadoop with Scala, R, Java, Python.

You can use following ways to setup Spark:

• Building from Source
• Using prebuilt release

Though there are various ways to build Spark from Source.
First I tried building Spark source with SBT but that requires hadoop. To avoid those issues, I used pre-built release.

Instead of Source,I downloaded Prebuilt release for hadoop 2.x version and ran it. For this you need to install Scala as prerequisite.

I have collated all steps here :
How to run Apache Spark on Windows7 in standalone mode

Hope it'll help you..!!!

Trying to work with spark-2.x.x, building Spark source code didn't work for me.

1. So, although I'm not going to use Hadoop, I downloaded the pre-built Spark with hadoop embeded : spark-2.0.0-bin-hadoop2.7.tar.gz

2. Point SPARK_HOME on the extracted directory, then add to PATH: ;%SPARK_HOME%\bin;

3. Download the executable winutils from the Hortonworks repository, or from Amazon AWS platform winutils.

4. Create a directory where you place the executable winutils.exe. For example, C:\SparkDev\x64. Add the environment variable %HADOOP_HOME% which points to this directory, then add %HADOOP_HOME%\bin to PATH.

5. Using command line, create the directory:

   mkdir C:\tmp\hive
   

6. Using the executable that you downloaded, add full permissions to the file directory you created but using the unixian formalism:

   %HADOOP_HOME%\bin\winutils.exe chmod 777 /tmp/hive
   

7. Type the following command line:

   %SPARK_HOME%\bin\spark-shell
   

Scala command line input should be shown automatically.

Remark : You don't need to configure Scala separately. It's built-in too.

Here's the fixes to get it to run in Windows without rebuilding everything - such as if you do not have a recent version of MS-VS. (You will need a Win32 C++ compiler, but you can install MS VS Community Edition free.)

I've tried this with Spark 1.2.2 and mahout 0.10.2 as well as with the latest versions in November 2015. There are a number of problems including the fact that the Scala code tries to run a bash script (mahout/bin/mahout) which does not work of course, the sbin scripts have not been ported to windows, and the winutils are missing if hadoop is not installed.

(1) Install scala, then unzip spark/hadoop/mahout into the root of C: under their respective product names.

(2) Rename \mahout\bin\mahout to mahout.sh.was (we will not need it)

(3) Compile the following Win32 C++ program and copy the executable to a file named C:\mahout\bin\mahout (that's right - no .exe suffix, like a Linux executable)

#include "stdafx.h"
#define BUFSIZE 4096
#define VARNAME TEXT("MAHOUT_CP")
int _tmain(int argc, _TCHAR* argv[]) {
    DWORD dwLength;     LPTSTR pszBuffer;
    pszBuffer = (LPTSTR)malloc(BUFSIZE*sizeof(TCHAR));
    dwLength = GetEnvironmentVariable(VARNAME, pszBuffer, BUFSIZE);
    if (dwLength > 0) { _tprintf(TEXT("%s\n"), pszBuffer); return 0; }
    return 1;
}

(4) Create the script \mahout\bin\mahout.bat and paste in the content below, although the exact names of the jars in the _CP class paths will depend on the versions of spark and mahout. Update any paths per your installation. Use 8.3 path names without spaces in them. Note that you cannot use wildcards/asterisks in the classpaths here.

set SCALA_HOME=C:\Progra~2\scala
set SPARK_HOME=C:\spark
set HADOOP_HOME=C:\hadoop
set MAHOUT_HOME=C:\mahout
set SPARK_SCALA_VERSION=2.10
set MASTER=local[2]
set MAHOUT_LOCAL=true
set path=%SCALA_HOME%\bin;%SPARK_HOME%\bin;%PATH%
cd /D %SPARK_HOME%
set SPARK_CP=%SPARK_HOME%\conf\;%SPARK_HOME%\lib\xxx.jar;...other jars...
set MAHOUT_CP=%MAHOUT_HOME%\lib\xxx.jar;...other jars...;%MAHOUT_HOME%\xxx.jar;...other jars...;%SPARK_CP%;%MAHOUT_HOME%\lib\spark\xxx.jar;%MAHOUT_HOME%\lib\hadoop\xxx.jar;%MAHOUT_HOME%\src\conf;%JAVA_HOME%\lib\tools.jar
start "master0" "%JAVA_HOME%\bin\java" -cp "%SPARK_CP%" -Xms1g -Xmx1g org.apache.spark.deploy.master.Master --ip localhost --port 7077 --webui-port 8082 >>out-master0.log 2>>out-master0.err
start "worker1" "%JAVA_HOME%\bin\java" -cp "%SPARK_CP%" -Xms1g -Xmx1g org.apache.spark.deploy.worker.Worker spark://localhost:7077 --webui-port 8083 >>out-worker1.log 2>>out-worker1.err
...you may add more workers here...
cd /D %MAHOUT_HOME%
"%JAVA_HOME%\bin\java" -Xmx4g -classpath "%MAHOUT_CP%" "org.apache.mahout.sparkbindings.shell.Main"

The name of the variable MAHOUT_CP should not be changed, as it is referenced in the C++ code.

Of course you can comment-out the code that launches the Spark master and worker because Mahout will run Spark as-needed; I just put it in the batch job to show you how to launch it if you wanted to use Spark without Mahout.

(5) The following tutorial is a good place to begin:

https://mahout.apache.org/users/sparkbindings/play-with-shell.html

You can bring up the Mahout Spark instance at:

"C:\Program Files (x86)\Google\Chrome\Application\chrome" --disable-web-security http://localhost:4040

You can download spark from here:

http://spark.apache.org/downloads.html

I recommend you this version: Hadoop 2 (HDP2, CDH5)

Since version 1.0.0 there are .cmd scripts to run spark in windows.

Unpack it using 7zip or similar.

To start you can execute /bin/spark-shell.cmd --master local[2]

To configure your instance, you can follow this link: http://spark.apache.org/docs/latest/

Steps to install Spark in local mode:

1. Install Java 7 or later. To test java installation is complete, open command prompt type java and hit enter. If you receive a message 'Java' is not recognized as an internal or external command. You need to configure your environment variables, JAVA_HOME and PATH to point to the path of jdk.

2. Download and install Scala.

   Set SCALA_HOME in Control Panel\System and Security\System goto "Adv System settings" and add %SCALA_HOME%\bin in PATH variable in environment variables.

3. Install Python 2.6 or later from Python Download link.

4. Download SBT. Install it and set SBT_HOME as an environment variable with value as <<SBT PATH>>.

5. Download winutils.exe from HortonWorks repo or git repo. Since we don't have a local Hadoop installation on Windows we have to download winutils.exe and place it in a bin directory under a created Hadoop home directory. Set HADOOP_HOME = <<Hadoop home directory>> in environment variable.

6. We will be using a pre-built Spark package, so choose a Spark pre-built package for Hadoop Spark download. Download and extract it.

   Set SPARK_HOME and add %SPARK_HOME%\bin in PATH variable in environment variables.

7. Run command: spark-shell

8. Open http://localhost:4040/ in a browser to see the SparkContext web UI.

Here are seven steps to install spark on windows 10 and run it from python:

Step 1: download the spark 2.2.0 tar (tape Archive) gz file to any folder F from this link - https://spark.apache.org/downloads.html. Unzip it and copy the unzipped folder to the desired folder A. Rename the spark-2.2.0-bin-hadoop2.7 folder to spark.

Let path to the spark folder be C:\Users\Desktop\A\spark

Step 2: download the hardoop 2.7.3 tar gz file to the same folder F from this link - https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz. Unzip it and copy the unzipped folder to the same folder A. Rename the folder name from Hadoop-2.7.3.tar to hadoop. Let path to the hadoop folder be C:\Users\Desktop\A\hadoop

Step 3: Create a new notepad text file. Save this empty notepad file as winutils.exe (with Save as type: All files). Copy this O KB winutils.exe file to your bin folder in spark - C:\Users\Desktop\A\spark\bin

Step 4: Now, we have to add these folders to the System environment.

4a: Create a system variable (not user variable as user variable will inherit all the properties of the system variable) Variable name: SPARK_HOME Variable value: C:\Users\Desktop\A\spark

Find Path system variable and click edit. You will see multiple paths. Do not delete any of the paths. Add this variable value - ;C:\Users\Desktop\A\spark\bin

4b: Create a system variable

Variable name: HADOOP_HOME Variable value: C:\Users\Desktop\A\hadoop

Find Path system variable and click edit. Add this variable value - ;C:\Users\Desktop\A\hadoop\bin

4c: Create a system variable Variable name: JAVA_HOME Search Java in windows. Right click and click open file location. You will have to again right click on any one of the java files and click on open file location. You will be using the path of this folder. OR you can search for C:\Program Files\Java. My Java version installed on the system is jre1.8.0_131. Variable value: C:\Program Files\Java\jre1.8.0_131\bin

Find Path system variable and click edit. Add this variable value - ;C:\Program Files\Java\jre1.8.0_131\bin

Step 5: Open command prompt and go to your spark bin folder (type cd C:\Users\Desktop\A\spark\bin). Type spark-shell.

C:\Users\Desktop\A\spark\bin>spark-shell

It may take time and give some warnings. Finally, it will show welcome to spark version 2.2.0

Step 6: Type exit() or restart the command prompt and go the spark bin folder again. Type pyspark:

C:\Users\Desktop\A\spark\bin>pyspark

It will show some warnings and errors but ignore. It works.

Step 7: Your download is complete. If you want to directly run spark from python shell then: go to Scripts in your python folder and type

pip install findspark

in command prompt.

In python shell

import findspark
findspark.init()

import the necessary modules

from pyspark import SparkContext
from pyspark import SparkConf

If you would like to skip the steps for importing findspark and initializing it, then please follow the procedure given in importing pyspark in python shell