Spark how to run spark file from spark shell

Question

I am using CDH  5 2  I am able to use spark-shell to run the commands     How can I run the file file spark  which contain spark commands   Is there any way to run compile the scala programs in CDH 5 2 without sbt    Thanks in advance

User · Accepted Answer

To load an external file from spark-shell simply do   load PATH TO FILE   This will call everything in your file    I don t have a solution for your SBT question though sorry  -

User · Answer

Just to give more perspective to the answers    Spark-shell is a scala repl  You can type  help to see the list of operation that are possible inside the scala shell  scala gt   help All commands can be abbreviated  e g    he instead of  help   edit  lt id gt   lt line gt         edit history  help  command           print this summary or command-specific help  history  num            show the history  optional num is commands to show   h   lt string gt              search the history  imports  name name      show import history  identifying sources of names  implicits  -v           show the implicits in scope  javap  lt path class gt       disassemble a file or class name  line  lt id gt   lt line gt         place line s  at the end of history  load  lt path gt              interpret lines in a file  paste  -raw   path      enter paste mode or paste a file  power                   enable power user mode  quit                    exit the interpreter  replay  options         reset the repl and replay all previous commands  require  lt path gt           add a jar to the classpath  reset  options          reset the repl to its initial state  forgetting all session entries  save  lt path gt              save replayable session to a file  sh  lt command line gt        run a shell command  result is implicitly   gt  List String    settings  lt options gt       update compiler options  if possible  see reset  silent                  disable enable automatic printing of results  type  -v   lt expr gt         display the type of an expression without evaluating it  kind  -v   lt expr gt         display the kind of expression s type  warnings                show the suppressed warnings from the most recent line which had any    load              interpret lines in a file

User · Answer

You can use either sbt or maven to compile spark programs  Simply add the spark as dependency to maven   lt repository gt         lt id gt Spark repository lt  id gt         lt url gt http   www sparkjava com nexus content repositories spark  lt  url gt   lt  repository gt    And then the dependency    lt dependency gt         lt groupId gt spark lt  groupId gt         lt artifactId gt spark lt  artifactId gt         lt version gt 1 2 0 lt  version gt   lt  dependency gt    In terms of running a file with spark commands   you can simply do this   echo     import org apache spark sql      ssc   new SQLContext sc     ssc sql  select   from mytable   collect    gt  spark input   Now run the commands script   cat spark input   spark-shell

User · Answer

In command line  you can use   spark-shell -i file scala   to run code which is written in file scala

User · Answer

Tested on both spark-shell version 1 6 3 and spark2-shell version 2 3 0 2 6 5 179-4  you can directly pipe to the shell s stdin like  spark-shell  lt  lt  lt   1 1    or in your use case   spark-shell  lt  file spark

User · Answer

You can run as you run your shell script  This example to run from command line environment example    bin spark-shell  - this is the path of your spark-shell under bin  home fold1 spark program py  - This is the path where your python program is there   So     bin spark-shell  home fold1 spark prohram py

[scala] Spark : how to run spark file from spark shell

Examples related to scala

Examples related to apache-spark

Examples related to cloudera-cdh

Examples related to cloudera-manager