[apache-spark] How to check Spark Version

I want to check the spark version in cdh 5.7.0. I have searched on the internet but not able to understand. Please help.

This question is related to apache-spark hadoop cloudera

The answer is


You can get the spark version by using the following command:

spark-submit --version

spark-shell --version

spark-sql --version

You can visit the below site to know the spark-version used in CDH 5.7.0

http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_new_in_cdh_57.html#concept_m3k_rxh_1v


According to the Cloudera documentation - What's New in CDH 5.7.0 it includes Spark 1.6.0.


Examples related to apache-spark

Select Specific Columns from Spark DataFrame Select columns in PySpark dataframe What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? How to find count of Null and Nan values for each column in a PySpark dataframe efficiently? Spark dataframe: collect () vs select () How does createOrReplaceTempView work in Spark? Spark difference between reduceByKey vs groupByKey vs aggregateByKey vs combineByKey Filter df when values matches part of a string in pyspark Filtering a pyspark dataframe using isin by exclusion Convert date from String to Date format in Dataframes

Examples related to hadoop

Hadoop MapReduce: Strange Result when Storing Previous Value in Memory in a Reduce Class (Java) What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? How to check Spark Version What are the pros and cons of parquet format compared to other formats? java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient How to export data from Spark SQL to CSV How to copy data from one HDFS to another HDFS? How to calculate Date difference in Hive Select top 2 rows in Hive Spark - load CSV file as DataFrame?

Examples related to cloudera

How to check Spark Version JsonParseException: Unrecognized token 'http': was expecting ('true', 'false' or 'null') Hive External Table Skip First Row Buiding Hadoop with Eclipse / Maven - Missing artifact jdk.tools:jdk.tools:jar:1.6