SyntaxFix

[apache-spark] How to check Spark Version

You can get the spark version by using the following command:

spark-submit --version

spark-shell --version

spark-sql --version

You can visit the below site to know the spark-version used in CDH 5.7.0

http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_new_in_cdh_57.html#concept_m3k_rxh_1v

Examples related to apache-spark

• Select Specific Columns from Spark DataFrame • Select columns in PySpark dataframe • What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? • How to find count of Null and Nan values for each column in a PySpark dataframe efficiently? • Spark dataframe: collect () vs select () • How does createOrReplaceTempView work in Spark? • Spark difference between reduceByKey vs groupByKey vs aggregateByKey vs combineByKey • Filter df when values matches part of a string in pyspark • Filtering a pyspark dataframe using isin by exclusion • Convert date from String to Date format in Dataframes

Examples related to hadoop

• Hadoop MapReduce: Strange Result when Storing Previous Value in Memory in a Reduce Class (Java) • What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? • How to check Spark Version • What are the pros and cons of parquet format compared to other formats? • java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient • How to export data from Spark SQL to CSV • How to copy data from one HDFS to another HDFS? • How to calculate Date difference in Hive • Select top 2 rows in Hive • Spark - load CSV file as DataFrame?

Examples related to cloudera

• How to check Spark Version • JsonParseException: Unrecognized token 'http': was expecting ('true', 'false' or 'null') • Hive External Table Skip First Row • Buiding Hadoop with Eclipse / Maven - Missing artifact jdk.tools:jdk.tools:jar:1.6