SyntaxFix

[apache-spark] How to check Spark Version

Addition to @Binary Nerd

If you are using Spark, use the following to get the Spark version:

spark-submit --version

or

Login to the Cloudera Manager and goto Hosts page then run inspect hosts in cluster

Examples related to apache-spark

• Select Specific Columns from Spark DataFrame • Select columns in PySpark dataframe • What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? • How to find count of Null and Nan values for each column in a PySpark dataframe efficiently? • Spark dataframe: collect () vs select () • How does createOrReplaceTempView work in Spark? • Spark difference between reduceByKey vs groupByKey vs aggregateByKey vs combineByKey • Filter df when values matches part of a string in pyspark • Filtering a pyspark dataframe using isin by exclusion • Convert date from String to Date format in Dataframes

Examples related to hadoop

• Hadoop MapReduce: Strange Result when Storing Previous Value in Memory in a Reduce Class (Java) • What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? • How to check Spark Version • What are the pros and cons of parquet format compared to other formats? • java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient • How to export data from Spark SQL to CSV • How to copy data from one HDFS to another HDFS? • How to calculate Date difference in Hive • Select top 2 rows in Hive • Spark - load CSV file as DataFrame?

Examples related to cloudera

• How to check Spark Version • JsonParseException: Unrecognized token 'http': was expecting ('true', 'false' or 'null') • Hive External Table Skip First Row • Buiding Hadoop with Eclipse / Maven - Missing artifact jdk.tools:jdk.tools:jar:1.6