[apache-spark] How to stop INFO messages displaying on spark console?

Edit your conf/log4j.properties file and change the following line:

log4j.rootCategory=INFO, console

to

log4j.rootCategory=ERROR, console

Another approach would be to :

Start spark-shell and type in the following:

import org.apache.log4j.Logger
import org.apache.log4j.Level

Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)

You won't see any logs after that.

Other options for Level include: all, debug, error, fatal, info, off, trace, trace_int, warn

Details about each can be found in the documentation.

Examples related to apache-spark

Select Specific Columns from Spark DataFrame Select columns in PySpark dataframe What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? How to find count of Null and Nan values for each column in a PySpark dataframe efficiently? Spark dataframe: collect () vs select () How does createOrReplaceTempView work in Spark? Spark difference between reduceByKey vs groupByKey vs aggregateByKey vs combineByKey Filter df when values matches part of a string in pyspark Filtering a pyspark dataframe using isin by exclusion Convert date from String to Date format in Dataframes

Examples related to log4j

No log4j2 configuration file found. Using default configuration: logging only errors to the console How to stop INFO messages displaying on spark console? What is log4j's default log file dumping path How to give environmental variable path for file appender in configuration file in log4j No appenders could be found for logger(log4j)? SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". in a Maven Project log4j:WARN No appenders could be found for logger (running jar file, not web app) Error: "setFile(null,false) call failed" when using log4j java.lang.ClassNotFoundException: org.apache.log4j.Level How can I create 2 separate log files with one log4j config file?

Examples related to spark-submit

Add jars to a Spark Job - spark-submit How to stop INFO messages displaying on spark console?