[apache-spark] How to stop INFO messages displaying on spark console?

I'd like to stop various messages that are coming on spark shell.

I tried to edit the log4j.properties file in order to stop these message.

Here are the contents of log4j.properties

# Define the root logger with appender file
log4j.rootCategory=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO

But messages are still getting displayed on the console.

Here are some example messages

15/01/05 15:11:45 INFO SparkEnv: Registering BlockManagerMaster
15/01/05 15:11:45 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20150105151145-b1ba
15/01/05 15:11:45 INFO MemoryStore: MemoryStore started with capacity 0.0 B.
15/01/05 15:11:45 INFO ConnectionManager: Bound socket to port 44728 with id = ConnectionManagerId(192.168.100.85,44728)
15/01/05 15:11:45 INFO BlockManagerMaster: Trying to register BlockManager
15/01/05 15:11:45 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager 192.168.100.85:44728 with 0.0 B RAM
15/01/05 15:11:45 INFO BlockManagerMaster: Registered BlockManager
15/01/05 15:11:45 INFO HttpServer: Starting HTTP Server
15/01/05 15:11:45 INFO HttpBroadcast: Broadcast server star

How do I stop these?

Edit your conf/log4j.properties file and change the following line:

log4j.rootCategory=INFO, console

to

log4j.rootCategory=ERROR, console

Another approach would be to :

Start spark-shell and type in the following:

import org.apache.log4j.Logger
import org.apache.log4j.Level

Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)

You won't see any logs after that.

Other options for Level include: all, debug, error, fatal, info, off, trace, trace_int, warn

Details about each can be found in the documentation.

I just add this line to all my pyspark scripts on top just below the import statements.

SparkSession.builder.getOrCreate().sparkContext.setLogLevel("ERROR")

example header of my pyspark scripts

from pyspark.sql import SparkSession, functions as fs
SparkSession.builder.getOrCreate().sparkContext.setLogLevel("ERROR")

You set disable the Logs by setting its level to OFF as follows:

Logger.getLogger("org").setLevel(Level.OFF);
Logger.getLogger("akka").setLevel(Level.OFF);

or edit log file and set log level to off by just changing the following property:

log4j.rootCategory=OFF, console

Right after starting spark-shell type ;

sc.setLogLevel("ERROR")

In Spark 2.0 (Scala):

spark = SparkSession.builder.getOrCreate()
spark.sparkContext.setLogLevel("ERROR")

API Docs : https://spark.apache.org/docs/2.2.0/api/scala/index.html#org.apache.spark.sql.SparkSession

For Java:

spark = SparkSession.builder.getOrCreate();
spark.sparkContext().setLogLevel("ERROR");

All the methods collected with examples

Intro

Actually, there are many ways to do it. Some are harder from others, but it is up to you which one suits you best. I will try to showcase them all.

#1 Programatically in your app

Seems to be the easiest, but you will need to recompile your app to change those settings. Personally, I don't like it but it works fine.

Example:

import org.apache.log4j.{Level, Logger}

val rootLogger = Logger.getRootLogger()
rootLogger.setLevel(Level.ERROR)

Logger.getLogger("org.apache.spark").setLevel(Level.WARN)
Logger.getLogger("org.spark-project").setLevel(Level.WARN)

You can achieve much more just using log4j API.
Source: [Log4J Configuration Docs, Configuration section]

#2 Pass log4j.properties during spark-submit

This one is very tricky, but not impossible. And my favorite.

Log4J during app startup is always looking for and loading log4j.properties file from classpath.

However, when using spark-submit Spark Cluster's classpath has precedence over app's classpath! This is why putting this file in your fat-jar will not override the cluster's settings!

Add -Dlog4j.configuration=<location of configuration file> to spark.driver.extraJavaOptions (for the driver) or
spark.executor.extraJavaOptions (for executors).

Note that if using a file, the file: protocol should be explicitly provided, and the file needs to exist locally on all the nodes.

To satisfy the last condition, you can either upload the file to the location available for the nodes (like hdfs) or access it locally with driver if using deploy-mode client. Otherwise:

upload a custom log4j.properties using spark-submit, by adding it to the --files list of files to be uploaded with the application.

Source: Spark docs, Debugging

Steps:

Example log4j.properties:

# Blacklist all to warn level
log4j.rootCategory=WARN, console

log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Whitelist our app to info :)
log4j.logger.com.github.atais=INFO

Executing spark-submit, for cluster mode:

spark-submit \
    --master yarn \
    --deploy-mode cluster \
    --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties" \
    --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties" \
    --files "/absolute/path/to/your/log4j.properties" \
    --class com.github.atais.Main \
    "SparkApp.jar"

Note that you must use --driver-java-options if using client mode. Spark docs, Runtime env

Executing spark-submit, for client mode:

spark-submit \
    --master yarn \
    --deploy-mode client \
    --driver-java-options "-Dlog4j.configuration=file:/absolute/path/to/your/log4j.properties \
    --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties" \
    --files "/absolute/path/to/your/log4j.properties" \
    --class com.github.atais.Main \
    "SparkApp.jar"

Notes:

1. Files uploaded to spark-cluster with --files will be available at root dir, so there is no need to add any path in file:log4j.properties.
2. Files listed in --files must be provided with absolute path!
3. file: prefix in configuration URI is mandatory.

#3 Edit cluster's conf/log4j.properties

This changes global logging configuration file.

update the $SPARK_CONF_DIR/log4j.properties file and it will be automatically uploaded along with the other configurations.

Source: Spark docs, Debugging

To find your SPARK_CONF_DIR you can use spark-shell:

atais@cluster:~$ spark-shell 
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.1
      /_/   

scala> System.getenv("SPARK_CONF_DIR")
res0: String = /var/lib/spark/latest/conf

Now just edit /var/lib/spark/latest/conf/log4j.properties (with example from method #2) and all your apps will share this configuration.

#4 Override configuration directory

If you like the solution #3, but want to customize it per application, you can actually copy conf folder, edit it contents and specify as the root configuration during spark-submit.

To specify a different configuration directory other than the default “SPARK_HOME/conf”, you can set SPARK_CONF_DIR. Spark will use the configuration files (spark-defaults.conf, spark-env.sh, log4j.properties, etc) from this directory.

Source: Spark docs, Configuration

Steps:

1. Copy cluster's conf folder (more info, method #3)
2. Edit log4j.properties in that folder (example in method #2)

3. Set SPARK_CONF_DIR to this folder, before executing spark-submit,
   example:

   export SPARK_CONF_DIR=/absolute/path/to/custom/conf
   
   spark-submit \
       --master yarn \
       --deploy-mode cluster \
       --class com.github.atais.Main \
       "SparkApp.jar"
   

Conclusion

I am not sure if there is any other method, but I hope this covers the topic from A to Z. If not, feel free to ping me in the comments!

Enjoy your way!

tl;dr

For Spark Context you may use:

sc.setLogLevel(<logLevel>)

where loglevel can be ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE or WARN.

Details-

Internally, setLogLevel calls org.apache.log4j.Level.toLevel(logLevel) that it then uses to set using org.apache.log4j.LogManager.getRootLogger().setLevel(level).

You may directly set the logging levels to OFF using:

LogManager.getLogger("org").setLevel(Level.OFF)

You can set up the default logging for Spark shell in conf/log4j.properties. Use conf/log4j.properties.template as a starting point.

Setting Log Levels in Spark Applications

In standalone Spark applications or while in Spark Shell session, use the following:

import org.apache.log4j.{Level, Logger}

Logger.getLogger(classOf[RackResolver]).getLevel
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)

Disabling logging(in log4j):

Use the following in conf/log4j.properties to disable logging completely:

log4j.logger.org=OFF

Reference: Mastering Spark by Jacek Laskowski.

If anyone else is stuck on this,

nothing of the above worked for me. I had to remove

implementation group: "ch.qos.logback", name: "logback-classic", version: "1.2.3"
implementation group: 'com.typesafe.scala-logging', name: "scala-logging_$scalaVersion", version: '3.9.2'

from my build.gradle for the logs to disappear. TLDR: Don't import any other logging frameworks, you should be fine just using org.apache.log4j.Logger

An interesting idea is to use the RollingAppender as suggested here: http://shzhangji.com/blog/2015/05/31/spark-streaming-logging-configuration/ so that you don't "polute" the console space, but still be able to see the results under $YOUR_LOG_PATH_HERE/${dm.logging.name}.log.

    log4j.rootLogger=INFO, rolling

log4j.appender.rolling=org.apache.log4j.RollingFileAppender
log4j.appender.rolling.layout=org.apache.log4j.PatternLayout
log4j.appender.rolling.layout.conversionPattern=[%d] %p %m (%c)%n
log4j.appender.rolling.maxFileSize=50MB
log4j.appender.rolling.maxBackupIndex=5
log4j.appender.rolling.file=$YOUR_LOG_PATH_HERE/${dm.logging.name}.log
log4j.appender.rolling.encoding=UTF-8

Another method that solves the cause is to observe what kind of loggings do you usually have (coming from different modules and dependencies), and set for each the granularity for the logging, while turning "quiet" third party logs that are too verbose:

For instance,

    # Silence akka remoting
log4j.logger.Remoting=ERROR
log4j.logger.akka.event.slf4j=ERROR
log4j.logger.org.spark-project.jetty.server=ERROR
log4j.logger.org.apache.spark=ERROR
log4j.logger.com.anjuke.dm=${dm.logging.level}
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO

1. Adjust conf/log4j.properties as described by other log4j.rootCategory=ERROR, console
2. Make sure while executing your spark job you pass --file flag with log4j.properties file path
3. If it still doesn't work you might have a jar that has log4j.properties that is being called before your new log4j.properties. Remove that log4j.properties from jar (if appropriate)

Adding the following to the PySpark did the job for me:

self.spark.sparkContext.setLogLevel("ERROR")

self.spark is the spark session (self.spark = spark_builder.getOrCreate())

Use below command to change log level while submitting application using spark-submit or spark-sql:

spark-submit \
--conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=file:<file path>/log4j.xml" \
--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:<file path>/log4j.xml"

Note: replace <file path> where log4j config file is stored.

Log4j.properties:

log4j.rootLogger=ERROR, console

# set the log level for these components
log4j.logger.com.test=DEBUG
log4j.logger.org=ERROR
log4j.logger.org.apache.spark=ERROR
log4j.logger.org.spark-project=ERROR
log4j.logger.org.apache.hadoop=ERROR
log4j.logger.io.netty=ERROR
log4j.logger.org.apache.zookeeper=ERROR

# add a ConsoleAppender to the logger stdout to write to the console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
# use a simple message format
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

log4j.xml

<?xml version="1.0" encoding="UTF-8" ?>_x000D_
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">_x000D_
_x000D_
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">_x000D_
   <appender name="console" class="org.apache.log4j.ConsoleAppender">_x000D_
    <param name="Target" value="System.out"/>_x000D_
    <layout class="org.apache.log4j.PatternLayout">_x000D_
    <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n" />_x000D_
    </layout>_x000D_
  </appender>_x000D_
    <logger name="org.apache.spark">_x000D_
        <level value="error" />_x000D_
    </logger>_x000D_
    <logger name="org.spark-project">_x000D_
        <level value="error" />_x000D_
    </logger>_x000D_
    <logger name="org.apache.hadoop">_x000D_
        <level value="error" />_x000D_
    </logger>_x000D_
    <logger name="io.netty">_x000D_
        <level value="error" />_x000D_
    </logger>_x000D_
    <logger name="org.apache.zookeeper">_x000D_
        <level value="error" />_x000D_
    </logger>_x000D_
   <logger name="org">_x000D_
        <level value="error" />_x000D_
    </logger>_x000D_
    <root>_x000D_
        <priority value ="ERROR" />_x000D_
        <appender-ref ref="console" />_x000D_
    </root>_x000D_
</log4j:configuration>

Switch to FileAppender in log4j.xml if you want to write logs to file instead of console. LOG_DIR is a variable for logs directory which you can supply using spark-submit --conf "spark.driver.extraJavaOptions=-D.

<appender name="file" class="org.apache.log4j.DailyRollingFileAppender">_x000D_
        <param name="file" value="${LOG_DIR}"/>_x000D_
        <param name="datePattern" value="'.'yyyy-MM-dd"/>_x000D_
        <layout class="org.apache.log4j.PatternLayout">_x000D_
            <param name="ConversionPattern" value="%d [%t] %-5p %c %x - %m%n"/>_x000D_
        </layout>_x000D_
    </appender>

Another important thing to understand here is, when job is launched in distributed mode ( deploy-mode cluster and master as yarn or mesos) the log4j configuration file should exist on driver and worker nodes (log4j.configuration=file:<file path>/log4j.xml) else log4j init will complain-

log4j:ERROR Could not read configuration file [log4j.properties]. java.io.FileNotFoundException: log4j.properties (No such file or directory)

Hint on solving this problem-

Keep log4j config file in distributed file system(HDFS or mesos) and add external configuration using log4j PropertyConfigurator. or use sparkContext addFile to make it available on each node then use log4j PropertyConfigurator to reload configuration.

Simply add below param to your spark-shell OR spark-submit command

--conf "spark.driver.extraJavaOptions=-Dlog4jspark.root.logger=WARN,console"

Check exact property name (log4jspark.root.logger here) from log4j.properties file. Hope this helps, cheers!

In addition to all the above posts, here is what solved the issue for me.

Spark uses slf4j to bind to loggers. If log4j is not the first binding found, you can edit log4j.properties files all you want, the loggers are not even used. For example, this could be a possible SLF4J output:

SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/C:/Users/~/.m2/repository/org/slf4j/slf4j-simple/1.6.6/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/C:/Users/~/.m2/repository/org/slf4j/slf4j-log4j12/1.7.19/slf4j-log4j12-1.7.19.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]

So here the SimpleLoggerFactory was used, which does not care about log4j settings.

Excluding the slf4j-simple package from my project via

<dependency>
        ...
        <exclusions>
            ...
            <exclusion>
                <artifactId>slf4j-simple</artifactId>
                <groupId>org.slf4j</groupId>
            </exclusion>
        </exclusions>
    </dependency>

resolved the issue, as now the log4j logger binding is used and any setting in log4j.properties is adhered to. F.Y.I. my log4j properties file contains (besides the normal configuration)

log4j.rootLogger=WARN, stdout
...
log4j.category.org.apache.spark = WARN
log4j.category.org.apache.parquet.hadoop.ParquetRecordReader = FATAL
log4j.additivity.org.apache.parquet.hadoop.ParquetRecordReader=false
log4j.logger.org.apache.parquet.hadoop.ParquetRecordReader=OFF

Hope this helps!

sparkContext.setLogLevel("OFF")

This one worked for me. For only ERROR messages to be displayed as stdout, log4j.properties file may look like:

# Root logger option
log4j.rootLogger=ERROR, stdout
# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

NOTE: Put log4j.properties file in src/main/resources folder to be effective. And if log4j.properties doesn't exist (meaning spark is using log4j-defaults.properties file) then you can create it by going to SPARK_HOME/conf and then mv log4j.properties.template log4j.properties and then proceed with above said changes.

If you don't have the ability to edit the java code to insert the .setLogLevel() statements and you don't want yet more external files to deploy, you can use a brute force way to solve this. Just filter out the INFO lines using grep.

spark-submit --deploy-mode client --master local <rest-of-cmd> | grep -v -F "INFO"

In Python/Spark we can do:

def quiet_logs( sc ):
  logger = sc._jvm.org.apache.log4j
  logger.LogManager.getLogger("org"). setLevel( logger.Level.ERROR )
  logger.LogManager.getLogger("akka").setLevel( logger.Level.ERROR )

The after defining Sparkcontaxt 'sc' call this function by : quiet_logs( sc )

Simple to do on the command line...

spark2-submit --driver-java-options="-Droot.logger=ERROR,console" ..other options..

Answers above are correct but didn't exactly help me as there was additional information I required.

I have just setup Spark so the log4j file still had the '.template' suffix and wasn't being read. I believe that logging then defaults to Spark core logging conf.

So if you are like me and find that the answers above didn't help, then maybe you too have to remove the '.template' suffix from your log4j conf file and then the above works perfectly!

http://apache-spark-user-list.1001560.n3.nabble.com/disable-log4j-for-spark-shell-td11278.html

Another way of stopping logs completely is:

    import org.apache.log4j.Appender;
    import org.apache.log4j.BasicConfigurator;
    import org.apache.log4j.varia.NullAppender;

    public class SomeClass {

        public static void main(String[] args) {
            Appender nullAppender = new NullAppender();
            BasicConfigurator.configure(nullAppender);

            {...more code here...}

        }
    }

This worked for me. An NullAppender is

An Appender that ignores log events. (https://logging.apache.org/log4j/2.x/log4j-core/apidocs/org/apache/logging/log4j/core/appender/NullAppender.html)