How to stop INFO messages displaying on spark console

Question

I d like to stop various messages that are coming on spark shell   I tried to edit the log4j properties file in order to stop these message   Here are the contents of log4j properties    Define the root logger with appender file log4j rootCategory WARN  console log4j appender console org apache log4j ConsoleAppender log4j appender console target System err log4j appender console layout org apache log4j PatternLayout log4j appender console layout ConversionPattern  d yy MM dd HH mm ss   p  c 1    m n    Settings to quiet third party logs that are too verbose log4j logger org eclipse jetty WARN log4j logger org eclipse jetty util component AbstractLifeCycle ERROR log4j logger org apache spark repl SparkIMain exprTyper INFO log4j logger org apache spark repl SparkILoop SparkILoopInterpreter INFO   But messages are still getting displayed on the console   Here are some example messages  15 01 05 15 11 45 INFO SparkEnv  Registering BlockManagerMaster 15 01 05 15 11 45 INFO DiskBlockManager  Created local directory at  tmp spark-local-20150105151145-b1ba 15 01 05 15 11 45 INFO MemoryStore  MemoryStore started with capacity 0 0 B  15 01 05 15 11 45 INFO ConnectionManager  Bound socket to port 44728 with id   ConnectionManagerId 192 168 100 85 44728  15 01 05 15 11 45 INFO BlockManagerMaster  Trying to register BlockManager 15 01 05 15 11 45 INFO BlockManagerMasterActor BlockManagerInfo  Registering block manager 192 168 100 85 44728 with 0 0 B RAM 15 01 05 15 11 45 INFO BlockManagerMaster  Registered BlockManager 15 01 05 15 11 45 INFO HttpServer  Starting HTTP Server 15 01 05 15 11 45 INFO HttpBroadcast  Broadcast server star   How do I stop these

User · Answer

Right after starting spark-shell type      sc setLogLevel  ERROR     In Spark 2 0  Scala    spark   SparkSession builder getOrCreate   spark sparkContext setLogLevel  ERROR     API Docs   https   spark apache org docs 2 2 0 api scala index html org apache spark sql SparkSession  For Java   spark   SparkSession builder getOrCreate    spark sparkContext   setLogLevel  ERROR

User · Answer

An interesting idea is to use the RollingAppender as suggested here  http   shzhangji com blog 2015 05 31 spark-streaming-logging-configuration  so that you don t  polute  the console space  but still be able to see the results under  YOUR LOG PATH HERE   dm logging name  log       log4j rootLogger INFO  rolling  log4j appender rolling org apache log4j RollingFileAppender log4j appender rolling layout org apache log4j PatternLayout log4j appender rolling layout conversionPattern   d   p  m   c  n log4j appender rolling maxFileSize 50MB log4j appender rolling maxBackupIndex 5 log4j appender rolling file  YOUR LOG PATH HERE   dm logging name  log log4j appender rolling encoding UTF-8   Another method that solves the cause is to observe what kind of loggings do you usually have  coming from different modules and dependencies   and set for each the granularity for the logging  while turning  quiet  third party logs that are too verbose   For instance          Silence akka remoting log4j logger Remoting ERROR log4j logger akka event slf4j ERROR log4j logger org spark-project jetty server ERROR log4j logger org apache spark ERROR log4j logger com anjuke dm   dm logging level  log4j logger org eclipse jetty WARN log4j logger org eclipse jetty util component AbstractLifeCycle ERROR log4j logger org apache spark repl SparkIMain exprTyper INFO log4j logger org apache spark repl SparkILoop SparkILoopInterpreter INFO

User · Answer

sparkContext setLogLevel  OFF

User · Answer

This one worked for me  For only ERROR messages to be displayed as stdout  log4j properties file may look like     Root logger option log4j rootLogger ERROR  stdout   Direct log messages to stdout log4j appender stdout org apache log4j ConsoleAppender log4j appender stdout Target System out log4j appender stdout layout org apache log4j PatternLayout log4j appender stdout layout ConversionPattern  d yyyy-MM-dd HH mm ss   -5p  c 1   L -  m n      NOTE  Put log4j properties file in src main resources folder to be   effective     And if log4j properties doesn t exist  meaning spark is using log4j-defaults properties file  then you can create it by going to SPARK HOME conf and then mv log4j properties template log4j properties and then proceed with above said changes

User · Answer

You set disable the Logs by setting its level to OFF as follows   Logger getLogger  org   setLevel Level OFF   Logger getLogger  akka   setLevel Level OFF     or edit log file and set log level to off by just changing the following property   log4j rootCategory OFF  console

User · Answer

Another way of stopping logs completely is       import org apache log4j Appender      import org apache log4j BasicConfigurator      import org apache log4j varia NullAppender       public class SomeClass            public static void main String   args                Appender nullAppender   new NullAppender                BasicConfigurator configure nullAppender                    more code here                        This worked for me   An NullAppender is      An Appender that ignores log events   https   logging apache org log4j 2 x log4j-core apidocs org apache logging log4j core appender NullAppender html

User · Answer

I just add this line to all my pyspark scripts on top just below the import statements   SparkSession builder getOrCreate   sparkContext setLogLevel  ERROR     example header of my pyspark scripts  from pyspark sql import SparkSession  functions as fs SparkSession builder getOrCreate   sparkContext setLogLevel  ERROR

User · Answer

Answers above are correct but didn t exactly help me as there was additional information I required   I have just setup Spark so the log4j file still had the   template  suffix and wasn t being read   I believe that logging then defaults to Spark core logging conf   So if you are like me and find that the answers above didn t help  then maybe you too have to remove the   template  suffix from your log4j conf file and then the above works perfectly   http   apache-spark-user-list 1001560 n3 nabble com disable-log4j-for-spark-shell-td11278 html

User · Answer

Use below command to change log level while submitting application using spark-submit or spark-sql   spark-submit   --conf  spark driver extraJavaOptions -Dlog4j configuration file  lt file path gt  log4j xml    --conf  spark executor extraJavaOptions -Dlog4j configuration file  lt file path gt  log4j xml    Note  replace  lt file path gt  where log4j config file is stored   Log4j properties   log4j rootLogger ERROR  console    set the log level for these components log4j logger com test DEBUG log4j logger org ERROR log4j logger org apache spark ERROR log4j logger org spark-project ERROR log4j logger org apache hadoop ERROR log4j logger io netty ERROR log4j logger org apache zookeeper ERROR    add a ConsoleAppender to the logger stdout to write to the console log4j appender console org apache log4j ConsoleAppender log4j appender console layout org apache log4j PatternLayout   use a simple message format log4j appender console layout ConversionPattern  d yyyy-MM-dd HH mm ss   -5p  c 1   L -  m n   log4j xml   x000D   x000D   lt  xml version  1 0  encoding  UTF-8    gt  x000D   lt  DOCTYPE log4j configuration SYSTEM  log4j dtd  gt  x000D   x000D   lt log4j configuration xmlns log4j  http   jakarta apache org log4j   gt  x000D      lt appender name  console  class  org apache log4j ConsoleAppender  gt  x000D       lt param name  Target  value  System out   gt  x000D       lt layout class  org apache log4j PatternLayout  gt  x000D       lt param name  ConversionPattern  value   d yyyy-MM-dd HH mm ss   -5p  c 1   L -  m n    gt  x000D       lt  layout gt  x000D     lt  appender gt  x000D       lt logger name  org apache spark  gt  x000D           lt level value  error    gt  x000D       lt  logger gt  x000D       lt logger name  org spark-project  gt  x000D           lt level value  error    gt  x000D       lt  logger gt  x000D       lt logger name  org apache hadoop  gt  x000D           lt level value  error    gt  x000D       lt  logger gt  x000D       lt logger name  io netty  gt  x000D           lt level value  error    gt  x000D       lt  logger gt  x000D       lt logger name  org apache zookeeper  gt  x000D           lt level value  error    gt  x000D       lt  logger gt  x000D      lt logger name  org  gt  x000D           lt level value  error    gt  x000D       lt  logger gt  x000D       lt root gt  x000D           lt priority value   ERROR    gt  x000D           lt appender-ref ref  console    gt  x000D       lt  root gt  x000D   lt  log4j configuration gt  x000D   x000D   x000D    Switch to FileAppender in log4j xml if you want to write logs to file instead of console  LOG DIR is a variable for logs directory which you can supply using spark-submit --conf  spark driver extraJavaOptions -D    x000D   x000D   lt appender name  file  class  org apache log4j DailyRollingFileAppender  gt  x000D           lt param name  file  value    LOG DIR    gt  x000D           lt param name  datePattern  value     yyyy-MM-dd   gt  x000D           lt layout class  org apache log4j PatternLayout  gt  x000D               lt param name  ConversionPattern  value   d   t   -5p  c  x -  m n   gt  x000D           lt  layout gt  x000D       lt  appender gt  x000D   x000D   x000D    Another important thing to understand here is  when job is launched in distributed mode   deploy-mode cluster and master as yarn or mesos  the log4j configuration file should exist on driver and worker nodes  log4j configuration file  lt file path gt  log4j xml  else log4j init will complain-     log4j ERROR Could not read configuration file  log4j properties     java io FileNotFoundException  log4j properties  No such file or   directory    Hint on solving this problem-  Keep log4j config file in distributed file system HDFS or mesos  and add  external configuration using log4j PropertyConfigurator  or use sparkContext addFile to make it available on each node then use  log4j PropertyConfigurator to reload configuration

User · Answer

In Python Spark we can do   def quiet logs  sc      logger   sc  jvm org apache log4j   logger LogManager getLogger  org    setLevel  logger Level ERROR     logger LogManager getLogger  akka   setLevel  logger Level ERROR     The after defining Sparkcontaxt  sc  call this function by   quiet logs  sc

User · Answer

All the methods collected with examples  Intro  Actually  there are many ways to do it  Some are harder from others  but it is up to you which one suits you best  I will try to showcase them all      1 Programatically in your app  Seems to be the easiest  but you will need to recompile your app to change those settings  Personally  I don t like it but it works fine   Example   import org apache log4j  Level  Logger   val rootLogger   Logger getRootLogger   rootLogger setLevel Level ERROR   Logger getLogger  org apache spark   setLevel Level WARN  Logger getLogger  org spark-project   setLevel Level WARN    You can achieve much more just using log4j API  Source   Log4J Configuration Docs  Configuration section      2 Pass log4j properties during spark-submit  This one is very tricky  but not impossible  And my favorite   Log4J during app startup is always looking for and loading log4j properties file from classpath    However  when using spark-submit Spark Cluster s classpath has precedence over app s classpath  This is why putting this file in your fat-jar will not override the cluster s settings      Add -Dlog4j configuration  lt location of configuration file gt  to    spark driver extraJavaOptions  for the driver  or   spark executor extraJavaOptions  for executors         Note that if using a   file  the file  protocol should be explicitly provided  and the file   needs to exist locally on all the nodes     To satisfy the last condition  you can either upload the file to the location available for the nodes  like hdfs  or access it locally with driver if using deploy-mode client  Otherwise      upload a custom log4j properties using spark-submit  by adding it to   the --files list of files to be uploaded with the application     Source  Spark docs  Debugging  Steps   Example log4j properties     Blacklist all to warn level log4j rootCategory WARN  console  log4j appender console org apache log4j ConsoleAppender log4j appender console target System err log4j appender console layout org apache log4j PatternLayout log4j appender console layout ConversionPattern  d yy MM dd HH mm ss   p  c 1    m n    Whitelist our app to info    log4j logger com github atais INFO   Executing spark-submit  for cluster mode   spark-submit       --master yarn       --deploy-mode cluster       --conf  spark driver extraJavaOptions -Dlog4j configuration file log4j properties        --conf  spark executor extraJavaOptions -Dlog4j configuration file log4j properties        --files   absolute path to your log4j properties        --class com github atais Main        SparkApp jar    Note that you must use --driver-java-options if using client mode  Spark docs  Runtime env  Executing spark-submit  for client mode   spark-submit       --master yarn       --deploy-mode client       --driver-java-options  -Dlog4j configuration file  absolute path to your log4j properties       --conf  spark executor extraJavaOptions -Dlog4j configuration file log4j properties        --files   absolute path to your log4j properties        --class com github atais Main        SparkApp jar    Notes    Files uploaded to spark-cluster with --files will be available at root dir  so there is no need to add any path in file log4j properties   Files listed in --files must be provided with absolute path  file  prefix in configuration URI is mandatory       3 Edit cluster s conf log4j properties  This changes global logging configuration file      update the  SPARK CONF DIR log4j properties file and it will be   automatically uploaded along with the other configurations    Source  Spark docs  Debugging  To find your SPARK CONF DIR you can use spark-shell   atais cluster    spark-shell  Welcome to                                                                                                                    version 2 1 1               scala gt  System getenv  SPARK CONF DIR   res0  String    var lib spark latest conf   Now just edit  var lib spark latest conf log4j properties  with example from method  2  and all your apps will share this configuration      4 Override configuration directory  If you like the solution  3  but want to customize it per application  you can actually copy conf folder  edit it contents and specify as the root configuration during spark-submit      To specify a different configuration directory other than the default    SPARK HOME conf     you can set SPARK CONF DIR  Spark will use the configuration files  spark-defaults conf  spark-env sh  log4j properties  etc  from this directory    Source  Spark docs  Configuration  Steps    Copy cluster s conf folder  more info  method  3  Edit log4j properties in that folder  example in method  2  Set SPARK CONF DIR to this folder  before executing spark-submit  example   export SPARK CONF DIR  absolute path to custom conf  spark-submit       --master yarn       --deploy-mode cluster       --class com github atais Main        SparkApp jar       Conclusion  I am not sure if there is any other method  but I hope this covers the topic from A to Z  If not  feel free to ping me in the comments   Enjoy your way

User · Answer

In addition to all the above posts  here is what solved the issue for me   Spark uses slf4j to bind to loggers  If log4j is not the first binding found  you can edit log4j properties files all you want  the loggers are not even used  For example  this could be a possible SLF4J output      SLF4J  Class path contains multiple SLF4J bindings    SLF4J  Found binding in  jar file  C  Users    m2 repository org slf4j slf4j-simple 1 6 6 slf4j-simple-1 6 6 jar  org slf4j impl StaticLoggerBinder class    SLF4J  Found binding in  jar file  C  Users    m2 repository org slf4j slf4j-log4j12 1 7 19 slf4j-log4j12-1 7 19 jar  org slf4j impl StaticLoggerBinder class    SLF4J  See http   www slf4j org codes html multiple bindings for an explanation    SLF4J  Actual binding is of type  org slf4j impl SimpleLoggerFactory    So here the SimpleLoggerFactory was used  which does not care about log4j settings   Excluding the slf4j-simple package from my project via   lt dependency gt                       lt exclusions gt                               lt exclusion gt                   lt artifactId gt slf4j-simple lt  artifactId gt                   lt groupId gt org slf4j lt  groupId gt               lt  exclusion gt           lt  exclusions gt       lt  dependency gt    resolved the issue  as now the log4j logger binding is used and any setting in log4j properties is adhered to  F Y I  my log4j properties file contains  besides the normal configuration   log4j rootLogger WARN  stdout     log4j category org apache spark   WARN log4j category org apache parquet hadoop ParquetRecordReader   FATAL log4j additivity org apache parquet hadoop ParquetRecordReader false log4j logger org apache parquet hadoop ParquetRecordReader OFF   Hope this helps

User · Answer

Simply add below param to your spark-shell OR spark-submit command  --conf  spark driver extraJavaOptions -Dlog4jspark root logger WARN console    Check exact property name  log4jspark root logger here  from log4j properties file  Hope this helps  cheers

User · Answer

Simple to do on the command line     spark2-submit --driver-java-options  -Droot logger ERROR console    other options

User · Answer

tl dr     For Spark Context you may use   sc setLogLevel  lt logLevel gt         where loglevel can be ALL  DEBUG  ERROR  FATAL  INFO  OFF  TRACE or   WARN      Details-  Internally  setLogLevel calls org apache log4j Level toLevel logLevel  that it then uses to set using org apache log4j LogManager getRootLogger   setLevel level       You may directly set the logging levels to OFF using   LogManager getLogger  org   setLevel Level OFF     You can set up the default logging for Spark shell in conf log4j properties  Use conf log4j properties template as a starting point   Setting Log Levels in Spark Applications  In standalone Spark applications or while in Spark Shell session  use the following   import org apache log4j  Level  Logger   Logger getLogger classOf RackResolver   getLevel Logger getLogger  org   setLevel Level OFF  Logger getLogger  akka   setLevel Level OFF    Disabling logging in log4j    Use the following in conf log4j properties to disable logging completely   log4j logger org OFF     Reference  Mastering Spark by Jacek Laskowski

User · Answer

If anyone else is stuck on this   nothing of the above worked for me  I had to remove   implementation group   ch qos logback   name   logback-classic   version   1 2 3  implementation group   com typesafe scala-logging   name   scala-logging  scalaVersion   version   3 9 2    from my build gradle for the logs to disappear  TLDR  Don t import any other logging frameworks  you should be fine just using org apache log4j Logger

User · Answer

Adjust conf log4j properties as described by other  log4j rootCategory ERROR  console Make sure while executing your spark job you pass --file flag with log4j properties file path If it still doesn t work you might have a jar that has log4j properties that is being called before your new log4j properties  Remove that log4j properties from jar  if appropriate

User · Answer

Edit your conf log4j properties file and change the following line   log4j rootCategory INFO  console   to  log4j rootCategory ERROR  console   Another approach would be to    Start spark-shell and type in the following   import org apache log4j Logger import org apache log4j Level  Logger getLogger  org   setLevel Level OFF  Logger getLogger  akka   setLevel Level OFF    You won t see any logs after that   Other options for Level include  all  debug  error  fatal  info  off  trace  trace int  warn  Details about each can be found in the documentation

User · Answer

Thanks  AkhlD and  Sachin Janani for suggesting changes in  conf file   Following code solved my issue   1  Added import org apache log4j  Level  Logger  in import section  2  Added following line after creation of spark context object i e  after val sc   new SparkContext conf    val rootLogger   Logger getRootLogger   rootLogger setLevel Level ERROR

User · Answer

Adding the following to the PySpark did the job for me  self spark sparkContext setLogLevel  quot ERROR quot    self spark is the spark session  self spark   spark builder getOrCreate

User · Answer

If you don t have the ability to edit the java code to insert the  setLogLevel   statements and you don t want yet more external files to deploy  you can use a brute force way to solve this  Just filter out the INFO lines using grep    spark-submit --deploy-mode client --master local  lt rest-of-cmd gt    grep -v -F  INFO

[apache-spark] How to stop INFO messages displaying on spark console?

Examples related to apache-spark

Examples related to log4j

Examples related to spark-submit