Using logging in multiple modules

Question

I have a small python project that has the following structure -   Project   -- pkg01    -- test01 py  -- pkg02    -- test02 py  -- logging conf   I plan to use the default logging module to print messages to stdout and a log file  To use the logging module  some initialization is required -   import logging config  logging config fileConfig  logging conf   logger   logging getLogger  pyApp    logger info  testing     At present  I perform this initialization in every module before I start logging messages  Is it possible to perform this initialization only once in one place such that the same settings are reused by logging all over the project

User · Answer

Several of these answers suggest that at the top of a module you you do

import logging
logger = logging.getLogger(__name__)

It is my understanding that this is considered very bad practice. The reason is that the file config will disable all existing loggers by default. E.g.

#my_module
import logging

logger = logging.getLogger(__name__)

def foo():
    logger.info('Hi, foo')

class Bar(object):
    def bar(self):
        logger.info('Hi, bar')

And in your main module :

#main
import logging

# load my module - this now configures the logger
import my_module

# This will now disable the logger in my module by default, [see the docs][1] 
logging.config.fileConfig('logging.ini')

my_module.foo()
bar = my_module.Bar()
bar.bar()

Now the log specified in logging.ini will be empty, as the existing logger was disabled by fileconfig call.

While is is certainly possible to get around this (disable_existing_Loggers=False), realistically many clients of your library will not know about this behavior, and will not receive your logs. Make it easy for your clients by always calling logging.getLogger locally. Hat Tip : I learned about this behavior from Victor Lin's Website.

So good practice is instead to always call logging.getLogger locally. E.g.

#my_module
import logging

logger = logging.getLogger(__name__)

def foo():
    logging.getLogger(__name__).info('Hi, foo')

class Bar(object):
    def bar(self):
        logging.getLogger(__name__).info('Hi, bar')

Also, if you use fileconfig in your main, set disable_existing_loggers=False, just in case your library designers use module level logger instances.

User · Answer

New to python so I don t know if this is advisable  but it works great for not re-writing boilerplate   Your project must have an init py so it can be loaded as a module    Put this in your module s   init   py import logging config import sys    I used this dictionary test  you would put    logging config fileConfig  logging conf     The    entry in loggers is the root logger  tutorials always    use  root  but I can t get that to work logging config dictConfig        version   1       formatters              default                  format      asctime s   levelname s   name s   message s                         handlers              console                  level    DEBUG                class    logging StreamHandler                stream    ext   sys stdout                        loggers                                level    DEBUG                handlers     console                       def logger          Get the name from the caller of this function     return logging getLogger sys  getframe 1  f globals    name        sys  getframe 1  suggestion comes from here  Then to use your logger in any other file   from  your module name here  import logger  logger   debug  FOOOOOOOOO        Caveats    You must run your files as modules  otherwise import  your module  won t work    python -m  your module name   your filename without  py   The name of the logger for the entry point of your program will be   main    but any solution using   name   will have that issue

User · Answer

You could also come up with something like this   def get logger name None       default      app        formatter   logging Formatter    levelname s    asctime s   funcName s   lineno d  --   message s                                 datefmt   Y- m- d  H  M  S       log map       app      app log      basic log      file1 log      advance log      file2 log       if name          logger   logging getLogger name      else          logger   logging getLogger default      fh   logging FileHandler log map name       fh setFormatter formatter      logger addHandler fh      logger setLevel logging DEBUG      return logger   Now you could use multiple loggers in same module and across whole project if the above is defined in a separate module and imported in other modules were logging is required   a get logger    app      b get logger    basic log     a info  Starting logging    b debug  Debug Mode

User · Answer

The best practice would be to create a module separately which has only one method whose task we be to give a logger handler to the the calling method  Save this file as m logger py  import logger  logging  def getlogger          logger     logger   logging getLogger   name        logger setLevel logging DEBUG        create console handler and set level to debug      ch   logging StreamHandler       ch   logging FileHandler r log txt       ch setLevel logging DEBUG        create formatter     formatter   logging Formatter    asctime s -   levelname s -   message s         add formatter to ch     ch setFormatter formatter        add ch to logger     logger addHandler ch      return logger   Now call the getlogger   method whenever logger handler is needed   from m logger import getlogger logger   getlogger   logger info  My mssg

User · Answer

I always do it as below   Use a single python file to config my log as singleton pattern which named  log conf py    - -coding utf-8- -  import logging config  def singleton cls       instances          def get instance            if cls not in instances              instances cls    cls           return instances cls      return get instance     singleton class Logger        def   init   self           logging config fileConfig  logging conf           self logr   logging getLogger  root     In another module  just import the config   from log conf import Logger  Logger logr info  Hello World     This is a singleton pattern to log  simply and efficiently

User · Answer

Best practice is  in each module  to have a logger defined like this   import logging logger   logging getLogger   name      near the top of the module  and then in other code in the module do e g   logger debug  My message with  s    variable data     If you need to subdivide logging activity inside a module  use e g   loggerA   logging getLogger   name       A   loggerB   logging getLogger   name       B     and log to loggerA and loggerB as appropriate   In your main program or programs  do e g    def main         your program code   if   name         main         import logging config     logging config fileConfig   path to logging conf       main     or   def main        import logging config     logging config fileConfig   path to logging conf         your program code  if   name         main         main     See here for logging from multiple modules  and here for logging configuration for code which will be used as a library module by other code   Update  When calling fileConfig    you may want to specify disable existing loggers False if you re using Python 2 6 or later  see the docs for more information   The default value is True for backward compatibility  which causes all existing loggers to be disabled by fileConfig   unless they or their ancestor are explicitly named in the configuration  With the value set to False  existing loggers are left alone  If using Python 2 7 Python 3 2 or later  you may wish to consider the dictConfig   API which is better than fileConfig   as it gives more control over the configuration

User · Answer

There are several answers  i ended up with a similar yet different solution that makes sense to me  maybe it will make sense to you as well  My main objective was to be able to pass logs to handlers by their level  debug level logs to the console  warnings and above to files    from flask import Flask import logging from logging handlers import RotatingFileHandler  app   Flask   name       make default logger output everything to the console logging basicConfig level logging DEBUG   rotating file handler   RotatingFileHandler filename  logs log   rotating file handler setLevel logging INFO   app logger addHandler rotating file handler    created a nice util file named logger py   import logging  def get logger name       return logging getLogger  flask app     name    the flask app is a hardcoded value in flask  the application logger is always starting with flask app as its the module s name   now  in each module  i m able to use it in the following mode   from logger import get logger logger   get logger   name     logger info  new log     This will create a new log for  app flask MODULE NAME  with minimum effort

User · Answer

Throwing in another solution    In my module s init py I have something like     mymodule   init   py import logging  def get module logger mod name     logger   logging getLogger mod name    handler   logging StreamHandler     formatter   logging Formatter             asctime s   name -12s   levelname -8s   message s     handler setFormatter formatter    logger addHandler handler    logger setLevel logging DEBUG    return logger   Then in each module I need a logger  I do     mymodule foo py from  modname  import get module logger logger   get module logger   name      When the logs are missed  you can differentiate their source by the module they came from

User · Answer

Yarkee s solution seemed better  I would like to add somemore to it -  class Singleton type        instances           def   call   cls   args    kwargs           if cls not in cls  instances keys                cls  instances cls    super Singleton  cls    call    args    kwargs          return cls  instances cls    class LoggerManager object         metaclass     Singleton       loggers           def   init   self   args    kwargs           pass       staticmethod     def getLogger name None           if not name              logging basicConfig               return logging getLogger           elif name not in LoggerManager  loggers keys                logging basicConfig               LoggerManager  loggers name    logging getLogger str name           return LoggerManager  loggers name        log LoggerManager   getLogger  Hello   log setLevel level logging DEBUG    So LoggerManager can be a pluggable to the entire application  Hope it makes sense and value

User · Answer

A simple way of using one instance of logging library in multiple modules for me was following solution   base logger py  import logging  logger   logging logger basicConfig format    asctime s -   message s   level logging INFO    Other files  from base logger import logger  if   name         main         logger info  This is an info message

User · Answer

Actually every logger is a child of the parent s package logger  i e  package subpackage module inherits configuration from package subpackage   so all you need to do is just to configure the root logger  This can be achieved by logging config fileConfig  your own config for loggers  or logging basicConfig  sets the root logger   Setup logging in your entry module    main   py or whatever you want to run  for example main script py    init   py works as well  using basicConfig    package   main   py import logging import sys  logging basicConfig stream sys stdout  level logging INFO   using fileConfig    package   main   py import logging import logging config  logging config fileConfig  logging conf    and then create every logger using    package submodule py   or   package subpackage submodule py import logging log   logging getLogger   name     log info  quot Hello logging  quot    For more information see Advanced Logging Tutorial

[python] Using logging in multiple modules

Examples related to python

Examples related to logging

Examples related to config