[python] Using logging in multiple modules

I have a small python project that has the following structure -

Project 
 -- pkg01
   -- test01.py
 -- pkg02
   -- test02.py
 -- logging.conf

I plan to use the default logging module to print messages to stdout and a log file. To use the logging module, some initialization is required -

import logging.config

logging.config.fileConfig('logging.conf')
logger = logging.getLogger('pyApp')

logger.info('testing')

At present, I perform this initialization in every module before I start logging messages. Is it possible to perform this initialization only once in one place such that the same settings are reused by logging all over the project?

This question is related to python logging config

The answer is


There are several answers. i ended up with a similar yet different solution that makes sense to me, maybe it will make sense to you as well. My main objective was to be able to pass logs to handlers by their level (debug level logs to the console, warnings and above to files):

from flask import Flask
import logging
from logging.handlers import RotatingFileHandler

app = Flask(__name__)

# make default logger output everything to the console
logging.basicConfig(level=logging.DEBUG)

rotating_file_handler = RotatingFileHandler(filename="logs.log")
rotating_file_handler.setLevel(logging.INFO)

app.logger.addHandler(rotating_file_handler)

created a nice util file named logger.py:

import logging

def get_logger(name):
    return logging.getLogger("flask.app." + name)

the flask.app is a hardcoded value in flask. the application logger is always starting with flask.app as its the module's name.

now, in each module, i'm able to use it in the following mode:

from logger import get_logger
logger = get_logger(__name__)

logger.info("new log")

This will create a new log for "app.flask.MODULE_NAME" with minimum effort.


I always do it as below.

Use a single python file to config my log as singleton pattern which named 'log_conf.py'

#-*-coding:utf-8-*-

import logging.config

def singleton(cls):
    instances = {}
    def get_instance():
        if cls not in instances:
            instances[cls] = cls()
        return instances[cls]
    return get_instance()

@singleton
class Logger():
    def __init__(self):
        logging.config.fileConfig('logging.conf')
        self.logr = logging.getLogger('root')

In another module, just import the config.

from log_conf import Logger

Logger.logr.info("Hello World")

This is a singleton pattern to log, simply and efficiently.


You could also come up with something like this!

def get_logger(name=None):
    default = "__app__"
    formatter = logging.Formatter('%(levelname)s: %(asctime)s %(funcName)s(%(lineno)d) -- %(message)s',
                              datefmt='%Y-%m-%d %H:%M:%S')
    log_map = {"__app__": "app.log", "__basic_log__": "file1.log", "__advance_log__": "file2.log"}
    if name:
        logger = logging.getLogger(name)
    else:
        logger = logging.getLogger(default)
    fh = logging.FileHandler(log_map[name])
    fh.setFormatter(formatter)
    logger.addHandler(fh)
    logger.setLevel(logging.DEBUG)
    return logger

Now you could use multiple loggers in same module and across whole project if the above is defined in a separate module and imported in other modules were logging is required.

a=get_logger("__app___")
b=get_logger("__basic_log__")
a.info("Starting logging!")
b.debug("Debug Mode")

@Yarkee's solution seemed better. I would like to add somemore to it -

class Singleton(type):
    _instances = {}

    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances.keys():
            cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
        return cls._instances[cls]


class LoggerManager(object):
    __metaclass__ = Singleton

    _loggers = {}

    def __init__(self, *args, **kwargs):
        pass

    @staticmethod
    def getLogger(name=None):
        if not name:
            logging.basicConfig()
            return logging.getLogger()
        elif name not in LoggerManager._loggers.keys():
            logging.basicConfig()
            LoggerManager._loggers[name] = logging.getLogger(str(name))
        return LoggerManager._loggers[name]    


log=LoggerManager().getLogger("Hello")
log.setLevel(level=logging.DEBUG)

So LoggerManager can be a pluggable to the entire application. Hope it makes sense and value.


The best practice would be to create a module separately which has only one method whose task we be to give a logger handler to the the calling method. Save this file as m_logger.py

import logger, logging

def getlogger():
    # logger
    logger = logging.getLogger(__name__)
    logger.setLevel(logging.DEBUG)
    # create console handler and set level to debug
    #ch = logging.StreamHandler()
    ch = logging.FileHandler(r'log.txt')
    ch.setLevel(logging.DEBUG)
    # create formatter
    formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
    # add formatter to ch
    ch.setFormatter(formatter)
    # add ch to logger
    logger.addHandler(ch)
    return logger

Now call the getlogger() method whenever logger handler is needed.

from m_logger import getlogger
logger = getlogger()
logger.info('My mssg')

Actually every logger is a child of the parent's package logger (i.e. package.subpackage.module inherits configuration from package.subpackage), so all you need to do is just to configure the root logger. This can be achieved by logging.config.fileConfig (your own config for loggers) or logging.basicConfig (sets the root logger). Setup logging in your entry module (__main__.py or whatever you want to run, for example main_script.py. __init__.py works as well)

using basicConfig:

# package/__main__.py
import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO)

using fileConfig:

# package/__main__.py
import logging
import logging.config

logging.config.fileConfig('logging.conf')

and then create every logger using:

# package/submodule.py
# or
# package/subpackage/submodule.py
import logging
log = logging.getLogger(__name__)

log.info("Hello logging!")

For more information see Advanced Logging Tutorial.


A simple way of using one instance of logging library in multiple modules for me was following solution:

base_logger.py

import logging

logger = logging
logger.basicConfig(format='%(asctime)s - %(message)s', level=logging.INFO)

Other files

from base_logger import logger

if __name__ == '__main__':
    logger.info("This is an info message")

New to python so I don't know if this is advisable, but it works great for not re-writing boilerplate.

Your project must have an init.py so it can be loaded as a module

# Put this in your module's __init__.py
import logging.config
import sys

# I used this dictionary test, you would put:
# logging.config.fileConfig('logging.conf')
# The "" entry in loggers is the root logger, tutorials always 
# use "root" but I can't get that to work
logging.config.dictConfig({
    "version": 1,
    "formatters": {
        "default": {
            "format": "%(asctime)s %(levelname)s %(name)s %(message)s"
        },
    },
    "handlers": {
        "console": {
            "level": 'DEBUG',
            "class": "logging.StreamHandler",
            "stream": "ext://sys.stdout"
        }
    },
    "loggers": {
        "": {
            "level": "DEBUG",
            "handlers": ["console"]
        }
    }
})

def logger():
    # Get the name from the caller of this function
    return logging.getLogger(sys._getframe(1).f_globals['__name__'])

sys._getframe(1) suggestion comes from here

Then to use your logger in any other file:

from [your module name here] import logger

logger().debug("FOOOOOOOOO!!!")

Caveats:

  1. You must run your files as modules, otherwise import [your module] won't work:
    • python -m [your module name].[your filename without .py]
  2. The name of the logger for the entry point of your program will be __main__, but any solution using __name__ will have that issue.

Several of these answers suggest that at the top of a module you you do

import logging
logger = logging.getLogger(__name__)

It is my understanding that this is considered very bad practice. The reason is that the file config will disable all existing loggers by default. E.g.

#my_module
import logging

logger = logging.getLogger(__name__)

def foo():
    logger.info('Hi, foo')

class Bar(object):
    def bar(self):
        logger.info('Hi, bar')

And in your main module :

#main
import logging

# load my module - this now configures the logger
import my_module

# This will now disable the logger in my module by default, [see the docs][1] 
logging.config.fileConfig('logging.ini')

my_module.foo()
bar = my_module.Bar()
bar.bar()

Now the log specified in logging.ini will be empty, as the existing logger was disabled by fileconfig call.

While is is certainly possible to get around this (disable_existing_Loggers=False), realistically many clients of your library will not know about this behavior, and will not receive your logs. Make it easy for your clients by always calling logging.getLogger locally. Hat Tip : I learned about this behavior from Victor Lin's Website.

So good practice is instead to always call logging.getLogger locally. E.g.

#my_module
import logging

logger = logging.getLogger(__name__)

def foo():
    logging.getLogger(__name__).info('Hi, foo')

class Bar(object):
    def bar(self):
        logging.getLogger(__name__).info('Hi, bar')    

Also, if you use fileconfig in your main, set disable_existing_loggers=False, just in case your library designers use module level logger instances.


Throwing in another solution.

In my module's init.py I have something like:

# mymodule/__init__.py
import logging

def get_module_logger(mod_name):
  logger = logging.getLogger(mod_name)
  handler = logging.StreamHandler()
  formatter = logging.Formatter(
        '%(asctime)s %(name)-12s %(levelname)-8s %(message)s')
  handler.setFormatter(formatter)
  logger.addHandler(handler)
  logger.setLevel(logging.DEBUG)
  return logger

Then in each module I need a logger, I do:

# mymodule/foo.py
from [modname] import get_module_logger
logger = get_module_logger(__name__)

When the logs are missed, you can differentiate their source by the module they came from.


Examples related to python

programming a servo thru a barometer Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? python variable NameError Why my regexp for hyphenated words doesn't work? Comparing a variable with a string python not working when redirecting from bash script is it possible to add colors to python output? Get Public URL for File - Google Cloud Storage - App Engine (Python) Real time face detection OpenCV, Python xlrd.biffh.XLRDError: Excel xlsx file; not supported Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to logging

How to redirect docker container logs to a single file? Console logging for react? Hide strange unwanted Xcode logs Where are logs located? Retrieve last 100 lines logs Spring Boot - How to log all requests and responses with exceptions in single place? How do I get logs from all pods of a Kubernetes replication controller? Where is the Docker daemon log? How to log SQL statements in Spring Boot? How to do logging in React Native?

Examples related to config

How to create multiple output paths in Webpack config Get environment value in controller Is it possible to get the current spark context settings in PySpark? How to check if ZooKeeper is running or up from command prompt? Spring Boot and multiple external configuration files How change default SVN username and password to commit changes? How to use ConfigurationManager Python: How would you save a simple settings/config file? Using logging in multiple modules Why Git is not allowing me to commit even after configuration?