How To Do Logging In Python

Logging

This is a plain and simple guide to logging in Python.

Read this guide to fully make sense of the Python logging landscape.

Introduction

Sooner or later, every Python application needs logging.

You probably first learned logging in Python by using the “print()” function:

print('Hello world!')

It could be that you simply want to know the system’s state:

print('Application successfully started on port 8000')

Or you need to record error messages when an exception occurs:

try:
    doSomethingErrorProne()
except Exception as e:
    print('Error: ' + str(e))

But there is a better way! Python bundles a utility in the standard library called logging:

import logging
logger = logging.getLogger()
logger.info('Hello world!')

But why is this better? Why have 3 lines when one will suffice?

What does logging give us?

Using the standard library’s logging module gives us several things:

This last one is a little known fact–just because you print() something, it doesn’t mean your program will print it out (source).

Using the logging API to emit your logs helps guarantee that your program logs every message you intend it to (source)!

How to Log

Beginning with a simple example, you can incorporate logging into your Python program by obtaining a logger object in your module’s global scope:

import logging
logger = logging.getLogger()
def my_function():
    logger.info(“Hello world!”)

When you obtain a logger object, you can optionally identify your logger object by name:

logger = logging.getLogger('mylogger')   # a custom string (only letters, numbers and "." characters are allow)
logger = logging.getLogger(__name__)    # the name of the module in which this runs

If you don’t provide a name to .getLogger() then you obtain the “root” logger of your program which has the only drawback that it can’t be granularly configured.

The documentation for Django covers this topic and the merits of naming your loggers.

Where to Log

stdout and stderr

import logging
import sys
logger = logging.getLogger()
handler = logging.StreamHandler(sys.stdout)
logger.addHandler(handler)

logger.warning('Hello')

Source

Files & Sockets

In the same way, you can create logging handlers similar to logging.StreamHandler that write your program’s log messages to files or sockets.

In fact, given a single logger, we can add two handlers to one logger so that log messages are directed to two or more different destinations:

logger.addHandler(logging.StreamHandler(sys.stdout))
logger.addHandler(logging.FileHandler('myfile.out'))

There are a number of other useful loggers covered here.

Your choice of output destination depends on what type of application you’re developing.

For web services, my suggestion and what is suggested as a best practice by 12 Factor Apps is to always log to stdout.

This makes development easier, and simplifies log management when deploying to production hosting environments.

Configuration

We’ve seen a few ways of customizing our logger behavior–here is some more information about how to do it in a declarative way.

One way is to load configuration from a file, e.g. logging.conf.fileConfig('yourlogging.conf') (source).

Another is to read configuration from a dict, e.g. logging.conf.dictConfig(configDict) (source).

I am not going to cover these options in detail, just follow the source links to read more if you are so inclined.

The Right Way to Log

Given the various ways of configuring and invoking logging, how should you integrate logging with your Python program?

Here are a few principles to follow:

For more info about data analytics pipelines, see my friend Ben Weber’s free ebook: Data Science for Startups

About the Author

Max Mautner is a Senior Software Engineer at Netflix, where he builds high-scale services for messaging with customers.

· programming, python, logging, data