February 15, 2021

How To Do Logging In Python

Logging

This is a plain and simple guide to logging in Python.

Read this guide to fully make sense of the Python logging landscape.

Introduction

Sooner or later, every Python application needs logging.

You probably first learned logging in Python by using the “print()” function:

print('Hello world!')

It could be that you simply want to know the system’s state:

print('Application successfully started on port 8000')

Or you need to record error messages when an exception occurs:

try:
    doSomethingErrorProne()
except Exception as e:
    print('Error: ' + str(e))

But there is a better way! Python bundles a utility in the standard library called logging:

import logging
logger = logging.getLogger()
logger.info('Hello world!')

But why is this better? Why have 3 lines when one will suffice?

What does logging give us?

Using the standard library’s logging module gives us several things:

This last one is a little known fact–just because you print() something, it doesn’t mean your program will print it out (source).

Using the logging API to emit your logs helps guarantee that your program logs every message you intend it to (source)!

How to Log

Beginning with a simple example, you can incorporate logging into your Python program by obtaining a logger object in your module’s global scope:

import logging
logger = logging.getLogger()
def my_function():
    logger.info(Hello world!”)

When you obtain a logger object, you can optionally identify your logger object by name:

logger = logging.getLogger('mylogger')   # a custom string (only letters, numbers and "." characters are allow)
logger = logging.getLogger(__name__)    # the name of the module in which this runs

If you don’t provide a name to .getLogger() then you obtain the “root” logger of your program which has the only drawback that it can’t be granularly configured.

The documentation for Django covers this topic and the merits of naming your loggers.

Where to Log

stdout and stderr

import logging
import sys
logger = logging.getLogger()
handler = logging.StreamHandler(sys.stdout)
logger.addHandler(handler)

logger.warning('Hello')

Source

Files & Sockets

In the same way, you can create logging handlers similar to logging.StreamHandler that write your program’s log messages to files or sockets.

In fact, given a single logger, we can add two handlers to one logger so that log messages are directed to two or more different destinations:

logger.addHandler(logging.StreamHandler(sys.stdout))
logger.addHandler(logging.FileHandler('myfile.out'))

There are a number of other useful loggers covered here.

Your choice of output destination depends on what type of application you’re developing.

For web services, my suggestion and what is suggested as a best practice by 12 Factor Apps is to always log to stdout.

This makes development easier, and simplifies log management when deploying to production hosting environments.

Configuration

We’ve seen a few ways of customizing our logger behavior–here is some more information about how to do it in a declarative way.

One way is to load configuration from a file, e.g. logging.conf.fileConfig('yourlogging.conf') (source).

Another is to read configuration from a dict, e.g. logging.conf.dictConfig(configDict) (source).

I am not going to cover these options in detail, just follow the source links to read more if you are so inclined.

The Right Way to Log

Given the various ways of configuring and invoking logging, how should you integrate logging with your Python program?

Here are a few principles to follow:

  • Make logs usable, i.e. publish your software’s logs to a dedicated service for viewing and searching them. This could be a self-hosted Kibana dashboard, Amazon CloudWatch, Google Cloud Logging.
  • Be careful of what you log! Logs are a common security vulnerability–where private data (customer emails, passwords, payment info) and private credentials could leak. Government regulations like GDPR and CCPA may make your business obliged to purge log data that references a given user of your software–depending on how you are archiving your logs, this type of data purging may be very labor-intensive.
  • Use logs as part of your monitoring pipeline–treat your logs as data to be analyzed & monitored: how many requests is my service receiving? Per second, per minute, per day? What percentage are failures?

For more info about data analytics pipelines, see my friend Ben Weber’s free ebook: Data Science for Startups

About the Author

Max Mautner is a Senior Software Engineer at Netflix, where he builds high-scale services for messaging with customers.