This is a plain and simple guide to logging in Python.
Read this guide to fully make sense of the Python logging landscape.
Sooner or later, every Python application needs logging.
You probably first learned logging in Python by using the “print()” function:
print('Hello world!')
It could be that you simply want to know the system’s state:
print('Application successfully started on port 8000')
Or you need to record error messages when an exception occurs:
try:
doSomethingErrorProne()
except Exception as e:
print('Error: ' + str(e))
But there is a better way! Python bundles a utility in the standard library called logging:
import logging
logger = logging.getLogger()
logger.info('Hello world!')
But why is this better? Why have 3 lines when one will suffice?
Using the standard library’s logging module gives us several things:
This last one is a little known fact–just because you print()
something, it doesn’t mean your program will print it out (source).
Using the logging
API to emit your logs helps guarantee that your program logs every message you intend it to (source)!
Beginning with a simple example, you can incorporate logging into your Python program by obtaining a logger object in your module’s global scope:
import logging
logger = logging.getLogger()
def my_function():
logger.info(“Hello world!”)
When you obtain a logger object, you can optionally identify your logger object by name:
logger = logging.getLogger('mylogger') # a custom string (only letters, numbers and "." characters are allow)
logger = logging.getLogger(__name__) # the name of the module in which this runs
If you don’t provide a name to .getLogger()
then you obtain the “root” logger of your program which has the only drawback that it can’t be granularly configured.
The documentation for Django covers this topic and the merits of naming your loggers.
import logging
import sys
logger = logging.getLogger()
handler = logging.StreamHandler(sys.stdout)
logger.addHandler(handler)
logger.warning('Hello')
In the same way, you can create logging handlers similar to logging.StreamHandler
that write your program’s log messages to files or sockets.
In fact, given a single logger, we can add two handlers to one logger so that log messages are directed to two or more different destinations:
logger.addHandler(logging.StreamHandler(sys.stdout))
logger.addHandler(logging.FileHandler('myfile.out'))
There are a number of other useful loggers covered here.
Your choice of output destination depends on what type of application you’re developing.
For web services, my suggestion and what is suggested as a best practice by 12 Factor Apps is to always log to stdout.
This makes development easier, and simplifies log management when deploying to production hosting environments.
We’ve seen a few ways of customizing our logger behavior–here is some more information about how to do it in a declarative way.
One way is to load configuration from a file, e.g. logging.conf.fileConfig('yourlogging.conf')
(source).
Another is to read configuration from a dict
, e.g. logging.conf.dictConfig(configDict)
(source).
I am not going to cover these options in detail, just follow the source links to read more if you are so inclined.
Given the various ways of configuring and invoking logging, how should you integrate logging with your Python program?
Here are a few principles to follow:
For more info about data analytics pipelines, see my friend Ben Weber’s free ebook: Data Science for Startups
Max Mautner is a Senior Software Engineer at Netflix, where he builds high-scale services for messaging with customers.