How To Do Logging In Python
- Introduction
- What does logging give us?
- How to Log
- Where to Log
- Configuration
- The Right Way to Log
- About the Author
This is a plain and simple guide to logging in Python.
Read this guide to fully make sense of the Python logging landscape.
Introduction
Sooner or later, every Python application needs logging.
You probably first learned logging in Python by using the “print()” function:
print('Hello world!')
It could be that you simply want to know the system’s state:
print('Application successfully started on port 8000')
Or you need to record error messages when an exception occurs:
try:
doSomethingErrorProne()
except Exception as e:
print('Error: ' + str(e))
But there is a better way! Python bundles a utility in the standard library called logging:
import logging
logger = logging.getLogger()
logger.info('Hello world!')
But why is this better? Why have 3 lines when one will suffice?
What does logging give us?
Using the standard library’s logging module gives us several things:
- Message formatting (using LogRecord attributes)
- Timestamp
- Process & thread ID
- Server & service ID
- Log “levels” or severity
- Control of log severity (both in first party code & 3rd party code that uses logging)
- Guarantees around flushing stdout/stderr
This last one is a little known fact–just because you print() something, it doesn’t mean your program will print it out (source).
Using the logging API to emit your logs helps guarantee that your program logs every message you intend it to (source)!
How to Log
Beginning with a simple example, you can incorporate logging into your Python program by obtaining a logger object in your module’s global scope:
import logging
logger = logging.getLogger()
def my_function():
logger.info(“Hello world!”)
When you obtain a logger object, you can optionally identify your logger object by name:
logger = logging.getLogger('mylogger') # a custom string (only letters, numbers and "." characters are allow)
logger = logging.getLogger(__name__) # the name of the module in which this runs
If you don’t provide a name to .getLogger() then you obtain the “root” logger of your program which has the only drawback that it can’t be granularly configured.
The documentation for Django covers this topic and the merits of naming your loggers.
Where to Log
stdout and stderr
import logging
import sys
logger = logging.getLogger()
handler = logging.StreamHandler(sys.stdout)
logger.addHandler(handler)
logger.warning('Hello')
Files & Sockets
In the same way, you can create logging handlers similar to logging.StreamHandler that write your program’s log messages to files or sockets.
In fact, given a single logger, we can add two handlers to one logger so that log messages are directed to two or more different destinations:
logger.addHandler(logging.StreamHandler(sys.stdout))
logger.addHandler(logging.FileHandler('myfile.out'))
There are a number of other useful loggers covered here.
Your choice of output destination depends on what type of application you’re developing.
For web services, my suggestion and what is suggested as a best practice by 12 Factor Apps is to always log to stdout.
This makes development easier, and simplifies log management when deploying to production hosting environments.
Configuration
We’ve seen a few ways of customizing our logger behavior–here is some more information about how to do it in a declarative way.
One way is to load configuration from a file, e.g. logging.conf.fileConfig('yourlogging.conf') (source).
Another is to read configuration from a dict, e.g. logging.conf.dictConfig(configDict) (source).
I am not going to cover these options in detail, just follow the source links to read more if you are so inclined.
The Right Way to Log
Given the various ways of configuring and invoking logging, how should you integrate logging with your Python program?
Here are a few principles to follow:
- Make logs usable, i.e. publish your software’s logs to a dedicated service for viewing and searching them. This could be a self-hosted Kibana dashboard, Amazon CloudWatch, Google Cloud Logging.
- Be careful of what you log! Logs are a common security vulnerability–where private data (customer emails, passwords, payment info) and private credentials could leak. Government regulations like GDPR and CCPA may make your business obliged to purge log data that references a given user of your software–depending on how you are archiving your logs, this type of data purging may be very labor-intensive.
- Use logs as part of your monitoring pipeline–treat your logs as data to be analyzed & monitored: how many requests is my service receiving? Per second, per minute, per day? What percentage are failures?
For more info about data analytics pipelines, see my friend Ben Weber’s free ebook: Data Science for Startups
About the Author
Max Mautner is a Senior Software Engineer at Netflix, where he builds high-scale services for messaging with customers.