Python Logging - Working with Filters
Python’s logging module provides support for filters. You can add a Filter to a Handler or a Logger object for more sophisticated filtering than you can get through log levels.
Python’s documentation gives the following example of how you might use a logging filter:
A filter initialized with ‘A.B’ will allow events logged by loggers ‘A.B’, ‘A.B.C’, ‘A.B.C.D’, ‘A.B.D’ etc. but not ‘A.BB’, ‘B.A.B’ etc. If initialized with the empty string, all events are passed.
A handler object with a Filter object applied to it will filter events before the handler emits them. If you apply the filter to the logger, it applies before the event is sent to the handler. Why does that matter? Well, if you have descendant logger objects, they will not be filtered unless you add the filter to the descendants too.
Here is what you will learn in this article:
Creating a filter
Applying the filter to a logger
Applying the filter to a handler
Let’s get started!
Creating a filter
The first step you must understand is how to create a logging filter. There are two primary methods:
Subclassing from
logging.FilterCreating a class with a
filter()method
You will look at an example of both of these methods.
To start, open up your Python IDE or a text editor and create a new file named logging_filter.py. Then enter the following code:
Here is an example of subclassing logging.Filter. Then, you create a filter() method that takes in a Log Record instance. At that point, you have a Log Record object that you can use to get more information about it.
For this example, you check to see if the function that the Log Record emits has a name that begins with “a”. If it does, you return False. Otherwise, you return True. But what does this mysterious code do?
When you return False, it tells the logging object (or the handler) that you are filtering out any function or method whose name starts with “a” and is attempting to log. For whatever reason, you don’t need those logs right now, so they are filtered out. When you return True, those log records will get logged.
Now you’re ready to look at an example where you create your class without subclassing logging.Filter.
Create a new file named logging_filter_no_subclass.py and enter the following code:
Wait a minute! That code’s the same as the last example!!! Well, not quite. This time you’re not subclassing anything. That’s the difference. Otherwise, you’re right. It is the same!
Starting in Python 3.12, there is a third way to create a filter. You can simply use a function or other callable as a filter. The logging module will check if the filter object has a filter attribute. If it does, then that will be called. Otherwise, the logging module will assume it’s a callable and just pass in the log record as a single parameter.
Here is an example function that does the same thing as the class above:
Of course, creating a filter class or function is fun, but you probably want to see this code in action. Have no fear—that’s what you will learn next.
Applying the Filter to a Logger
The addFilter() method makes it easy to add a filter to a logger object. You’ll soon see how easy it is to take the filter from the previous section and apply it to a logger object.
Open up your IDE again and create a new file named logging_filter.py. Then enter the following code:
The first third of this code contains some imports and your filter, which is called MyFilter and subclasses logging.Filter. You follow this up with two functions, a() and b(). All these functions log a message using the debug log level.
The last block of code creates a logger object. You set the logger to output to stderr, which is the same as creating a StreamHandler. Then you add the filter to your logger object like this: logger.addFilter(MyFilter()).
Pretty easy, right? When you run this code, you will see the following output in your terminal:
DEBUG:filter_test:Message from BYour filter was a success! You only logged from function b() and the logs from a() were filtered out.
Applying the Filter to a Handler
Applying a filter to a handler is similar to adding one to a logger. For this one, you will create a filter that adds two new attributes to the Log Record object using monkey patching to provide contextual information about the logs.
The following example is based on one from Python’s logging cookbook.
Open up your Python IDE or text editor and create a new file named context_filter.py. Then enter the following code:
Your filter() method modifies the log record to add a user and language attribute. You choose a random user and language and set them. Then in the following code, you create your boilerplate:
logger - a logger named “test”
handler - a
StreamHandlerinstance that emits to stdoutmy_filter - an instance of
ContextFilterformatter - a formatter object where you insert your new logger attributes
Pay attention in this code. Here you are doing handler.addFilter(my_filter) instead of logger.addFilter(). If you were using multiple handlers, the other handler would not have the filter applied to it.
The last few lines of your code log out ten lines at random log levels.
When you run this code, you will get something similar to this, albeit your results will be considerably different since the users and languages are randomly picked:
test - INFO User: Rodrigo Lang: C++ This is an info message with silly parameters
test - CRITICAL User: Rodrigo Lang: PHP A message with CRITICAL level
test - CRITICAL User: Mike Lang: Python A message with CRITICAL level
test - WARNING User: Mike Lang: Python A message with WARNING level
test - CRITICAL User: Stephen Lang: Ruby A message with CRITICAL level
test - ERROR User: Mike Lang: Ruby A message with ERROR level
test - ERROR User: Stephen Lang: Python A message with ERROR level
test - CRITICAL User: Stephen Lang: C++ A message with CRITICAL level
test - INFO User: Rodrigo Lang: PHP A message with INFO level
test - WARNING User: Rodrigo Lang: Ruby A message with WARNING level
test - DEBUG User: Rodrigo Lang: Python A message with DEBUG level
Hmmm. That output looks like some fields get set to a static size. How did that happen? Go back to the code and examine the formatting string. You will notice a “-8s” on some of the formatted objects (e.g., “%(levelname)-8s”). That tells the Formatter object to make that field eight characters wide. If the inserted string is less than eight characters, your code will pad it with spaces.
It looks good, right? Try changing the number of characters to other values and formatting the various fields as you see fit. Practicing that is a great way to cement what you’re learning.
Wrapping Up
Logging filters allow you to refine what ends up in your logs further. Most engineers rely on log levels to control what goes in their logs, but you can use filters to give you even more fine-grained control.
In this tutorial, you learned about the following:
Creating a filter
Applying the filter to a logger
Applying the filter to a handler
Remember that filters only apply to the loggers or handlers you attach them to. If you notice that some items are slipping through, you probably forgot to add the filter to a descendant logger object or to a handler.
As always, practice is the best way to learn to use something new. Try logging filters and see if they can help improve your logging!
Where to Purchase
You can get my Python Logging book at the following websites:
Download the Code
You can get the code for the book from GitHub: