Logging from Multiple Modules with Python

As a beginner, you usually write your application code in a single file. But soon, you’ll find that you have hundreds or thousands of lines of code in a file, which can become hard to manage.

Then, you break up your code into multiple files. In Python, each Python file is considered a module. A group of related modules is called a package.

Why does that matter? You’ll need to add logging to your application, and most applications have multiple files. This chapter shows you how to add logging to a multi-module application.

To keep the code simpler, you will learn how to add logging using the following methods:

• Logging using the logging API

• Logging using dictConfig

You’ll start your journey using the logging API method first!

Logging Using the Logging API

The logging API is where you use logging classes, methods, and attributes to configure and log data. In this section, you will create two modules to log data from.

Here are the names of the two files:

• main.py - The main entry point of your fake application

• other_mod.py - A second module to log from

Your main.py will set up your logger object and anything else you need. The other_mod.py file will only need to call getLogger() with the appropriate name to use the logger you configured in main.py.

While this sounds a bit complicated, you’ll soon see it’s fairly straightforward. Open up your favorite Python IDE or text editor and create the main.py file.

Then enter the following code:

Here, you create a logger named “test”, give it a FileHandler, and apply a formatter. Then, you log an information message, call other_mod.add(), log another message, and return.

Pretty simple, right? You have seen 99% of this type of code already. The only new bit is calling another module. But that’s where the logging to the same logger across multiple modules comes in.

To see how that works, go ahead and create the other_mod.py file in your Python IDE and add the following code to it:

Your code imports the logging module and then uses logging.getLogger() to get the “test” logger instance. Now you can log to the same file as you were logging to in main.py. You don’t need to create a handler or a formatter or do any other setup in this module. That’s already been taken care of.

When you run main.py, you will see something like the following log output in your multi.log file:

You can look at these logs and see when a log message is written. You can also see which Python file the log messages come from. But what if you wanted to use a dictConfig? You’ll learn how to do that next!

Logging Using dictConfig

For this example, you will create a fake Python package or application that you want to add logging to. Experimenting with logging in these varied scenarios will help you add logging to your own applications.

Here is an example tree view of a folder that you will need to create on your computer:

The first step is to create a sample_package directory. Inside of that directory, you need the following items:

• __init__.py

• main.py

• settings.py

• utils directory

Then inside the utils sub-directory, you will need the following files:

• __init__.py

• minimath.py

When you dig into Python’s logging documentation and guide, you may notice that it says you can use __name__ for your logger object’s name. Using the __name__ object is a dynamic way to create separate loggers for each module you want to add logging to. Unlike the previous example, which only used the “test” logger, this one will create several logger objects.

Because you will have several loggers, your setup will be more complex. However, you will quickly see how powerful the logging module is and how granular you can be with it.

Start by re-opening your Python IDE or text editor and creating an empty __init__.py file and your new main.py file.

The empty __init__.py files may seem strange at first. However, these files tell Python that the folder they are in is now a part of a package. That means you can now import the folder itself as if it were a Python module. You can do more with these files, but that is outside the scope of this book.

Then add the following code to main.py:

Here, you import the settings module and the utils.minimath module, which you still need to create. Then, you load up the logging config from the settings module. The main() function gets a logger called __main__.

This brings up the strange quirk in Python where you see if __name__ == "__main__" at the bottom of Python files. What this conditional is doing is checking whether you ran the module directly or imported it. If you run the file directly, it’s __name__ equals "__main__", otherwise it equals the actual name of the module.

You then log out a debug message and call minimath.add(). That’s it!

You are now ready to learn how to configure your code to make this log work. In your Python IDE or text editor, create a settings.py file and save it in the same location as your main.py file from above. Then enter the following code into it:

Your dictionary configuration defines three loggers:

• “” - An empty string which maps to the root logger

• utils.minimath - The name of the submodule you create in the utils subfolder

• "__main__" - The logger to use if a module is executed directly

Your dictionary also defines two formatters:

• “fileHandler” - An instance of logging.FileHandler that writes to “settings.log”

• “default” - The default handler that writes to stdout via a logging.StreamHandler

Finally, you also configure two separate formatter objects, one for the file handler and one for the stream handler.

Next, you will need to create a utils directory in the same directory as your main.py, settings.py, and __init__.py files. Inside the utils folder, you must create an empty __init__.py file and the minimath.py file.

For the minimath.py file, you will need to add the following code:

Here, you once again create a logger using the __name__ object and add a quick informational log message to the add() function.

Now you are ready to try running main.py. Give it a try. You should see the following output in your terminal or your IDE’s console:

Here, the name emitted from main.py is __main__ while the name emitted from minimath.py is utils.minimath. The latter is a submodule. When you added that __init__.py script to the utils folder, it made the utils folder into a package name that you can import; this also allows you to import any Python files that utils contains.

If you open the settings.log file, you will see that it contains only a single log that is similar to the following:

You will need to study the configuration file closely. When you do, you will find that the “main“ logger is connected to the StreamHandler only. The “utils.minimath” is attached to BOTH the StreamHandler and the FileHandler, which is why you see the same output for that module in both the console and in the file.

If you added the if __name__ == "__main__" line to your minimath.py file, it could be run directly instead, but the output would be a little different. You can try that out on your own as an exercise.

Wrapping Up

Python’s logging module gives you great flexibility when logging from multiple modules in your code.

You learned how to do this task in a couple of different ways:

• Logging using the logging API

• Logging using dictConfig

These aren’t the only ways to log from multiple modules, but they are the most common. However, you will almost always have your logging API code in a separate module from main.py. Using a dictConfig, a fileConfig, or the logging API, you can create very powerful log handlers and formatters. You can also add filters here if you want to.

Start experimenting and see if you can add some logging to your code!