Hi all,
I was thinking that since logging performs I/O, there could be ways to improve its performance.
First, I was thinking that with the new free threaded python, it might be a good idea to have python logging running in its own thread.
Also, perhaps an asyncio interface to logger might be useful? But if the idea is to eventually make free threaded python the default, maybe that might not be such a great idea since it would involve large scale changes to existing code bases.
Thoughts?
Do you have a benchmark that your proposed changes should improve?
There’s not really a good reason to make logging async, the actual logging calls are negligible. You can use the recipe documented in the logging cookbook to ensure the I/O is done in a separate thread in applications that benefit from this.
I didn’t know that the QueueListener class existed. Now I do 
You will find that unless you are producing a huge amount of logging that the OS (linux, windows, etc) will take the data from the python process and complete the write in micro seconds. Only if you do a fsync to force the data to disk is there any slow down.
Edit: How do I know this? Because I measured it on a heavly loaded production system.
Usually more time is taken in formatting the buffer to log then writing the buffer to the disk (via the kernel’s buffers).
Intuition is a almost always wrong about performance issues.
Benchmark what is important and test any change you think may help.
Only accept the change if the benchmark improves and the maintenance cost of the change is also acceptable.
You may be interested in looking at picologging which has drop-in API compatibility as a design goal, but aims to be much faster than the stdlib. It hasn’t had a PyPI release for a while, but it seems to still have some activity on GitHub. Personally, I do not run into problems with the performance of logging, but I can understand why improvements are desirable in very large applications.
I’m not sure what the benefit of an asyncio interface would be, I can’t imagine why I would ever communicate with the logging system. It seems possible that some of the guts would benefit from concurrency but I don’t have any particular thoughts about that.
If don’t know if this fits your needs, but you can significantly reduce I/O overhead by writing log messages in batches with MemoryHandler.