# Improving logging performance **URL:** \ **Category:** Ideas\ **Created:** [December 11, 2024, 9:49pm UTC](https://discuss.python.org/t/improving-logging-performance/74001 "2024-12-11T21:49:45Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![rlippmann](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rlippmann/32/12560_2.png) [@rlippmann](https://discuss.python.org/u/rlippmann)\ **Post date:** [December 11, 2024, 9:49pm UTC](https://discuss.python.org/t/improving-logging-performance/74001/1 "2024-12-11T21:49:45Z")
Hi all, I was thinking that since logging performs I/O, there could be ways to improve its performance. First, I was thinking that with the new free threaded python, it might be a good idea to have python logging running in its own thread. Also, perhaps an asyncio interface to logger might be useful? But if the idea is to eventually make free threaded python the default, maybe that might not be such a great idea since it would involve large scale changes to existing code bases. Thoughts? ---
**Author:** ![barry-scott](https://avatars.discourse-cdn.com/v4/letter/b/e9c0ed/32.png) [@barry-scott](https://discuss.python.org/u/barry-scott)\ **Post date:** [December 11, 2024, 10:10pm UTC](https://discuss.python.org/t/improving-logging-performance/74001/2 "2024-12-11T22:10:25Z")
Do you have a benchmark that your proposed changes should improve? ---
**Author:** ![mikeshardmind](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/mikeshardmind/32/14381_2.png) [@mikeshardmind](https://discuss.python.org/u/mikeshardmind)\ **Post date:** [December 11, 2024, 10:25pm UTC](https://discuss.python.org/t/improving-logging-performance/74001/3 "2024-12-11T22:25:03Z")
There’s not really a good reason to make logging async, the actual logging calls are negligible. You can use the recipe documented [in the logging cookbook](https://docs.python.org/3/howto/logging-cookbook.html#dealing-with-handlers-that-block) to ensure the I/O is done in a separate thread in applications that benefit from this. ---
**Author:** ![rlippmann](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rlippmann/32/12560_2.png) [@rlippmann](https://discuss.python.org/u/rlippmann)\ **Post date:** [December 13, 2024, 8:26pm UTC](https://discuss.python.org/t/improving-logging-performance/74001/4 "2024-12-13T20:26:53Z")
I didn’t know that the QueueListener class existed. Now I do 🙂 ---
**Author:** ![barry-scott](https://avatars.discourse-cdn.com/v4/letter/b/e9c0ed/32.png) [@barry-scott](https://discuss.python.org/u/barry-scott)\ **Post date:** [December 13, 2024, 9:16pm UTC](https://discuss.python.org/t/improving-logging-performance/74001/5 "2024-12-13T21:16:06Z")
You will find that unless you are producing a huge amount of logging that the OS (linux, windows, etc) will take the data from the python process and complete the write in micro seconds. Only if you do a fsync to force the data to disk is there any slow down. Edit: How do I know this? Because I measured it on a heavly loaded production system. Usually more time is taken in formatting the buffer to log then writing the buffer to the disk (via the kernel’s buffers). Intuition is a almost always wrong about performance issues. Benchmark what is important and test any change you think may help. Only accept the change if the benchmark improves and the maintenance cost of the change is also acceptable. ---
**Author:** ![SnoopJ](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/snoopj/32/7944_2.png) [@SnoopJ](https://discuss.python.org/u/SnoopJ)\ **Post date:** [December 14, 2024, 5:06am UTC](https://discuss.python.org/t/improving-logging-performance/74001/6 "2024-12-14T05:06:04Z")
You may be interested in looking at [`picologging`](https://pypi.org/project/picologging/) which has drop-in API compatibility as a design goal, but aims to be much faster than the stdlib. It hasn’t had a PyPI release for a while, but it seems to still have some activity on GitHub. Personally, I do not run into problems with the performance of `logging`, but I can understand why improvements are desirable in very large applications. I’m not sure what the benefit of an asyncio interface would be, I can’t imagine why I would ever communicate with the logging system. It seems possible that some of the guts would benefit from concurrency but I don’t have any particular thoughts about that. ---
**Author:** ![Wombat](https://avatars.discourse-cdn.com/v4/letter/w/e480ec/32.png) [@Wombat](https://discuss.python.org/u/Wombat)\ **Post date:** [December 15, 2024, 4:10am UTC](https://discuss.python.org/t/improving-logging-performance/74001/7 "2024-12-15T04:10:16Z")
> [@rlippmann](#): > > I was thinking that since logging performs I/O, there could be ways to improve its performance If don’t know if this fits your needs, but you can significantly reduce I/O overhead by writing log messages in batches with [MemoryHandler](https://docs.python.org/3/library/logging.handlers.html#logging.handlers.MemoryHandler).