Python Website Backend Gradually Stops Responding to API Requests After Running Continuously

[benstokes]

Hello Python Community,

I am currently dealing with one persistent problem in the Python backend of my website, and I am hoping someone can help me understand what might be causing it. The core issue is that the website initially handles requests normally, but after the application has been running continuously for an extended period, API requests gradually become slower and eventually some requests stop receiving a response within the expected time. The website itself may still be reachable, but specific API endpoints become extremely slow or appear to hang until the application process is restarted. Restarting the Python application immediately restores normal response times, but the same behaviour eventually returns after the application has been running again for several hours. I am trying to determine what could cause a Python web application to gradually lose responsiveness without producing an obvious fatal exception or immediately terminating the process.

The application handles normal website requests as well as several API endpoints that retrieve and update information from a database. Under normal conditions, these endpoints respond quickly and consistently, and I do not see any unusual resource consumption when the application starts. However, as uptime increases, I can see the response time of certain requests gradually increasing even though the overall traffic to the website has not changed significantly. Eventually, requests to the affected endpoints can remain pending for a long time before the client receives a response. The rest of the server remains accessible during this period, so it does not appear to be a complete server outage. Restarting only the Python application is enough to return the API to normal operation, which makes me suspect that something inside the application process or one of the resources it manages is accumulating over time.

I have already started monitoring the Python process and have been checking memory usage, CPU usage, and the number of active requests while the problem develops. Memory consumption does increase gradually, although I have not yet established whether the increase is directly responsible for the slowdown. CPU usage is not consistently high when the API becomes unresponsive, which makes this different from a straightforward CPU-intensive operation. I have also checked the application logs and do not see a clear traceback immediately before the affected requests begin hanging. Some requests complete successfully while others remain pending, so the application is not completely dead. This partial degradation makes me wonder whether the process could be waiting on a resource such as a database connection, network operation, file descriptor, thread, or another object that is not being released correctly.

I have reviewed the application code for obvious resource-management issues and have tried to make sure that external resources are closed appropriately after they are used. Database operations are performed through a connection layer, and I am investigating whether connections are being returned correctly after every request. The application also makes requests to external services as part of certain API operations, so I am checking whether any of those calls could remain open longer than expected. However, I have not found one specific function that consistently causes the problem. The fact that the website works normally after a fresh application restart but becomes less responsive only after extended uptime makes me suspect there may be a resource leak or lifecycle issue that is difficult to see during short development tests.

I have also attempted to reproduce the problem in a development environment, but it is much harder to trigger there because the application does not run continuously under the same conditions as production. In production, the Python process remains active for long periods and receives requests throughout the day, which seems to make the problem more noticeable. I have considered whether the issue could be related to the web server or application server configuration, but I want to understand the Python-side behaviour first rather than randomly changing production settings. I am particularly interested in learning which Python profiling or diagnostic tools would be appropriate for capturing information while the application is still responsive and then comparing that information with the state of the process when requests begin hanging.

I would appreciate guidance from the Python community on how best to diagnose this type of gradual loss of responsiveness in a long-running Python web application. I would especially like advice on identifying leaked database connections, unclosed network resources, blocked threads, growing queues, excessive objects, or other resources that can accumulate without immediately causing an exception. If there are recommended Python profiling techniques, stack inspection methods, memory diagnostics, or production-safe approaches for identifying which requests or operations are waiting indefinitely, I would be grateful for suggestions. My goal is to find the underlying cause of the degradation and fix it properly so that the website’s API remains responsive during long periods of continuous operation without requiring me to restart the Python application manually. Sorry for long post!

[Rosuav (Chris Angelico)]

@benstokes

The website itself may still be reachable, but specific API endpoints become extremely slow or appear to hang until the application process is restarted.
…
Some requests complete successfully while others remain pending, so the application is not completely dead.

Specific endpoints. Some are simply not affected. Given that symptom, I would look at exactly what endpoints are and aren’t affected, and pick two that are as similar as possible but get different results. And then I would try to transform one into another.

The most likely case is that there’s something that all the hanging calls are doing, but the non-hanging ones aren’t. There’s some feature - say, session management - that is running into a problem, and then every request that works with the user session will hang, but those that don’t are still fine. To figure that out, you could create a secret debug endpoint that’s just like one of the non-hanging calls but adds a little bit of (unnecessary) code from one of the failing ones, and/or an endpoint that’s just like one of the hanging calls but stubs out one piece of code. If that moves it from “hanging” to “non-hanging”, that’s a strong indication that that’s your problem.

You’ve checked database, you’ve checked memory. Sessions are a definite possibility, if you use them. It’s also possible that something to do with cryptography could be getting into issues, but unlikely. Since this is a progressive issue, I’d be looking for something that is inefficiently accumulating something that it has to scan through; for example, if there’s an ID generator that has to test for collisions, and it stores them in a list instead of a set/dict, that could slow down over time.

@benstokes

would especially like advice on identifying leaked database connections, unclosed network resources, blocked threads, growing queues, excessive objects, or other resources that can accumulate without immediately causing an exception.

Let’s see. Leaked connections to other applications or services might be detectable via OS level tools. I don’t know what OS you’re running on, but on Linux, you can get a ton of helpful information from the /proc filesystem. Right now, looking at one of my servers (not written in Python but it does more work than any of my Python servers do), I can see the following:

• /proc/[PID]/fd - there are nearly 200 open file descriptors, most of them sockets. That suggests that there are a lot of open network connections. Probably some of them are dead, but 200ish isn’t a big deal; however, if there were thousands, I would suspect a problem. (Also, this is a server that actually does do long-term connections; for something that exclusively does non-websocket HTTP requests, you would probably expect that to be lower.)
• /proc/[PID]/status - a lot of human-readable information about what it’s doing. “State: S (sleeping)” means it’s not busy. (Check this one multiple times to see how its status is, how often it’s running.) Memory usage is here too.
• /proc/[PID]/net/sockstat and sockstat6 - I have apparently used 3645 IPv4 and 979 IPv6 sockets. That’s a good few. Again, you can check it multiple times to see how much it’s changing.
• /proc/[PID]/syscall - you probably won’t have to worry about this one. I spent far more time than I’d like staring at this file when trying to figure out problems that turned out to be Intel’s fault. No I’m not bitter or anything, why do you ask?

Poke around in there, see what else you can find. Might be something useful.

[benstokes]

@benstokes

I am currently dealing with one persistent problem in the Python backend of my website, and I am hoping someone can help me understand what might be causing it. The core issue is that the website initially handles requests normally, but after the application has been running continuously for an extended period, API requests gradually become slower and eventually some requests stop receiving a response within the expected time. The website itself may still be reachable, but specific API endpoints become extremely slow or appear to hang until the application process is restarted. Restarting the Python application immediately restores normal response times, but the same behaviour eventually returns after the application has been running again for several hours. I am trying to determine what could cause a Python web application to gradually lose responsiveness without producing an obvious fatal exception or immediately terminating the process.

The application handles normal website requests as well as several API endpoints that retrieve and update information from a database. Under normal conditions, these endpoints respond quickly and consistently, and I do not see any unusual resource consumption when the application starts. However, as uptime increases, I can see the response time of certain requests gradually increasing even though the overall traffic to the website has not changed significantly. Eventually, requests to the affected endpoints can remain pending for a long time before the client receives a response. The rest of the server remains accessible during this period, so it does not appear to be a complete server outage. Restarting only the Python application is enough to return the API to normal operation, which makes me suspect that something inside the application process or one of the resources it manages is accumulating over time.

I have already started monitoring the Python process and have been checking memory usage, CPU usage, and the number of active requests while the problem develops. Memory consumption does increase gradually, although I have not yet established whether the increase is directly responsible for the slowdown. CPU usage is not consistently high when the API becomes unresponsive, which makes this different from a straightforward CPU-intensive operation. I have also checked the application logs and do not see a clear traceback immediately before the affected requests begin hanging. Some requests complete successfully while others remain pending, so the application is not completely dead. This partial degradation makes me wonder whether the process could be waiting on a resource such as a database connection, network operation, file descriptor, thread, or another object that is not being released correctly.

I have reviewed the application code for obvious resource-management issues and have tried to make sure that external resources are closed appropriately after they are used. Database operations are performed through a connection layer, you can check the url, and I am investigating whether connections are being returned correctly after every request. The application also makes requests to external services as part of certain API operations, so I am checking whether any of those calls could remain open longer than expected. However, I have not found one specific function that consistently causes the problem. The fact that the website works normally after a fresh application restart but becomes less responsive only after extended uptime makes me suspect there may be a resource leak or lifecycle issue that is difficult to see during short development tests.

I have also attempted to reproduce the problem in a development environment, but it is much harder to trigger there because the application does not run continuously under the same conditions as production. In production, the Python process remains active for long periods and receives requests throughout the day, which seems to make the problem more noticeable. I have considered whether the issue could be related to the web server or application server configuration, but I want to understand the Python-side behaviour first rather than randomly changing production settings. I am particularly interested in learning which Python profiling or diagnostic tools would be appropriate for capturing information while the application is still responsive and then comparing that information with the state of the process when requests begin hanging.

I would appreciate guidance from the Python community on how best to diagnose this type of gradual loss of responsiveness in a long-running Python web application. I would especially like advice on identifying leaked database connections, unclosed network resources, blocked threads, growing queues, excessive objects, or other resources that can accumulate without immediately causing an exception. If there are recommended Python profiling techniques, stack inspection methods, memory diagnostics, or production-safe approaches for identifying which requests or operations are waiting indefinitely, I would be grateful for suggestions. My goal is to find the underlying cause of the degradation and fix it properly so that the website’s API remains responsive during long periods of continuous operation without requiring me to restart the Python application manually.

That’s a really useful way to approach it. I was looking at the problem more broadly, but comparing one endpoint that keeps working with a very similar endpoint that eventually hangs should make it much easier to isolate the common piece of code involved. I especially like the idea of temporarily stubbing out individual parts of the failing endpoint and seeing whether its behaviour changes. I’ll also investigate session handling and anything that could be accumulating over time rather than just focusing on memory usage. The /proc/[PID]/fd and /proc/[PID]/status checks are helpful too, since they should give me a better picture of whether open sockets, file descriptors, or other resources are steadily increasing while the application runs. Thanks for the detailed suggestions.