Hello. I was working on a project, but encountered an issue that I’d like to share. The problem is with a data transfer operation between databases in Python. I was transferring some columns from a table in database A to a table in database B. I attempted to achieve this using multiprocessing. However, even with the use of the Pool class and starmap, processes are going into a sleeping state. I am seeking a solution for this issue. How can I address this problem?
What do mean by a “sleep state”?
How did you determine this information?
Is this a case of the process waiting for I/O to your database?
What OS are you using?
I am using windows operating system. If I express the problem I am experiencing more clearly; I have large data and I want to transfer data between databases. I use Pool.starmap for this. Coming to the problem, when I start the processes, only one of the processes that should run in parallel is running and the other processes go into sleeping state. I’m having a bit of a time crunch because this situation slows things down. Here, would it be useful to use pool.join() and pool.close() structures to prevent processes from slepping, or am I experiencing a synchronization problem in the database? I just want to give you an idea. Could there be another reason for this situation?
My first thought is that your processes are waiting on locks on the database. Is your sql locking out access so that only one writer at a time is allowed?