Increase speed of codes

[dset (d)]

Goodmorning,

I’m writing a complex python code and I need to increse speed of calculation. I added multiprocessing tools with pools, I use comprehension way to declare big list etc.

Otherwise, I call many times external python codes from other python codes. I call them using classic way os.system(“python nameOfCode.py arg1 arg2 arg3 …”).

I don’t know if this last approach reduce speed and is better pre-compile codes or not.

I would ask you suggestions to increase velocity and usefull tutorials for that. Maybe also general approaches and then I will study them. I read online many opinions but I don’t understand the better way.

Many thanks

[jeanas (Jean Abou Samra)]

There is no such thing as the “better way”. It all depends on why your code is slow.

• It is heavy in mathematical computations: try the popular, highly efficient NumPy package for dealing with arrays and common numeric algorithms.

• It has a lot of blocking input and output. Read about “async I/O” (Async IO in Python: A Complete Walkthrough – Real Python).

• os.system is not really a nice way to run other Python code. Just think that the executable might need to be called python3 on many systems. Import modules and use them.

• Tweak your algorithms first.

• Profile your code (The Python Profilers — Python 3.9.2 documentation) and optimize the slow parts.

• Try PyPy or Numba.

[steven.daprano]

Hello D,

To make code in ANY language, not just Python, faster, start by:

• Complex code is usually slow code. Try to make it simpler.

• Check your algorithm. Is there are faster algorithm you can use?

• Profile the code to find where it is slow, and concentrate on speeding
  that part up.

• If all else fails, use a faster computer with more RAM.

There is no magic setting to make Python faster. If there was, we would
use it all the time. To really help you, we would need to see your code.
However, I can see at least one thing that will be slow:

“”"
Otherwise, I call many times external python codes from other python
codes. I call them using classic way os.system(“python nameOfCode.py
arg1 arg2 arg3 …”).
“”"

This will definitely be slow. Every time you call os.system, the
current Python interpreter has to pause, ask the OS to find the Python
interpreter on disk, the OS launches that interpreter, searches for the
file nameOfCode.py, runs it through the interpreter, passes the
arguments as strings, which means they probably have to be converted
back to ints or some other data value, wait for the calculation to be
done, then finally the current interpreter can continue.