I’ve recently been working on a challenging project that involves solving complex problems in Python. While I’ve made some progress, I’m curious to know if there are any optimal approaches or best practices for tackling such problems efficiently.
Specifically, I’m interested in understanding how experienced Python developers approach problems that require advanced algorithms, data structures, or computational optimizations. Are there any specific libraries or techniques that you recommend for maximizing performance and achieving elegant solutions?
I would greatly appreciate any insights, tips, or resources you can share. Your expertise would greatly benefit my ongoing project and help me further develop my Python skills.
Some general thoughts; tbh these aren’t Python specific, but more general software engineering. They’re also strictly my own opinions, not those of my employer, the Python community, etc.
Always start with the simplest solution that could possibly work,
When possible, write automated tests that describe what a correct solution is. These become regression tests for when you rewrite your solution from #1. If not, at least keep a copy of solution #1 even if it is slow, because you can compare the output of future versions against it (a “test oracle”).
If the simplest solution isn’t fast enough, do research. Often there are existing algorithms that solve your problem (or part of your problem). CS is a big field and people have already solved a lot of problems, so use it
If that isn’t fast enough, measure your code with a profiler. This gives objective goals and also informs you where the biggest slowdowns are. Don’t guess! Often profiling will find the “low hanging fruit” that give large speed ups. (I’ve had good luck with py-spy, but there are other tools.)
If you change something in the code to make it faster and it makes the code less obvious to read, comment the heck out of it explaining why the code does the non-obvious thing.
You’ve done all that and it still isn’t fast enough. Now and only now rewrite your algorithm in C or C++ (edit: or Rust!) as a native extension module that you can call from Python code. (Unless you are very practiced at writing native extension modules and could just write one up in your sleep, I guess.) If you are into C++, pybind11 is a nice way to make these.
Oh, and never write your own cryptographic primitives.
But THIS opinion is shared by nearly everyone. Seriously, the number of ways you can do crypto wrong without realising it… I mean, when there are attacks like “point a mobile phone camera at the power LED and get video footage of the private key” (due to differences in power consumption in the shift/multiply steps), you really REALLY want to use what someone else has figured out.
Actually, I tend to like Cython for such things. It’s not very far from CPython syntactically and semantically, and if you watch your implicit type conversions can rival C++ in performance. I’ve even had projects where I kept the two versions (CPython and Cython) in the same .m4 file, automatically creating each with m4 -Dwhatever. That way, I have something slow-but-clear that’ll also run on Pypy3, while having a fast C extension module for the CPythons.
I may try doing that with Shedskin as well sometime, now that Shedskin is mostly Python 3-compatible. It transpiles from an implicitly-static dialect of Python to C++.
