I have always wondered a bit why I’m not getting a bit more feedback on my (by now ancient) project called Shed Skin.
In a nutshell, it allows one to translate small programs (up to a few thousand lines perhaps), to C++ with often a very large speedup as a result. It can also generate extension modules that can be used in larger (non-compiled) programs.
The main problem and perhaps reason it is not used much, is that it restricts the Python language, or this would not be possible. Most importantly, variable must be (implicitly) statically typed, so no dynamic typing. Also only a part of the standard library is supported, besides other restrictions.
A nice example of its usage I like to give is a DOOM WAD renderer in 999 lines, that becomes many times faster after compilation:
In the shedskin/examples section, there are actually 84 example programs at this point, though not all as spectacular as this one..
I guess I would like to ask for some community feedback at this point. Why do you guys/gals think I’m not seeing much usage/feedback? Is it too restrictive..? Are people expecting it to work for random programs, and giving up after the first warning/error?
I will probably keep following CPython releases, and do a yearly update following these, but I could improve things much faster if I were to focus my full attention on it.. It actually is a lot of fun to hack on (both from a research and coding perspective), but I also have other things going on..
Thanks for any feedback, suggestions, and considering helping out!
Hi Mark, thanks for all your work. I am a fan of Shedskin - I tried it three or four years back, before you recently revamped it. I remember finding it tricky to install back then - I needed to run it using Python 2 in a Docker container. Is that side of it much more frictionless now?
I could happily dispense with “anything really dynamic”, in my Python even if not using type hints. But the biggest pain points regarding the language restrictions for me were “nested functions and classes”, “closures”, and “full unicode support” (is there really nothing in in the C++ stdlib that implements unicode-supporting strings).
I just haven’t encountered many situations where I needed the speed up from compiling Python to C++. And when I start having to write any specific flavour of Python, my mind inevitably wanders and thinks about alternatives. So unfortunately I’ve just never had a pressing need to return to Shedskin since I had a look out of curiosity.
Yes, Shedskin should be much easier to use these days, as it is no longer stuck on Python 2. In fact, the latest release brought compatibility up to 3.14. On the Windows side, we use CMake and Conan these days, which is also much easier than before.
Support for nested functions, closures and ‘full unicode’ could all be added, but typically these aren’t crucial for the algorithmic-type code that Shedskin really targets I guess, so they haven’t had priority so far.
Thanks for your suggestion! There is a large set of working examples here:
But as can be seen from looking through them, these are mostly algorithmic-type programs. Shed Skin is not really meant to work with any kind of Python program (such as FastAPI), but just to allow one to run very performance critical parts/algorithms at speeds close to manual C/C++. That in itself may for sure limit the target audience by a lot (fewer and fewer people will be writing low-level algorithms unfortunately.. :P)
As compared to Numpy, think of a chess-program, a project Euler program, an emulator, or any kind of low-level algorithm that isn’t really suited to be expressed with matrices. As for Pandas, that seems something else entirely.
Ah okay. What about by comparison with Cython then? That seems more similar. Notably, with Cython, you could cythonize one module and then have the rest of the program remain in Python with full flexibility.
Yeah, that is actually a good comparison. Except there’s no need to add any type declarations/hints, as Shed Skin figures them out by itself. On the flip side, the module is more restricted as it can only use a small subset of all Python standard library modules, for example.
Just like Cython, Shed Skin can generate an extension module for you that can be imported in a larger program. Many of the Shed Skin examples, such as the doom renderer, rely on this functionality for integration with pygame.
I haven’t used Shed Skin but I think the main thing I’d be looking for if I were thinking about using it is some kind of comparison with other tools in a similar space, like numba or Cython.
Hi Mark. I think the short answer is because people don’t understand how it would benefit them. In the marketing world, where the purpose is to create desire for products and services, they say “Sell the sizzle, not the sausage” (from an old TV ad that sold sausages by showing them sizzling gently in a pan). What makes your sausage preferable to any similar solution the reader might know of?
Perhaps try changing your repo “About” text from “Shed Skin is a restricted-Python-to-C++ compiler …” to “Shed Skin lets you run many Python programs up to X times faster. Read the introduction to find out how, and which ones.”
Thank you Brendan, Steve, for taking the time to respond. I will update the github front page with a less technical and more to-the-point introduction, and am working on a comprehensive comparison plot (not just comparing examples/sieve but ~69 of the shedskin examples/benchmarks). Hopefully that will make things a bit more attractive.. :’-) Thanks again.
I’ve admired shedskin for years. Unfortunately I mostly code to make a living and my employers usually add more pain to my life if I choose software that is GPL. While there are many reasons one picks a license and it’s not an easy choice, if you wanted broader adoption, you may consider a license that has fewer restrictions.
It looks like the runtime bits of shedskin are BSD licensed so it doesn’t make your compiled programs GPL (in much the same way as GCC is GPL but that shouldn’t really be a concern for people who are just using it to compile stuff).
So I don’t think the license for shedskin should be a huge concern (although convincing a legal department of that my still be an ordeal)
I’ve known about Shedskin and Nuitka for a long long time. But I never had the opportunity to use them at work because I already have Cython and Numba to cover most of my performance-critical “loopy” code. There are a lot of tools to choose from in this space, and it’s often hard to justify using a “weird” tool when time and energy are limited and other tools that do similar jobs are more popular. And there are other “weird” options available too such as Mypyc, PyPy, and even GraalPython, all of which involve different tradeoffs and compromises.
Yeah, at least this was the intention. I’m happy to provide a statement as such. In fact I should probably mention it somewhere, on the front page or at least somewhere in the documentation..
Hi all, thanks again for your constructive comments.
I took some time to create a performance comparison with PyPy, Nuitka, CPython 3.10 and CPython 3.14 and added it to the front page. I also added a ‘when to use Shed Skin’ section. I hope this all makes it a bit clearer when to use (or not to use) Shed Skin, or the other implementations. Let’s see if I can add the performance comparison plot here, too:
I will analyze the results in more detail in a blog post, but for the Shed Skin examples at least (bias alert!!), Shed Skin is able to provide a speedup of about 29 times on average (not percent, times! :)), whereas PyPy is able to speed up the same code by about 16 times on average. This is very much apples-to-oranges of course, and PyPy does an impressive job given that it has its hands tied behind its back.
Also, it’s a bit hard to see here, but CPython 3.14 is about 60% faster than CPython 3.10 for these examples. To me, that is very promising for the future of ‘pure python performance’. Finally, Nuitka unfortunately cannot match the improvements in CPython since 3.10 and comes in slower than 3.14..
Yeah, massive bias alert there But that’s to be expected. I’m trying to get a rough idea of what would make a program a good candidate for shedskin; the README has at least some information there, but I’m trying to get my head around what the restrictions are.
The chess example feels like C code rewritten into Python, the sort of thing that I’d do when building something for a Raspberry Pi when the example code was for an Arduino. (That sounds weirdly specific; it is, and it’s exactly what a brother of mine is in the middle of doing.) Lines like if (mv & 0x01000000): # double step in particular don’t really feel like Python code, they’re definitely low-level thinking encoded in Python syntax. So this feels like it’s in the same space as Cython, and not really a general-purpose “make Python faster” tool.
Mandelbrot looks a lot more Pythonic, though. Is there a reason for not using the complex type? (Side question: The output looks rotated; swapping the cr and ci so the real part comes from x and the imaginary from y gives the traditional orientation. Was there a reason for doing it that way around?) I switched to the more natural way of writing it and the time in CPython improved slightly:
c = complex(x/40 - 0.5, y/40)
z = 0
for i in range(max_iterations):
z = z * z + c
if abs(z) > bailout:
line.append(" ")
break
else:
line.append("#")
So, from the examples given, there’s still a lot of difference between normal Python code and skin-sheddable Python. And you say that it might be necessary to rewrite from scratch.
I’m not sure how fair it is to compare to Python performance, but I would definitely be interested in a broad comparison of “time/effort required to speed up this module by…” with ShedSkin, Cython, and some other options. But the relevant time here will be programmer time, not just execution time.
It would be better if the vertical scale on the plot was logarithmic (and had an axis label). Always with performance plots you should make very clear whether bigger or smaller means better or worse.
I need to test it!!!