In our C extension I am lifting the gill anyway to do some computation, so I was wondering if I can also have the creation of the output variable (bytearray) with the GIL lifted?
Creating a varibale like bytes/bytearray involves malloc/calloc so it also takes a somewhat significant amount of time.
Since the extension is the only one reading/writing, it is safe to do so. It seems to work, but I don’t know if there’s some check that I just haven’t triggered yet in my testing?
I tried to look through the cpython code, but it’s quite a lot to check.
You need to hold the GIL anytime you call the python C API (unless stated you do not need the GIL).
Otherwise you will risk corruption of the python objects from other threads.
But I am creating the new Python object from within my C extension, no other thread can access it before my function finishes.
So how could that corrupt something?
The memory allocation itself requires the GIL to be held. It might happen to work on CPython, but that’s not guaranteed or anything. However, once you have created the bytearray, you can obtain a buffer reference, which can be freely written to without the GIL held.