Our C API for capsule objects supporting changing the name and the destructor of a capsule after it was constructed. This seems like a strange feature, because a capsule’s name and destructor are part of the capsule’s logical type (usually wrapping a C or C++-level physical type) and being able to change them willy-nilly breaks the contract around a given capsule.
Theoretically, this implies that a capsule destructor should be careful to handle the case where the capsule’s name doesn’t match the logical type that the destructor was implemented for (either because the name or destructor was externally changed).
Which further begs the question: what should the destructor do in that case? It can’t destroy what it was meant to destroy (the logical type doesn’t match), but it also cannot do the expected destruction for the capsule’s current logical type (it has no knowledge of how to do that). It can only print a warning and let the encapsulated resource leak.
I tried to read through the design discussion that led to PyCapsule, but it’s surprisingly terse. There is a 2009 ticket where @larry mentions that the design was informed by private discussions with @guido . @larry also posted a mailing-list topic which didn’t entice any feedback from other core developers.
I’m therefore curious:
was there originally any specific rationale for providing this functionality of changing a capsule’s name or destructor?
is anyone currently relying on this particular feature?
The design discussion was held offline because it was filling in a massive security hole. CObjects were primarily used to hold a sort of vtable for an extension module, and someone pointed out a user could easily crash a Python interpreter by swapping CObjects between modules. If you overwrote the datetime CObject (think “vtable”) with the CObject (vtable) from socket, then created a datetime object, whammo! And since Google App Engine was a brand new thing at the time, designed to run multiple users’ code from the same Python process, this was cause for alarm.
For my part I was mostly a go-between. Neil Norwitz handed me the issue and said “go for it”, Guido basically dictated the API to me, and Jim Fulton strangely claimed this was all unnecessary and the existing object (his creation) was fine and needed no change. I was just the monkey behind the keyboard actually implementing the new object.
I couldn’t tell you why the capsule API permits renaming a module or changing its destructor, apart from “it seemed like it could be useful”. Let a thousand flowers bloom–let the users experiment with the API and find their own uses. Like, maybe a capsule can’t be initialized all in one go, so maybe a module would have reason to create the capsule early then fill it in later. Personally I’ve never needed that functionality, and I have no strong rationale for keeping that corner of the API. I bet nobody is using it, and therefore it’s fine by me if we remove it.
Changing the name (but not the destructor) of a PyCapsule is relied on by DLPack - how it does that is documented in Python Specification for DLPack — DLPack 0.6.0 documentation. It is done as a method of exchanging information about data having been “consumed” (i.e., being in use). With the current Python API (post __dlpack__ introduction) I don’t think the rename requirements were essential, but I suspect that in older DLPack versions they were. The old API, still in wide use, was to have separate functions to_dlpack/from_dlpack, where the to_ flavor would return a raw PyCapsule that the user was then responsible for passing to the from_ function. That had to be done exactly once - if not, the data would be either leaked or memory could be used after being freed. Changing the name was signalling that the data was used once, and hence it was guarding against such memory leaks.
Here is what user code tends to look like:
# Using PyTorch and CuPy as an example here, could also be NumPy/JAX/TensorFlow/etc.
import torch
import cupy as cp
x = cp.arange(10_000) # creates a large array on GPU
# New DLPack API:
y = torch.from_dlpack(x) # y is a torch.Tensor; `x` and `y` share memory of data
# Old DLPack API:
capsule = x.toDlpack() # raw PyCapsule in hands of user
y = torch.utils.dlpack.from_dlpack(capsule)
DLPack is essentially a device-independent generalization of the buffer protocol, and is heavily used in deep learning in particular, when libraries or end users want to mix two frameworks. E.g., spaCy uses it for data sharing between PyTorch and CuPy. So please don’t remove the ability to change capsule name without a pressing reason - that will cause a lot of work to update the DLPack spec and then all libraries that implement it.