If adding something to the packaging ecosystem is too far-fetched, then how about adding a specification to how capsules should expose an API? Part of the issue right now is that all capsules do is store a void*, and it’s up to the developer(s) of the library to decide how to ship the API. For example, NumPy does it with an array, but that’s specific to NumPy, other libraries might do it with a struct, or maybe have an entire capsule for every single function. The point is, that it’s unspecified.
I’m thinking that a capsule exposing an extension module API could be specified to be sort of like a schema, which can then be used to auto-generate header files, bindings, etc:
How exactly would direct linkage make it easier/possible to avoid defining all the signatures, typedefs etc when calling from rust into a C shared library?
Is it because you can generate some of them automatically from the symbol table?
You would still have to define the signatures, but that’s not what the problem is. In Rust, to call a C function, you would use extern "C", which would find the symbol name from the library, and then handle calling it via the C calling convention. But since you can’t link against the library, you can’t do that.
Instead, you have to reach out to the Python interpreter, and import the extension, find wherever the capsule is exposed, then get a void* from it, and then figure out how to get all the needed function pointers out of the void*.
Well, Numpy should be specifying how they do it, so it’s not unspecified at that point. It just isn’t standardised.
Maybe this is something that’s worth standardising? To me it feels like something that someone could write tooling for first, and then if that gains enough popularity then everyone will use it and you get the benefits without having to try and enforce a standard. If there’s no benefit, then a standard would’ve been a waste of time.
I mean, does NumPy count as an implementation? I think capsules have proven to be capable of making public extension API’s, it’s just a matter of standardizing it at this point. Personally, I would prefer linking against shared object files as opposed to making a standard capsule format, but if the latter is much easier to implement, it’s definitely something that should be looked in to.
Rust macros could be helpful, sure, but that’s besides the point. My point is that to write any sort of binding in the first place, you have to sift through the source code of whatever module you’re working with, which might be especially difficult if you’re a Rust developer that doesn’t know C, or vice versa.
This is also a potential reason not to standardise capsules. You yourself do not consider them to be ideal. The other part of standardisation besides having an implementation is reaching the point where people want to endorse a particular approach and then make a statement of intent to support that approach in perpetuity.
I’m not nearly qualified enough to have my personal opinion be taken into account here, it would be better if a core dev gave their opinion on the capsule approach. If linking against a shared object was possible, I think most would agree that would be a better solution than capsules, but if not, standardization of exporting public API’s via capsules is about as good as we can get.
It’s impossible to properly do direct system-level linking between extension modules that are distributed inside separate packages on macOS. And macOS is popular enough that imo there’s not much point in trying to build package standards that don’t work there.
The capsule thing should be fine though? Rust can call C function pointers.
That looks like a schema for describing ctypes objects. Perhaps exposing ctypes bindings, and perhaps generating headers or Rust bindings from them, is the way to go?
If we decide to go the schema route, instead of using strings like I proposed before, it might be easier to just use integers that map to a type. For example, 0 could refer to void and 1 could refer to int:
Realistically, some easily backportable macros could be added to the C API. Maybe something like TYPE_VOID being defined as 0? I’m not sure that packaging is the right category for this now, though. Would it be better to move this to ideas?
If the schema was described in a Python module using ctypes then it would be possible to test the schema directly from Python. Then the boilerplate for e.g. Rust bindings could be generated by introspecting the ctypes objects in that module.
I think importing and using ctypes attributes is a bit far fetched for a specification. ctypes is supposed to be a compatibility layer for working with native types from Python, not vice versa. I think there could certainly be additions to ctypes that can parse an exported capsule, and then those additions could be used to develop a tool made for generating Rust bindings.
I’m betting it’s an OS restriction similar to what Windows applies to app packages. Specific executables are defined as the entry point, and everything else in the bundle is restricted - you can’t load or execute it unless you’re already “inside” the package via one of the entry points.