Forget that setuptools and all other packaging tools exist. Let’s assume for a second that all we had was the PEPs we have written and even those could be changed if necessary (although it would be discouraged unless found to be really necessary). What would want from packaging to be useful to us today and what are we missing to make that dream come true?
Here’s a rough outline of what would be necessary from my understanding to both create a package for distribution as well as to work with it locally while developing:
pyproject.tomlto find out how to run the build tool
- Build tool builds extension modules and other non-Python code
- Appropriate metadata for the wheel gets generated
- Built extension modules, Python source, and metadata files get put into a wheel
- Read wheel file for required dependencies
- Repeat the above steps until down to the leaf nodes of the dependency tree
- Install all the required wheels
OK, so we have PEP 517 on how to execute the build tools. What is required of those build tools?
- Know how to build extension modules if there’s C code
- Know how to build anything else that’s unique (e.g. if you want Cython to be run as part of this instead of as a separate step outside the build tool)
- Know where/what those built artifacts are
- Know where the Python source code is
- Know all the required metadata for a wheel
- Be able to make the metadata files for a wheel
- Where to place files within the wheel
- Package it all up in a zip file with a
.whlextension name and appropriate tags
So how do we do those steps? I would argue building C extensions should be the job of a C build tool. The rest of steps is just metadata you should put in
pyproject.toml somewhere and how to get the appropriate files from that C build tool so they can be placed as appropriate in the wheel after knowing how to execute the C build tool. Could we go so far as to take meson’s approach of generating build files once that you directly execute over and over again and orient our build tools so they generate the C build tool configs (e.g. a build tool generated meson config files that people can execute directly post-generation instead of having to use the Python build tool to drive that every time)?
After that what does the above lack that setuptools has which people legitimately want? I can think of entry points and editable installs. For entry points I would personally redefine them as an option where you define the executable name and all it does is rename your
__main__.py in your repository to that executable in the wheel so that there’s no need to generate code and it’s easily supported by all install tools (it basically becomes a file rename in the wheel and the install tool just slaps on the
#! when it ends up being written to disk).
As for editable installs, that seems to require:
- Running the C build tool
- Copying the extension modules to the right place among your Python source code
- Creating a
.pthfile that updates
sys.pathappropriately (although I personally hate
.pthfiles and want to come up with a better supported solution, but one thing at a time)
Maybe I’m missing something but that all seems tractable. If I’m not off then really the key part that’s missing is PEP 517 somehow separating C compilation and the identification of where the extension modules are and where they should end up as their own step so the overall build tool can copy those files to the appropriate place and let the
.pth do its thing. (The other bit is running the installation tool for your dependencies but that can still be done by the build tool or even not by the build tool but by a normal installation tool that knows how to read
pyproject.toml if we standardize on the metadata specification more so dependencies are universally listed there).
Assuming the very rough outline above is not totally bonkers, some questions from me:
- Am I nuts in thinking that we can simplify things by punting more to C build tools so for Python-specific tools its more about identifying where the C build tool put the
- Would we standardize the universal metadata of projects in
pyproject.tomlif we didn’t have the history of
- If we still wanted entry points could they be as I described above?
- Could editable installs be more about breaking the build steps into a more fine-grained fashion so that it’s more “evoke C build, copy
.sofiles to the right place, spit out a
.pthfile” (and if we did universal metadata just use your installation tool to install your dependencies from
pyproject.tomlon your own)?
What am I missing? I feel like this is all tractable which honestly scares me , so I feel like I’m missing some really key thing here in the fundamentals of packaging which will make this way more complicated and harder to potentially accomplish (remember backwards-compatibility is not a concern in this exercise; this exercise is about what we want packaging to be).