Best practice suggestions (including bootstrap)

Hi!

pkgsrc has a lot of packages for python modules. I’ve read Why you shouldn't invoke setup.py directly and discovered that the method pkgsrc currently used is going away (setup.py build/setup.py test/setup.py install).

I’d like to know to what method we should switch.

Some more context/requirements:

  • pkgsrc installs the modules into ${PREFIX}/lib/python${PYVERSION}/site-packages so that they are available for all users
  • pkgsrc provides binary packages for all supported python versions that are also supported by the module
  • pkgsrc only packages the latest version (except for the cases where an older version is needed for python 2.x compatibility, then both this and the latest version are packaged)
  • the bootstrap path currently is:
    • install distuils (comes with python)
    • use it to install setuptools
    • use either of these to install everything else
  • .py, .pyo and .pyc files are installed

I’m interested in learning best practices about this (since a lot of time has passed since someone really invested time into this part of pkgsrc) and also what we should switch to, as noted above.

Thanks,
Thomas

1 Like

For building you should use GitHub - pypa/build: A simple, correct Python build frontend and move over to pyproject.toml. You can use https://cibuildwheel.readthedocs.io/ if you want help building wheels (which you want to always create).

For testing, there is no equivalent replacement, but that’s always been tool-specific. Most people like using Nox and Tox for this sort of thing.

For installation, it depends on why you are executing the installation manually on your project. If it’s for testing you can use pip -e .. If you’re installing a wheel or something from disk you can always pass the path to pip.

Hi!

Thanks for the reply, @brettcannon . I think it’s not quite what I was looking for. I’m not maintaining a python module myself, I’m coming from the distro side of things. What would I use a wheel for? pkgsrc has a binary package format that I want to use for python module binary packages as well.

I took a look at build today, and if I’m not mistaken, it depends on packaging, tomli, pep517, and colorama. So before I can use build, I have to install those. They have some more dependencies, e.g. packaging needs pyparsing etc. But how do I build and install those dependencies before I have a build package? For pkgsrc, I need a bootstrap path that starts from python-only.

I hope that makes it clearer.
Thomas

It’s what you install a Python package from. It contains all the files and data on what files go where.

You can use GitHub - pradyunsg/installer: A fork for making that other thing better. to help do the initial installation of whatever you need. But Python also ships with pip for most installations and so you (should) already have a way to do installs to solve your bootstrapping problem.

But the key thing is you can’t assume setuptools (i.e. setup.py) is used for all Python packages.

I have written GitHub - FFY00/python-bootstrap: Helper script to bootstrap a Python environment to help with this, it is a script that runs GitHub - pypa/build: A simple, correct PEP 517 build frontend and GitHub - pradyunsg/installer: A low-level library for installing from a Python wheel distribution. (with Add CLI by FFY00 · Pull Request #66 · pradyunsg/installer · GitHub) from source to build and install the required tooling to start building and installing Python packages.

I am planning to propose it for PyPA inclusion but wanted to write some docs about the bootstrapping process first.

distutils is deprecated and will be removed in Python 3.12, so I wanted to get the bootstrapping process in shape before that.

1 Like

I think we’re still talking past each other. I’m not using wheels. pkgsrc builds (pkgsrc) binary packages from pypi source package files and installs them using pkgsrc tools.

pip is not part of python. There is a separate pip package on pypi.

But perhaps it’s the right way for the future, I don’t know, that’s why I’m here.

pip does not seem to have any dependencies (except that I need to install pip itself from sources using only python, into my ${PREFIX}). How can I do that?

From the documentation, installer looks like the wrong tool, since it uses wheels and I want to use pkgsrc binary packages.

Hi Filipe!
Thanks for the pointer. I’m not sure how this fits into my bootstrapping story, but the example doesn’t work for me. I git cloned it and ran python -m bootstrap.build, and got an error:

Traceback (most recent call last):
  File "/usr/pkg/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/pkg/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/tmp/python-bootstrap/bootstrap/__init__.py", line 78, in <module>
    shutil.copytree(
  File "/usr/pkg/lib/python3.9/shutil.py", line 564, in copytree
    with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/python-bootstrap/external/build/src/build'

Did you also clone the submodules?

Filipe, no, I had not, but now I have and something’s still missing:

wiz@yt:/tmp/python-bootstrap> git submodule init
Submodule 'external/build' (https://github.com/pypa/build.git) registered for path 'external/build'
Submodule 'external/flit' (https://github.com/takluyver/flit.git) registered for path 'external/flit'
Submodule 'external/installer' (https://github.com/FFY00/installer.git) registered for path 'external/installer'
Submodule 'external/pep517' (https://github.com/pypa/pep517.git) registered for path 'external/pep517'
Submodule 'external/setuptools' (https://github.com/pypa/setuptools.git) registered for path 'external/setuptools'
Submodule 'external/tomli' (https://github.com/hukkin/tomli.git) registered for path 'external/tomli'
Submodule 'external/wheel' (https://github.com/pypa/wheel.git) registered for path 'external/wheel'
wiz@yt:/tmp/python-bootstrap> git submodule update
Cloning into '/tmp/python-bootstrap/external/build'...
Cloning into '/tmp/python-bootstrap/external/flit'...
Cloning into '/tmp/python-bootstrap/external/installer'...
Cloning into '/tmp/python-bootstrap/external/pep517'...
Cloning into '/tmp/python-bootstrap/external/setuptools'...
Cloning into '/tmp/python-bootstrap/external/tomli'...
Cloning into '/tmp/python-bootstrap/external/wheel'...
Submodule path 'external/build': checked out 'c940180b6ace5210e6f00149d45b1358f39fb274'
Submodule path 'external/flit': checked out '8b601d3222807c92004967ab3e2b89a627912aab'
Submodule path 'external/installer': checked out 'a8710697695b78d8a5b43ca61420b4cc31cdd199'
Submodule path 'external/pep517': checked out 'c4848b8899845e6ed746c75698634daa40dcc101'
Submodule path 'external/setuptools': checked out '4bce6d5a30accecc848d8e7f0dabd212bfea9475'
Submodule path 'external/tomli': checked out '9d416d91656fa14cd48eb97e20d3f0c72257fea6'
Submodule path 'external/wheel': checked out 'b8c4aa055cea0132776e7e53edce3538710d5b68'
wiz@yt:/tmp/python-bootstrap> python3 -m bootstrap.build
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/pkg/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/tmp/python-bootstrap/bootstrap/__init__.py", line 75, in <module>
    MODULES.mkdir(parents=True)
  File "/usr/pkg/lib/python3.9/pathlib.py", line 1323, in mkdir
    self._accessor.mkdir(self, mode)
FileExistsError: [Errno 17] File exists: '/tmp/python-bootstrap/.bootstrap/modules'

A co-developer pointed out to me that I was missing some important information:
pkgsrc (www.pkgsrc.org) packages all kinds of software in all kinds of languages. So python modules are just a subset of what we want to package.
pkgsrc bootstraps from a C compiler and also installs python itself (and ruby, rust, perl, …) and provides binary packages for users to install.
We currently have some 3000 python modules packaged this way, mostly using setup.py.

Your “pkgsrc binary packages” are presumably some sort of archive of files that the pkgsrc installer will copy into the ultimate desired location on the user’s system?

Wheels are similar, in the sense that they are an archive of the files that need to be on the user’s system in order for the user to make use of the package. So I’d argue that you should build wheels (using a modern Python tool like build) and then copy the content of those wheels into your pkgsrc binary packages. The wheel spec explains how to extract the files onto the target system - you can adapt that to decide how to extract into your archive format and subsequently how to extract the archive onto the user’s system.

That’s basically the “modern python best practice” for this type of workflow. If you want help with the practicalities of implementing it, there are people here who work on Linux distros, who would have more useful insights than I can offer (I’m a Windows user) - but the above is the basic idea.

Thanks for the reply, @pf_moore .
I took a look at wheels, and I think they are not the right solution for what I want:

  • Only Linux/macOS/Windows wheels can be uploaded to pypi. This means that even if I use the pure python wheels from pypi, I still need to find a way to build the ones that are operating system specific ones (since this leaves out NetBSD, FreeBSD, OpenBSD, Solaris, …).
  • Using wheels means using packages someone else created. I’d prefer to use the sources.
  • Using wheels means I can’t run the tests, which I like to do before updating a module in pkgsrc to verify it still works.
    In summary, wheels look like a binary package format - but pkgsrc already has one. I prefer (and for some packages, have to due to not using Linux/macOS/Windows) to build from sources.

[…]

In summary, wheels look like a binary package format - but pkgsrc
already has one. I prefer (and for some packages, have to due to
not using Linux/macOS/Windows) to build from sources.

I think there’s some confusion still. You don’t need to use existing
wheels. The idea is that instead of calling setup.py install to
compile and place files into the tree for your pkgsrc package, you
use the build utility to build a wheel from the sdist tarball and
then copy the files from the wheel into the equivalent path in your
package. The only functional difference between setup.py install
and build/wheel is that the former places the resulting files into a
filesystem tree while the latter places them into a zipball (which
can then be extracted to achieve the same results).

1 Like

Thanks, @fungi . Yes, that would be possible.
Which still gets me back to my original question, what the intended workflow is, starting from a basic python installation. I mentioned the one that pkgsrc is currently using in my first post.

Yep, I understand. You’re trying to repackage Python projects for pkgsrc.

True, but CPython ships with ensurepip which installs pip as part of installation. Basically it’s assumed (if you’re not on Debian :wink:) that if you have CPython installed that you will also have pip installed.

I think I understand where the confusion is. The reason we all keep saying “use wheels” isn’t for you to directly install wheels, but to build your own wheels and tear them apart to repackage them for pkgsrc. The thing is you can’t get what you’re after from straight sources (what we call a “source distribution” or “sdist”). An sdist only contains a configuration file (pyproject.toml for up-to-date projects, setup.py for older ones), and that’s it. You actually can’t deduce what files should go where on the file system by introspecting an sdist. You must build a wheel to get that information.

With my understanding that pkgsrc packages are binary artifacts and not a collection of source that you build upon install, what people are suggesting you do is:

  1. Download the sdist/source.
  2. Build a wheel.
  3. Tear the wheel apart based on its metadata and repackage its files using the pkgsrc format.

The bootstrap project that @FFY00 pointed you at seems to do the bootstrapping you’re after to get you the parts you need to build the wheel for you to then disect and reassemble into a pkgsrc package.

BTW, I am not sure if anyone has proposed to allow BSD-based wheels to be uploaded to PyPI. It would require specifying how to version them appropriately (e.g. the manylinux spec), teach Python how to detect the OS and version appropriately, etc., but I don’t think there’s a fundamental reason beyond lack of time and motivation why it couldn’t happen.

2 Likes

Using a wheel as an intermediate step may be fine, but what is the actual procedure for building a wheel and installing it into a destdir?

Before, starting with an sdist for foo-1.23, the installation procedure that worked reliably for almost all Python packages was more or less:

cd foo-1.23
python setup.py build
python setup.py install --root=${DESTDIR}

What’s the sequence of commands that one is supposed to execute in the new world order where python setup.py install doesn’t work, starting from an sdist, to get the same effect?

And, if the new procedure depends on having foobuilder or bartools or bazutils outside the cpython-3.x distribution, how is one supposed get those built from sdists first?

1 Like
python -m build --outdir ${DESTDIR} .

https://pypa-build.readthedocs.io/

@FFY00 pointed to their python-bootstrap project in Best practice suggestions (including bootstrap) - #5 by FFY00 . Otherwise wheels are zip files, so you can unpack them and copy files to the appropriate places.

build is not part of python, so the first command will not work.

I tried using python-bootstrap again with python 3.9.9 and nothing else installed and get:

# python3.9 -m bootstrap.build                                                                                                                                                     
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/pkg/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/tmp/python-bootstrap/bootstrap/__init__.py", line 88, in <module>
    import build.env  # noqa: E402
  File "/tmp/python-bootstrap/.bootstrap/modules/build/env.py", line 18, in <module>
    import packaging.requirements
ModuleNotFoundError: No module named 'packaging'

So we’re still missing step 0, get a build tool that builds the first wheels for the tools I need to build everything else.

Next time, please open an issue on the repo if you can. There aren’t any tests yet, so I missed that, it should be fixed now.

Thanks! Just to be clear, does this have the same effect as building a wheel, and then installing that wheel into ${DESTDIR}?

Would it also have the same effect to build into dist/ (as python -m build . with no --outdir ${DESTDIR} does by default), and then copy that into ${DESTDIR}?

# build stage
cd foo-1.32
python -m build .

# install stage
cd foo-1.23/dist
pax -rw -pe . ${DESTDIR}

(adjusted, perhaps, for appropriate subdirectories under foo-1.23/dist and/or ${DESTDIR}, like maybe pax -rw -pe . ${DESTDIR}${PREFIX}/lib/python3.X/site-packages)

Or does running python -m build --outdir ${DESTDIR} . or installing a wheel also do other things for installation?

Does it make a difference whether foo-1.23 is, e.g., the content of a git tag from a typical project, versus an sdist created with python -m build sdist or an sdist downloaded from pypi?

(I read the documentation at https://pypa-build.readthedocs.io/ but it didn’t clearly answer these questions. Apologies if I haven’t found documentation that should be obvious; I don’t do much Python packaging myself, and it has been several years since I worked with this stuff more directly.)