Best practice suggestions (including bootstrap)

[CAM-Gerlach]

@riastradh

Thanks! Just to be clear, does this have the same effect as building a wheel, and then installing that wheel into ${DESTDIR} ?

Would it also have the same effect to build into dist/ (as python -m build . with no --outdir ${DESTDIR} does by default), and then copy that into ${DESTDIR} ?

[…]

Or does running python -m build --outdir ${DESTDIR} . or installing a wheel also do other things for installation?

I’m not a packaging expert like some here, but as reading the source code and the docs confirms, all --outdir does is exactly what it says, changes the output directory of the built distribution packages from the default ./dist to one of your choosing, which you presumably want to set to whatever you want to store the output wheel for you to then repackage it. It doesn’t do anything additional, e.g. actually install the wheel. For that, you need installer as linked above, or unpack the wheel and move the .data files into the appropriate locations under the prefix, etc.

@riastradh

Does it make a difference whether foo-1.23 is, e.g., the content of a git tag from a typical project, versus an sdist created with python -m build sdist or an sdist downloaded from pypi?

I’ve never personally tried running build in an unpacked sdist, rather than just on the source tree, but so long as the MANFIEST.in, package_data and data_files are in sync, it should produce identical outputs? But I’d defer to the experts on that one.

[layday]

build does just that by default, packages your project into an sdist, extracts it and builds a wheel from the extracted sdist. This helps make sure your sdists are sound. As for whether you’ll get the same sdist or wheel out of the source code and the sdist, that depends entirely on your backend.

[riastradh]

@CAM-Gerlach

I’m not a packaging expert like some here, but as reading the source code and the docs confirms, all --outdir does is exactly what it says, changes the output directory of the built distribution packages from the default ./dist to one of your choosing, which you presumably want to set to whatever you want to store the output wheel for you to then repackage it. It doesn’t do anything additional, e.g. actually install the wheel. For that, you need installer as linked above, or unpack the wheel and move the .data files into the appropriate locations under the prefix, etc.

OK, so I asked what the equivalent of this is:

cd foo-1.23
python setup.py build
python setup.py install --root=${DESTDIR}

It sounds like python -m build --outdir ${DESTDIR} . is not, in fact, the equivalent of that at all, because it just creates an sdist and wheel archive in the outdir, and unlike the python setup.py install step it doesn’t install the content anywhere.

So what is the equivalent? Maybe something like this?

cd foo-1.23
python -m build
python -m installer --destdir=${DESTDIR} dist/foo-1.23-py3-none-any.whl

Is the command-line interface for python -m installer documented anywhere? I skimmed through https://installer.readthedocs.io/ but I don’t see anything about the command-line interface, and every page in the API reference has a warning on it that the API is not finalized, so I’m not sure what I can rely on here.

I’m sure I can cobble something together that works in a few cases I can test, by enough trial and error and reading the source, but I want to be sure I understand what the Python community actively intends to work reliably here, like python setup.py install did for many years (and, in most Python packages, still does).

[layday]

installer doesn’t have a CLI (yet). There’s no equivalent, but at least some people’s intention is to be able to invoke installer from the command line to perform the installation, yes.

[pf_moore (Paul Moore)]

@riastradh

I want to be sure I understand what the Python community actively intends to work reliably here

The intended usage is pip install, but you have said that doesn’t meet your needs (which is fine). So you’ll have to be prepared to do a certain amount of the work yourself right now. But ultimately, either installer will gain a CLI, or you will be able to invoke its API from a small wrapper script you write yourself^[1].

One possible confusion here is that the old setup.py install approach was rooted in the idea that everything used one tool, setuptools, to do builds. Modern packaging is working very hard to provide choice for package authors and users, by standardising interfaces rather than tools. So there’s not necessarily one “correct” way to do things, but rather there are tools for different use cases. Currently, a lot of resource is focused on “end user” tools like pip, and build backends like setuptools and flit, because those are what people are used to using, but lower level tools for system integration type tasks, such as build and installer, are being developed as well.

---

1. You can do that now. I don’t think you should be too concerned that the installer API is not finalised. It’s probably not going to change a lot, and changing your script wouldn’t be that hard anyway. ↩︎

[CAM-Gerlach]

@pf_moore

either installer will gain a CLI,

Other than it still being too early in development to have spent a modest amount of time working on at least a basic one, is there any rational reason why installer would not gain a CLI in the near future, or at the very least by the time it is production ready? I seem to recall there being an open issue/PR to do just that, in fact (by Filipe, IIRC)?

[pf_moore (Paul Moore)]

@CAM-Gerlach

Other than it still being too early in development to have spent a modest amount of time working on at least a basic one, is there any rational reason why installer would not gain a CLI in the near future, or at the very least by the time it is production ready

I’m not an installer developer, so I have no idea, TBH. But I thought it was intended as mostly a library, so while a CLI would be a convenience for users with simple needs, it would likely leave more complex uses^[1] to be handled by a dedicated script calling the library’s API.

I’d expect at some point that py -m installer some.whl will work. But I don’t know if a --prefix or --destdir command line option would be considered reasonable to include (install schemes are complicated beasts…)

---

1. Which, judging by the length of this thread, I suspect the OP’s needs would be described as. ↩︎

[riastradh]

@pf_moore

One possible confusion here is that the old setup.py install approach was rooted in the idea that everything used one tool, setuptools, to do builds. Modern packaging is working very hard to provide choice for package authors and users, by standardising interfaces rather than tools.

So this is indeed confusing.

In the past (and present, but maybe(?) not near future), the interface that reliably worked for almost all Python packages was python setup.py build && python setup.py install --root=${DESTDIR} (whether or not setup.py uses import setuptools under the hood as a tool).

But what I’m hearing here is that we’re supposed to use the specific tools build and installer—well, except there’s apparently two different variants of installer, one from FFY00 (part of python-bootstrap) which has a CLI and one from pradyunsg which doesn’t, as I learned today. And I’m still not sure how we’re supposed to use them, i.e., what the intended interface is.

@pf_moore

I’m not an installer developer, so I have no idea, TBH. But I thought it was intended as mostly a library, so while a CLI would be a convenience for users with simple needs, it would likely leave more complex uses[1] to be handled by a dedicated script calling the library’s API.

I’d expect at some point that py -m installer some.whl will work. But I don’t know if a --prefix or --destdir command line option would be considered reasonable to include (install schemes are complicated beasts…)

I think our goals are fairly simple.

Background:

• If the Python program foo-1.23 is to be used by import foo, Python will look in some path baked into the python executable like /usr/pkg/lib/python3.11/site-packages/foo to resolve the import, and in turn the code in foo might look for data files under there.
• If it has a command-line entry point, there’s a script at /usr/pkg/bin/foo with a #! line for sys.executable.
• (In this example, Python was built with sys.prefix = '/usr/pkg'—point is, it’s baked into the underlying python executable, not a parameter that’s relevant here.)

So I’m looking for a way, given the source tree for foo-1.23, to:

1. build any bytecode or entry point scripts or supporting data files from source code, and then
2. put all those files in their places, relative to a staging root directory like /tmp/work/foo-1.23/destdir so the build process itself doesn’t require privileges or interfere with the building environment.

Pretty much all general-purpose package managers—pkgsrc, FreeBSD ports, Debian, Fedora, Homebrew, &c.—have the same goal here, so I would hope there’s an easy way we can all do it!

(Tools that do more than this like pip will do essentially this as a subroutine. But getting pip not to do additional things like talk to the network, consult its own index, cache things, do its own dependency resolution, &c., takes a bit of work and I’m not sure all the measures we take in pkgsrc to avoid all that are reliable.)

[pradyunsg (Pradyun Gedam)]

Hiya!

installer has a working Python API which, depending on your use case, could be sufficient. The main reason for the stability warnings in the API is that there’s a chance that there might be some stupid bug in there that’ll need a backwards incompatible change.

FWIW, the only blocker to switching pip over to using that package is (a) someone finding time for it and (b) deciding on how to manage the behaviour difference of what happens when a file-to-write already exists.

If you need a CLI on top of it, it should be possible to write one today. As mentioned Add CLI by FFY00 · Pull Request #66 · pradyunsg/installer · GitHub is the open PR to add a CLI directly to the package, which has gotten a decent amount of feedback and has an ongoing discussion right now. There’s also a link to a package that provides a CLI based on installer too in there.