PEP 610 usage guidelines for Linux distributions

Packaging
1

Hello. I have just learned about PEP 610. As Fedora Python maintainers, when we designed our “modern” (read PEP 517/518 instead of setup.py build and setup.py install) RPM packaging, we have chosen pip as the tool we use. In practice, we do basically 2 steps after fetching and installing the dependencies to the build chroot:

Build:

$ python -m pip wheel --wheel-dir pyproject-wheeldir --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --no-clean --progress-bar off --verbose .

Install:

python -m pip install --root $RPM_BUILD_ROOT --no-deps --disable-pip-version-check --progress-bar off --verbose --ignore-installed --no-warn-script-location pyproject-wheeldir/*.whl
sed -i 's/pip/rpm/' $RPM_BUILD_ROOT/usr/lib*/python*/*.dist-info/INSTALLER

Up until now, this was working fine. With PEP 610, we now have a *.dist-info/direct_url.json file created. As a result, packages built that way show up in pip freeze as follows (only pluggy was built this way in the example):

...
packaging==20.3
pluggy @ file:///builddir/build/BUILD/pluggy-0.13.0/pyproject-wheeldir/pluggy-0.13.0-py2.py3-none-any.whl
py==1.8.0
...

For our users, this information is not useful. The /builddir/build/BUILD/pluggy-0.13.0/pyproject-wheeldir/ directory, neither the pluggy-0.13.0-py2.py3-none-any.whl file exist on user systems, they merely exist in the build chroot during the RPM build (and are discarded after the build is finished).

What is the best practice for Linux distributors who build RPM (DEB, etc.) packages with pip? I am inclined to “simply” stick:

rf -vf $RPM_BUILD_ROOT/usr/lib*/python*/*.dist-info/direct_url.json

…into the proces, to rollback to “previous behavior”. Is it the best practice? Should this be documented?

Crazy (not well thought) long term idea: Should we propose a new extension to PEP 610 to be able to say "this package is installed from an RPM package of xyz? Something like:

{
    "url": "rpm://python3-pluggy-0.13.0-1.fc33.noarch",
    "archive_info": {
        "name": "python3-pluggy",
        "epoch": "0",
        "version": "0.13.0",
        "release": "1.fc33",
        "arch": "noarch"
    }
}

(I know rpm:// is not a valid URL schema, please don’t bike-shed the data, just the high level idea.)

PS Sorry to pitch in after the whole thing got approved an implemented, I wasn’t aware of this before :blush: Given this is provisional, I think we are still good to define best practices for Linux packaging.

cc @encukou @torsava @vstinner @pradyunsg @sbidoul @cjerdonek @ncoghlan

1 Like
2

Off topic, where can I find information on this new PEP 517-backed RPM packaging tool? I had a couple of people reporting issues on a project of mine asking for setup.py because

for distro packaging (this is my situation too) we need to use setuptools, and we don’t generally use wheels either as that just adds additional container requirements for no good reason

I don’t agree with this, but it also does not help if I can’t provide an alternative either. It would greatly help the discussion if there is canonical resource I can point to and say “hey, Red Hat folks are doing this too, there’s no reason you don’t.”

3

It is very provisional, but see pyproject-rpm-macros. Feel free to open a separate topic about this. With @encukou we are looking into new Fedora Python packaging guidelines (the current guidelines still use %py3_build (setup.py build) and %py3_install (setup.py install --skip-build)).

1 Like
4

@hroncok thank you very much for testing the pip beta in depth!

Would it work for you if instead of

pip install ... pyproject-wheeldir/*.whl

you do

pip install ... --no-cache-dir --no-index --find-links pyproject-wheeldir/ projectname

With that approach no direct_url.json will be recorded.

projectname could be obtained with, e.g., $(ls pyproject-wheeldir | cut -d '-' -f 1).

[updated to add --no-index, thanks @EpicWink]

5

I can certainly try to do it that way. In the end, the result is the same as deleting the file, correct?

6

Yes, correct.

[updated] direct_url.json is part of the RECORD file.

7

I confirm it works, thanks.

8

That is not guaranteed to install the wheel in pyproject-wheeldir/ if the same version of the package is on PyPI (in fact, it will use PyPI if there is a newer version upstream).

You can make the guarantee if you build the package with a post-fix in the version (eg upstream 1.42.3, RPM build 1.42.3+rpm.1), then require that version

pip install --no-cache-dir --find-links pyproject-wheeldir/ 'projectname==1.42.3+rpm.1'

Alternatively, disable the index, so you don’t need to build the package with a different version (thanks @sbidoul)

pip install --no-deps --no-index --no-cache-dir --find-links pyproject-wheeldir/ projectname
9

I like this idea. The only question remains what url schema we reserve for the URL, I’m tempted to go with the install tool name.

10

Or this should do, right? The wheel names are that deterministic.

ls pyproject-wheeldir | sed -E 's/([^-]+)-([^-]+)-.+\.whl/\1==\2/'
11

Indeed, I forgot --no-index.

2 Likes
12

There can be multiple install tools. e.g. for the rpm package, it can be lower level tool (rpm) or a higher level tool (dnf/yum). Since the higher level tool uses rpm anyway, that would be rpm in this case and dpkg (i.e. not deb or apt) on Debian. Correct?

1 Like
13

I have not given much thought about that yet, but at first sight this does not fit well with the PEP 610 philosophy.

The presence of direct_url.json means a PEP 440 / PEP 508 direct URL was used to specify the distribution to install. Its absence means the distribution was obtained from some sort of package index, which is implied to be environmentally specified, and which we have no way to record explicitly today.

It probably makes sense to record the package index that provided the distribution, but I’d say this would be another spec than PEP 610.

2 Likes
14

Yes. The wheel spec says that the distribution name is before the first - in the wheel file name.

15

On second thought, it is not exactly the same, because pip adds direct_url.json to the RECORD file.

1 Like
16

Oh! Good to know.

1 Like
17

BTW, since you change the content of INSTALLER, you probably want to change its hash that is in RECORD.

18

Good point. Would it fly if we ask pip to have --set-installer=rpm option for this?

As a side note, we also need to put all opt levels bytecodes into RECORD.

19

I’d suggest opening a feature request on the pip issue tracker and discuss it there?

20