Packaging forks

[MrMino (Błażej Michalik)]

@westurner

It may be easier to discuss a specific set of requirements in order to determine what the current behavior is and what Provides-Dist support would add when this requirements.txt is installed with pip install -r requirements.txt :

# requirements.txt
pip
pipZ      # Provides-Dist: pip
pip-fork  # shadows pip (site-packages/pip)

My idea was to make pip reject this altoghether when it finds out that pipZ has Provides-Dist: pip, in the same way you cannot have

pip==x
pip==y

In a single requirement file.

@westurner

We’re certainly not suggesting that every time I pip install pip I need to do reverse lookups of packages that Provides-Dist: pip ?

Local lookups or index lookups? Index - no, local (site-packages) - yes, e.g. synthetic .dist-infos is one way (see my post above).

@westurner

• When I pip install pip , pip does not record which packages were installed by which command when (a log of package operations) and pip does not check for file collisions (for an intersection between the list of installed package files and the list of to-be-installed package files).

…

It would also be great to be able to determine from which index a package was installed through pip list ; and then whether the hashes and signatures match a key in a keyring for those requirements maybe in pip check ?

It does not do it right now either, so no-factor I guess? Not that I want to argue it’s a wrong idea, it’s just I’m trying not to loose focus over the goal I have in mind, and to me this is a separate topic. Fully secure supply chain is not included in my goal - that’s something that is way above my head.

Taking supply chain security into consideration is not the same as fully securing it.

@westurner

In order to determine that:
…

Are these necessary or sufficient:

…

Most of the things you list are non-goals for what I’m trying to achieve here - at least I think they aren’t? Why would I need to know whether package forked_foo is installed from a different index for Provides-Dist to work? Why would I need to know which command installed what? There’s no way to determine that information for normally installed packages currently, and so, in the same way keyring checks aren’t, this is not a factor for providing limited support for forking.

I’m hesitant to hypothesize compatibility with features that don’t exist yet.

---

Edit for the sake of not talking past each other: pip install foo being able to resolve to forked_foo is not my interest here. My interest is pip install forked_foo being proper substitute for pip install foo, if forked_foo contains Provides-Dist: foo.

[uranusjr (Tzu-ping Chung)]

@MrMino

This thread is basically me eagerly probing for whether there could be a concensus on this . I’m also thinking about creating a separate tool just to test it out, for the sake of making it more concrete.

Get concensus of what? The current specification already specifies the field to support your use case of it. I think most in this thread are kind of unsure if your approach is good because none of us really thought it through, but nobody can’t say you’re wrong for the same reason

I’d just start building that thing right now if I were you. If the idea works, people will pick it up when you tell them about it. Every single tool we have in Python packaging went through basically that story, as far as I know.

If what you’re looking for is some kind of confirmation, I think your idea would definitely work in theory, especially if you add a limit that the user must specify forked_foo as a top-level dependency, and must not specify foo at the same time. If you use resolvelib for your resolver, you’d want to collect Requires-Dist fields when you discover forked_foo candidates implement the provider so that when find_matches("foo") is called, return candidates backed by Provides-Dist fields in forked_foo instead of real foo candidates.

There are some practical difficulties, however. For example, since Requires-Dist fields are in a distribution’s metadata, you won’t be able to download distributions lazily only after they are chosen, but have to eagerly download forked_foo versions to resolve a foo version range. But that shouldn’t be a show-stopper if you’re only intending to solve some special cases to the feature, and even downloading packages eagerly might not be too bad an idea; once the use case gets adoption, you can propose changes to the Simple Repository Index specification to support optimisations.

I can’t guarantee your idea would work, but you should just do it. In the worst case scenario that the idea is a complete failure, at least you’ll be in a good position to propose changes around this to make Provides-Dist actually work.

[MrMino (Błażej Michalik)]

@uranusjr

I’d just start building that thing right now if I were you.

I guess I just needed this kind of a nudge for that, thanks

[tiran (Christian Heimes)]

@MrMino

pip install foo being able to resolve to forked_foo is not my interest here. My interest is pip install forked_foo being proper substitute for pip install foo , if forked_foo contains Provides-Dist: foo .

I’m interested in the same functionality for GitHub - tiran/certifi-system-store: certifi-system-store, a certifi hack to use system trust store on Linux and FreeBSD. The package provides a replacement for popular certifi package. For now the user has to perform an extra step to create a symlink certifi-3000.1.dist-info -> certifi_system_store-3000.1.dist-info to trick pip.

[tiran (Christian Heimes)]

@westurner

A unified TLS API for Python” appears to have at least 3 relevant bullet points

That PEP completely unrelated to packaging forks. My fork of certifi has nothing to do with my old PEP, too. Why are you bringing this up?

[pradyunsg (Pradyun Gedam)]

I’m sorry but TUF, sigstore etc have nothing to do with the discussion we’re having here.

This is about the metadata in the distribution files and not about how the files are being served.

[westurner]

@MrMino

pip install foo being able to resolve to forked_foo is not my interest here. My interest is pip install forked_foo being proper substitute for pip install foo , if forked_foo contains Provides-Dist: foo .

Define “proper substitute” in regards to certifi-system-store? Oh well. FWIW, you can just call a package whatever and put files in site-packages wherever (in multiple directories). Provides-Dist is a poor - or rather simply a metadata attribute - collision-detection mechanism.

Why should I trust any package author’s claim that A is a proper substitute for B?

Once I’ve installed forked_foo, what is it that I’m then verifying after I’ve already trusted it?

Thus keyrings, cached cert and manifest file hashes and fingerprints, and trust roots.

pip check: This page has moved - pip documentation v23.3.2

@pradyunsg

I’m sorry but TUF, sigstore etc have nothing to do with the discussion we’re having here.

This is about the metadata in the distribution files and not about how the files are being served.

This issue seems to have overlapped with the issue of verifying other-versioned packages (instead of “forks” with a different name).

You’ve now hidden my input by moderating those posts.

I’ll leave you to it.

[pradyunsg (Pradyun Gedam)]

@westurner

You’ve now hidden my input by moderating those posts.

I’ll leave you to it.

I don’t have any moderator powers here (thank god), so I certainly can’t hide your posts all on my own. Full disclosure: I’d only flagged the first one.

What you’re seeing, is what Discourse does to a post when multiple folks (3+ IIUC) report a post to the moderators IIUC. I think the wording implies the same.

It could also be that Discourse moderators decided to hide the post based on someone’s flagging, which they do via a reasonable concensus-based process (I believe so, given how they have handled past flags from me).

[MrMino (Błażej Michalik)]

@westurner

Why should I trust any package author’s claim that A is a proper substitute for B?

This thread is not about creating a cryptographically secure supply chain. Please stop bringing it up here. If you care about these issues, I suggest creating a separate thread for them.

What I and @tiran care about, is making sure that we can create packages that can be used to substitute others (e.g. dead projects) in dependency resolution.