Wanting a singular packaging tool/vision

[rgommers (Ralf Gommers)]

@pf_moore

So the key point here is to be able to check in advance whether a build would fail and at a minimum report “You need X, Y and Z, please install them before proceeding” or for more capable installers (essentially ones that are capable of installing non-Python dependencies) actually install the dependencies and then proceed to the build?

That’s would be great indeed, yes. I think that would be a second step/phase, only to be enabled once another package manager actively opts in to this system, so pip knows it can do this kind of querying. Because it’s very well possible that a user does have the right dependencies installed, but pip has no way of knowing yet.

I’d imagine that the first step, as an incremental improvement, would be:

1. Still start the build, just like right now.
2. If it fails, pip already prints an error message (pretty confusing right now, but it ends in “This is an issue with the package mentioned above, not pip.”. At that point showing also “this package has system dependencies in order to build, these are: …”.

@pf_moore

Excuse my ignorance here, but I assume the first thing we’d need is some means of naming such dependencies in a platform and installer independent way? Is that correct? For example, “g++” isn’t a very good name for a dependency on a C++ compiler for a Windows user, who would typically be using Visual C++. Is that the sort of thing that you’d expect to cover in the PEP?

Yes exactly, that naming is what I was getting at with the mapping names part. g++ indeed doesn’t work. The dependency in this particular case is something that can be fulfilled in multiple ways, so I’d imagine it would be something like <virtual> c-compiler or compiler('c') (excuse the lack of valid toml, that’s too detailed right now). There aren’t many of those virtual dependencies, and they can probably be explicitly enumerated. Most other dependencies that can’t be expressed are regular names for libraries or tools.

@CAM-Gerlach

it probably wouldn’t be hard to add a bit of per-distro logic if needed for finer-grained mapping.

I’d think that is best left to distros and other package managers themselves. Python packaging can provide them an interface to register it. Otherwise we create a significant of work in some central place that a few unlucky individuals get to maintain.

@CAM-Gerlach

You could do the same thing for non-Python packages similarly sourced from an upstream index (Rubygems, CRAN, CPAN, npm, etc).

Maybe, not sure yet. Two language-specific package managers is qualitatively different, since it’s a lot less clear that PyPI names are canonical and dependencies will only go one way (with system packages, those are always a base that Python packages come on top of). I’m aware there’s a lot of history here, and I believe one previous attempt at a PEP. I have a list of raw notes somewhere. This is the kind of thing to address in the scope part of the PEP.

@njs

pip looks for site-packages/*.dist-info dirs to figure out what’s installed, so it should handle these correctly already?

Ah yes, good point. I think actually that conda already does this - it’s not fully robust, but typically works. I’m not sure if it scales to all package managers like that though. Nor does it address name mapping for system dependencies, or possibly working towards getting all non-pure packages from the other package manager, even if they’re not already installed. But yes, .dist-info can be a relevant communication mechanism here.