That was my suggestion some time ago. What I thought @steve.dower was suggesting was that instead of inventing a mechanism which allows us to know those constraints ahead of time, we “discover” them as we go along, when we see an installed package with no RECORD file.
But yes, I think that PyPy somehow publishing a set of constraints up front is the simplest approach. It gives cross-platform resolves some issues, as they can’t query the installation for that data, but that’s something that cross-platform resolvers already need to deal with, so nothing new there.
Aha yes, I agree that that would be a bit more difficult. But still, I guess the installer could scan for installed packages beforehand and check if they could be uninstalled and treat them as fixed if not - but this is a potentially significant amount of work, not sure if that is worth it.
Although, even if the information is only discovered later on, shouldn’t it be discoverable the moment the package appears in a dependency list anywhere? It would just require scanning for local installers everytime a new package is added.
So I think it would be possible, just not worth it compared to having a standardize file somewhere for “fixed packages”.
Maybe we could add a “constraint source” attribute to a constraint? Then normal constraints could have the filename in there, and these new constraints could be given a source of “Python installation” or something. I think it would be some work to add, but not impossible. And in any case, I don’t consider a suboptimal error message to be a showstopper here. The first step is to find a way that installers (in general, not just pip) can discover the necessary constraint information. The second step is to implement the necessary changes to resolve correctly. Good error messages can be a follow-up to that (after all, pip’s existing error messages aren’t that good, in spite of the work we’ve done to improve them - this is “just” another case we need to work on).
I’m not sure what a good solution here is, but i want to add some considerations for universal lockfiles: In many cases, it’s possible to create a single lock that covers all platforms and (supported) python versions, all resolved in a single pass. If pypy has separate constraints, this means that we either force all other platforms to the same constraints as pypy, or we force a separate resolution for pypy, or make pypy opt-in.
Since the packages in question as i understand are managed by the pypy maintainers, would it be possible to make them shim packages? Their requirements would defined loading the real package or code on all other platforms, and the vendored version on pypy. This would make the tight coupling between pypy and e.g. cffi invisible to a resolver.
I wouldn’t have suggested doing it “as we go along”, but scanning ahead of time.
I know people love their sub-second resolve times, but realistically, everyone prefers installs that work. It’s worth the time to see what’s already there before choosing what needs to be installed.
This is a very similar problem to the one that EXTERNALLY-MANAGED solves. It goes a bit further in that you want to disallow upgrades as well as disallow uninstallation. That could be a flag in EXTERNALLY-MANAGED. EXTERNALLY-MANAGED already applies to specific installation directories, so it would separate the bundled modules from the user-writeable site-packages.
There is a path to smoother coordination with package-managers, and avoiding vendoring libraries, by embedding shims rather than vendoring the full library. Push some of the integration upstream. But this requires the library to then declare a dependency on the compatible version of the shim.