Insights into how poetry.lock works cross platform

Being able to see from the sdist metadata that dependencies are the same everywhere, without needing a build at all, is even better. I’d be OK with a locking tool that produced cross-platform lockfiles for sdists with non-dynamic dependencies, and lockfiles that were specific to the platform they were created on if one or more sdists needed declared their dependencies to be dynamic.

3 Likes

It’s not particularly hard to construct reasonable situations where this can be useful. For instance, you could have a wheel for an older version of macos prior to some system library being included as part of the “base” install that uses a backport or alternative mechanism for implementation, then a wheel that targets a newer version of macos that does not.

I don’t think it makes it impossible to support cross platform lock files, since you can (as you mentioned) fetch the dependencies from the wheel files as part of generating that lockfile. It makes it take more effort, but that’s unavoidable IMO since it’s sort of a fundamental part of how our packaging works currently and attempting to close the barn door after the horse has left the stable is going to (silently) produce invalid lock files, which seems like the worst possible option.

But fetching every wheel’s metadata is only hard right now, PEP 691 is accepted and implemented, and PEP 694 is being worked on right now. Those are stepping stones on the path to having dependencies show up in the repository API, which means that you can get all of those dependencies without fetching every single wheel.

It’s impossible to support portable lockfiles for arbitrary sdists as things currently stand, and just writing in the docs that “hey you can’t do that” isn’t going to stop people from doing it. It will just be silent errors that constantly pop up and cause a constant low level stream of friction.

If we wanted to do this, the only path forward would be to make dependencies not allowed to be dynamic in an sdist, but you still have the problem of the 3.6 million releases with 6.5 million individual files that don’t fit into that on PyPI today, and a hypothetical tool needs to deal with them as well.

Which, in the past, there was broad push back on doing that in sdists, because there are projects who do interesting things to select dependencies, and they just choose not to publish wheels in some cases as well because sometimes not even environment markers or the wheel tags are expressive enough for them.

I don’t object to wheels having different dependencies in principle, but should we not add to environment markers so they can express as much as the wheel tags, so the most common use cases can switch to environment markers and save a lot of people’s time?

3 Likes