I’m sorry if my words seemed harsh, let me rephrase: it would be very nice and much appreciated if we would not need to wait for an entire week to fix a bug that affects a wide range of projects and workflows.
I know it’s not easy to please everyone, and it’s bound you’ll break someone, sometimes. I maintain both tox and virtualenv nowadays, and those are also crucial projects. When I schedule a release I always try to be available the next two days to release any major breakage the release might cause.
I just wanted to say, I appreciate very much all the work people put in into making pip a better tool and my honest gratitude for it. I would recommend for breaking changes a different approach, revert first, deploy fix and then discuss how to go ahead. Instead of how now things stand, keep breakage, come up with a solution all agree, deploy fix. I’m saying this to improve confidence of people in our tools, I’ve encountered many people who were often smashing tools because new version always break and they have to pin CI all the time to not stop development on crucial projects. Again just my humble opinion and feel free to disagree.
With the reversion, I believe there are certain things that won’t work right. Like, in 19.0.3 I believe passing --no-use-pep517 caused build-system.requires to be ignored when it shouldn’t be when no build-backend was specified. This is one thing 19.1 should have fixed. I wonder if this will now cause problems since people have been instructed to start passing --no-use-pep517.
This does hit pipenv quite hard as we use --no-use-pep517 in combination with --no-build-isolation as a fallback during installation failures for cases where environments might use versions of pip + setuptools that took . off of the path.
Given the current pip release this puts is in a kind of weird predicament because a lot of people (self included) have pyproject.toml files which don’t specify a backend but do specify requirements. The mere presence of that metadata forces pip to treat this as a pep517 project now, so anyone using -e as an installation mode will automatically experience the failure. For us, if that user then adds the build-backend key, they won’t be able to use editable installs & existing pipfiles and lockfiles will be broken.
For remote URLs anything pointing at a project with a pyproject.toml that is also editable is now broken. It seems like there may not be any way around this besides telling everyone to go modify their dependencies and remove the editable flag or to set the environment variable that prevents virtualenv from installing the latest version of pip when creating environments.
I accept that there are a lot of issues with editable installs but I think this may merit a deprecation cycle or 20. I know I’ve chatted with a bunch of folks about this in the past, would love to chat with you too @cjerdonek / @pradyunsg / @pf_moore because we are doing some substantial jumping through hoops for dependency resolution on editable / VCS / URL packages that could be simplified with some consideration. Changes to this cause serious breakages for us even if it is ultimately thought to be an improvement.
It’s not clear, what are you suggesting should be deprecated? I think we’re all just talking about “what do we do while we work on the standard?” We can’t and shouldn’t go through deprecation cycles to transition people from one temporary behavior to another temporary behavior. Best to stick with a behavior that doesn’t break anything that currently exists (e.g. reverting), and try and fix bugs that come up with band-aids that don’t create any new reliance on weird hacked-together behavior.
To clarify, are you saying the 19.0.3 behavior hit pipenv quite hard, or that the reversion will hit pipenv quite hard given that 19.1 has already gone out?
I ask because there is still the possibility of restoring just the part of 19.1 that causes build-system.requires to no longer be ignored if --no-use-pep517 is passed. But given the urgency, that should only be done IMO if we know or expect that a simple reversion could cause additional problems.
Happy to talk! VCS / editable installs are of special interest to me (independent of anything related to PEP 517), and I am / have been actively working on improving the code base and functionality in that area.
From what I understand he’s saying exactly the opposite. Passing --no-use-pep517 in combination with --no-build-isolation should make editable installs work under all cases, both when pyproject.toml is present or when the requires+build-backend is specified. I would advise to go ahead with the release of the reverts.
The question of how to address the issue that originally prompted these changes is still outstanding, as well as the other possible issue @cjerdonek mentioned:
I get that this is a major problem, but IMO there’s still a lot of pressure being exerted to get the reversion rushed in. There are genuine questions that have been raised in good faith, and the key parties here have very little spare time - for better or worse. It’s increasingly difficult to focus what time we do have available on ensuring we don’t make things worse with a second problematic release.
Given that the reversion has been available in master since yesterday, can any of the people pushing for a quick reversion confirm if they’ve tested the current master against (a copy of) their live workflows and that it’s solid? That would be very helpful in reassuring the pip developers that things are looking OK to go ahead with the reversion as it stands, and worry about further fixes in a more considered fashion.
pip master works once more for my use-case (which was broken in 19.1). The use-case is “pip install -e” works again for https://github.com/crossbario/crossbar which uses pyproject.toml just for configuring the towncrier tool.
This is odd, I’m not able to get this to work with the latest pip master. I have a build-backend set as setuptools.build_meta, and I get ERROR: Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml. Is this what is intended?
@btskinn I’m not using that, so I’m not sure what’s wrong, but I think I’ve read enough of what @cjerdonek says to understand his concerns about future-proofing. So the best way forward for now may be to just remove build-backend until the spec process gets worked out. I’ll figure out a way to use direct setuptools to build my distribution packages in CI.
I’ve found you can include toml in build-system.requires and then use that in setup.py to re-extract the build requirements and add them as an extra, such that pip install .[build] creates an environment where python setup.py is valid. It seems like a hack to recursively bootstrap the build systems like this, but I guess it’s better than relying on coincidentally useful undefined behavior.
I have a feeling this predicts what’s going to happen if the revert is released as-is. We’ll need to make use-pep517 somehow not break things (maybe add a hack to drop it when editable is set?)
This is precisely the kind of thing I was warning about in a couple of my comments above (included again at bottom), though the case above is slightly different from what I mentioned. Basically, 19.1 broke things for certain people, and the advice for them to unbreak was to add --no-use-pep517. But reverting as-is could re-break things for those same people, with the “fix” being to remove the --no-use-pep517 which they just added. The question was whether that’s an acceptable user experience to folks given the urgency.
I agree it’s recently enough if you consider the whole community, but not for those who already adapted. We’ve broken things for them and tell them to fix their own things, and now we break their things again and tell them “yeah sry, you need to fix your thing again”? That would have left a very bad taste in my mouth if I were in their position.