Note: @pradyunsg posted after I’d started this comment. And I agree with what he says, that the core question for this topic is answered. But I will post this anyway, as I do want to address the question of incremental changes.
tl;dr; I think we can (and should) allow certain types of new feature in minor version changes, and I’m very uncomfortable with the idea that all new features require a major version, as it pushes us towards a state where gradual improvement isn’t possible.
(Previously written message follows)
I think there is some truth in the idea that packaging changes (and in particular changes to the wheel format) are slowing down, and we’re losing some of the agility around smaller, incremental changes. I don’t think it’s a formal policy, but it is a consequence of our policies (the PEP process can be rather heavyweight for small changes, and we don’t really have a good policy on how to exempt “small enough” changes from needing a PEP).
I think we do have a need to be cautious about compatibility and breakages when we release changes, but we also have a responsibility to not let things stagnate. And to be perfectly honest, I think we are tending to be over-cautious these days. It would be nice if the work on a “Packaging Vision” that @smm is leading, could result in a clearer statement of how we expect to handle “legacy” project distributions, workflows, and formats, so that we could realistically make decisions on what sort of breakage is acceptable when proposing new standards.
I’m not entirely sure what type of “design” you mean when you say the wheel format is locked out of incremental improvements. Is it purely the statement that installers must warn on minor changes and error on major ones? Because if so, then we only have one wheel format right now, so now is the perfect time to change this - I’d happily approve (if people consider that it’s up to me to do so) a change to that statement as a “minor modification” to the spec, so we can just get it done.
If you have something different in mind for why the wheel spec is locked out of incremental changes, can you explain? And in particular, how is the wheel spec more problematic than (for example) the sdist spec, or
pyproject.toml, or any of our other standards? Because if we’re inadvertantly designing standards that can’t be improved, let’s work out what we’re doing wrong and fix it!
To give a concrete example (so this isn’t all just handwaving ), adding symlink support to wheels would need to cater for what installers must do if the target environment doesn’t support symlinks. Fail, presumably. So the failure modes are:
- Wheel contains symlinks
a. No symlink support, new installer - install fails.
b. OS symlink support, new installer - install succeeds.
c. OS symlink support, old installer, major version bump - install fails.
d. OS symlink support, old installer, minor version bump - install fails because the symlinks (noted in
RECORD) are missing.
- Wheel does not contain symlinks. OS Symlink support is irrelevant
a. New installer - install succeeds.
b. Old installer, major version bump - install fails.
c. Old installer, minor version bump - install succeeds with warning.
Of these scenarios, only (2b) seems bad to me. Which says this should be a minor version bump. So why can’t we do a minor version increase to add symlink support, and why do you say the wheel format is by design locked out of such incremental improvements?
The trade-offs are very different for changes where all new-format wheels are unreadable by old-format installers (where case (2) above does not exist). That is what I’d consider to be a major change, though, and is where we should focus on thinking about migration strategies (which is what I’d thought this topic was intended to be about, but we’ve drifted somewhat )
To be clear, I’m not trying to say that you (@pradyunsg) are blocking incremental increments. But I do think that this whole discussion is tending to see the risks as more important than the benefits, and we’re becoming too risk-averse as a consequence. Although not many people are actively participating in the discussion here, so maybe the majority opinion is different.