PEP 641: Using an underscore in the version portion of Python 3.10 compatibility tags

+1 on providing explicit guidance for 3rd party projects on this.

I guess I should expand on that more.
Going from 39 to 310 and then to 400 seems straightforward, in the sense that anyone who finds a 40 tag should know what to do.
I don’t mind individual occurences of 3xx (like the wheel tag) changing from 3_xx or 3.xx. But version_nodots is not an individual occurence: it is widely used in and outside CPython (for various constrants, sometimes even “plain [0-9]+ integer”).

1 Like

The PEP is purposed to solve the problem which will occur only in Python 31.0 (because 1.10 and 2.10 did not exist). We have a plenty of time.

I propose two alternatives:

  • Always use two digits for minor version number. First affected version is 4.0 (as “400”). Additional advantage – versions are always correctly sorted by the numerical value: int('310') < int('400').

  • Use the same number of digits for major and minor version numbers. First affected version is 10.0 (as “1000”).

In any case we does not need to change anything right now.

2 Likes

As a PEP delegate, I am following close the discussion but I am trying to not influence it in any way so I can be as much objective as possible (I already said my initial thoughts on our first meeting in which I mention that my priorities would be unblocking the wheel situation for 3.10 as soon as possible and minimizing breakage as much as possible, but many new considerations have been raised since).

My biggest fear is that we are not considering something important that will only be revealed when 3rd party projects are subjected to our decision.

Also @storchaka raises some important points and alternatives that I would like to gather feedback on:

For now, as I understand from @hroncok analysis the breakage can be annoying to some projects but it will not be something fundamental that will involve a considerable cost. Given that this is a new release we are talking about, I find that those changes are not a deal-breaker.

I would also like to get some feedback here from the folks that initially requested this from Pypi.

Also, @bernatgabor, do you think this is something that will affect tox or its users deeply?

1 Like

I mean, Brett already said that packaging went for 3_10 because he preferred the aesthetics, but if we’re finding it unworkably inconsistent across the board then we can tell Brett to deal with it :slight_smile: (via rejecting the PEP)

2 Likes

tox and virtualenv already use only the first digit to denote the major version and is widely deployed so it’s unlikely are going to change. Granted once we release python 10 we’ll run into problems with this strategy but I’m not expecting this to be anytime soon. How the stdlib paths are handled should be less of a problem, as both projects use sysconfig to acquire them. Some plugins might hardcode them… but it will be up to them to fix it.

Sure, if readability == aesthetics then that’s a fair summary of why I made the change in ‘packaging’.

Yep, that’s the point of the PEP.

1 Like

I have reviewed the PEP several times and I here are my thoughts:

  • Many of the discussions regarding the format are based on the current status quo. This is of course an very important point to consider, but I am also considering the format itself as if this were the first time we are thinking about it. Of course mainly of the considerations are regarding breakage and backwards compatibility.

  • Seems that the bad usage related to %if %{python3_version_nodots} < ... are restricted to EPEL (in the outer wilds I have seen it only on SUSE). It seems that correcting these is acceptable as it is not widespread (please, correct me @hroncok @encukou). In other build systems, I have seen a considerable amount of: python_version_nodots=$(echo $PYTHON_VERSION | tr -d '.'), For what I understand for rest of the usage of python3_version_nodots that is not directly comparing we don’t expect any important breakage, just a name change.

  • There are some interesting points regarding integer comparison, but I am not sure that we should support direct comparison of tags based on integer comparison alone (we should not intentionally break it, but losing this property should not be a show stopper IMHO). Additionally, as stated in the PEP, the usage of the _ has the advantage that int("3_10") and int("310") will preserve comparisons inside Python (which I know is not the only place where these comparisons are made). In any case, as far as I understand, usage of python3_version_nodots is not critical enough to really weight that much on preserving integer comparison (which as stated in other places, is quite a dangerous way to compare tags anyway). If someone really thinks this property is really critical then the PEP should be updated to address this problem explicitly (or to counterargument the concern).

I think this is the most important point that I think the PEP currently doesn’t address. The most fundamental aspect that I don’t feel comfortable is the potential proliferation of all formats. It is true that every tool can decide how to adapt to this but certainly as @encukou and @pf_moore indicated I think that for the PEP to be accepted there should be some recommendations for 3rd part projects regarding the naming (and possible “migration”). We are doing this PEP precisely so this change does not pass silently, so we should also explicitly address this, I think.

Does anyone miss any important point that is not reflected here?

2 Likes

I was referring more if tox can addapt nicely to this regarding the version of Python to use. In other words, any of these changes are expected to affect how users refer to environments currently in tox (such as py27, py39, …).

1 Like

Today tox supports only py310. However, adding support for py3_10 is not too hard. The change actually needs to happen partially in virtualenv - see https://github.com/pypa/virtualenv/blob/main/src/virtualenv/discovery/py_spec.py#L45-L56, but we can just accept there optionally a _; so would not be too hard to implement. For what’s worth virtualenv already accepts explicit version constraints, and for that uses the . character instead of _.

> TOXENV=py%{python3_version_nodots} tox 

Technically breaks, but here you already relied on that tox follows python3_version_nodots syntax, which is not true (and if someone actually used it would not be too hard to add support for it, for what it’s worth). Also, I’ve never seen anyone use it like that until today.

1 Like

Is there a way to get a timeline for a decision? The weeks are going by and it is currently impossible to install packaged c-extension modules as wheels on 3.10 until this is resolved.
Matti

1 Like

I am waiting for @steve.dower or @brettcannon to comment on this:

1 Like

If the argument for disambiguation doesn’t apply to all formats, it doesn’t apply to wheel tags either.

If we can’t apply it consistently, we shouldn’t apply it at all (i.e. reject this PEP, modify packaging).

We can completely and safely redesign the tag scheme at any major release between Python 4 and Python 31, which sounds like it should give us plenty of time.

If the point of the PEP is advertising, then we ought to have one to advertise any other changes. I’m not volunteering to write it, so unless someone steps up, assume there won’t be one and decide on this accordingly.

1 Like

I personally only care about .whl files, so I have no comment on other file names (e.g. I really don’t care about .pyc files since those are an implementation detail and I personally never look in my __pycache__ directory).

A decision is necessary since the fact that pypa/packaing chose to implement this idea before python did causes a situation where projects cannot package wheels for the 3.10 alpha. The clock is ticking. If pypa/packaging needs to revert the change, it will take time to percolate out to new releases of pip and wheel. Is there still any information missing to make a decision?

Pip has their next release in January and I suspect we could ask @pf_moore and @pradyunsg and others to hold off on a pip release if it requires updating ‘packaging’. So at least in that specific case I don’t think there’s an issue getting the change out.

FWIW, I also prefer the 3_10 over 310, for aesthetic reasons.

TBH, I’m very much in the camp of “it doesn’t really matter that much, so someone pick one and we’ll be done with it”. Sooner the better because it’s nicer to have support in “already in the wild” versions, from a rollout/change management perspective (eg: pip had support for PEP 572 before PyPI such that when it was added to PyPI, most users had support for it already — which is nice to have).

I’m happy to update pip and cut a release, whenever we have a decision on this and make an out-of-band release for this, if we’re doing this in 2020 somehow. It’s still in the air who’s gonna be our release manager for 21.0/later for pip, but hey, we’re all reasonable people and I imagine it’d be the similar for all of us.

Apologies for the delay, but recently it has been quite challenging to find the time needed to review everything thoroughly again. As is clear that everyone’s priority is to move forward, I have reviewed everything again carefully and unfortunately, I am rejecting the PEP as I don’t feel comfortable (by the lack of a holistic view of all the side effects on systems and code we are not aware of) accepting it thinking that is clearly the best path moving forward.

As a release manager, I need to advocate for the stability of the release and reduce possible breakage as much as possible and as we have discussed, changes for the wheel tag also has an impact on other aspects that rely on py_version_nodot. It also seems that there is not a clear consensus among core developers that would suggest that the proposal is a clear winner.

The decision may seem disappointing for some folks but please understand that is driven by trying to reduce the end-user pain and confusion given that we don’t know for sure the unintended effects of altering the pattern to py_version_nodot. Also, the PEP may be re-submitted in the future (and maybe there is more consensus by then or other release manager :wink: ).

Thank you all for your understanding and thanks to the authors for their effort!

5 Likes

Thanks Pablo, and thanks Brett for putting the PEP together.

FYI I closed https://bugs.python.org/issue40747 and my PR.

I was planning/hoping to cut a release of packaging tomorrow, so I will fix this first and then hopefully still have enough time to cut a new release.

3 Likes