Python version caps don’t work that way, because the resolver will simply pick 0.1.4 over 0.1.4post1 on Python 3.12. There’s lots of discussion on why version caps are bad - you should be able to find discussions reasonably easily.
I did a quick check on PyPI data - of the released versions of all projects on PyPI, about 1% are “post” versions. And 6% are “dev” versions[1]. So removing those version segments would have a significant impact.
Yes, that seems high. From what I can see, releasing pre-alpha releases with .dev version numbers is not uncommon. ↩︎
I am aware of these — the maintainer would currently also have to yank every uncapped release for the post-release to make sense. My question was intended to clarify whether post-releases are an “acceptable“ thing or a “good” thing/best practice.
The main use case I’ve seen for “post” versions is when you need to
upload a rebuild that doesn’t involve changes to the project’s
source code (and so can’t logically be a separate real release). For
example, you want to link some C extension against newer library
headers, or rebuild with a newer build backend dependency in order
to fix some bug in the packaging of the associated release. I’m
dubious about the use case for adjusting metadata since I always
consider that to be fundamentally part of the source code still, but
I suppose not everyone does.
For downstream repackagers, the “post” release ought to be
fundamentally identical to the original release since they’re going
to be rebuilding it all again anyway, it’s more for the benefit of
people installing packages directly from PyPI. But of course,
whether that’s how everyone actually uses “post” versions is the
real question.
Both .post and .dev are supposed to be semantically newer than their referenced final release,
.postN is semantically newer than final, but .devN is oldest of all:
>>> from packaging.version import Version
>>> sorted(["1.0", "1.0a", "1.0.dev", "1.0b", "1.0rc1", "1.0.post2"], key=Version)
['1.0.dev', '1.0a', '1.0b', '1.0rc1', '1.0', '1.0.post2']
Could you make the following mapping?
PEP440 suffix
FreeBSD suffix
.devN
.dN
aN
aN
bN
bN
rcN
rN
.postN
pN
By the rules posted above, I think you would achieve the same ordering.
If you have build numbers, another approach could be to use the following mapping:
PyPI version
Distro version
x.y.z
x.y.z build 1
x.y.z.postN
x.y.z build N+1
If you need space to have multiple builds of the same release and don’t want to keep a table of paired versions/build numbers, you could use (N + 1) * 100 + sub-build.
For us distros, we update to every newest released version (subject to manpower availability) as we are alerted. Even if a post-release may be fundamentally identical to the referenced final release, as long as a release newer than the final exists, we update. Distro maintainers and our userbases by and large do not care about the nuances we are discussing here, it’s just “new version available, please update”.
Additionally, in other non-Python projects that employ similar post-release version notation like sudo, such do include bug fixes in the code.
When specifying as the “distribution version”, the PEP 440 suffixes aN, bN and rcN are already supported and mapped to our .aN, .bN and .rN respectively. In fact, the existing logic for “distribution version” mapping treats any trailing letter sequence as a pre-release and converts to [DOT][first letter]. As these three are pre-releases, they need to have the [DOT] on our end to be semantically older than a final release. I apparently missed the part where .devN is semantically older, so in this case the existing logic already fits.
However, problems still arise with .postN and the .postN.devM combo, which cannot be specified as a “distribution version”. They must instead be specified as “port version” which is not mapped/converted, but already follow our conventions. In this case, the challenge is mapping “port version” to the Python version specifier to reference the correct sdist and generated bdist names. In my prototype implementation, this involves blanket overriding the “distribution name” (here, the sdist filename) variable. I especially don’t like this part, since “distribution name” is a generic variable for individual ports’ exclusive use, and is needed for some individual Python cases where mixed-case package name propagates to the sdist and generated bdist despite being semantically identical.
I’m curious, what is your solution in this case? (I’m not familiar with sudo’s versioning, living the charmed life of a user who mostly trusts his platforms’ package managers to update that tool appropriately and correctly ).
I don’t see a realistic solution here if you don’t have some part of the package version which is reserved for the redistributor.
For example, let’s set aside the proposals to add a mapping scheme and instead consider how you handle these issues if you add a specifier for the redistributor’s version, separated by a hyphen:
1.2.4 # no distro version, default to 0
1.2.4-0 # semantically identical to the above
1.2.4-1 # successor to the above
Then every post release can bump the number. If you want to ship a dev package (not sure that’s wise but ), you can start at -0 and simply increment with each release.
Python code which knows it’s own version still behaves appropriately, not knowing about the “distro version” field.
Key to all of the above is that it’s an additive change which supports all current use-cases. We don’t have to try to remove something which people are already using (and yes, people use post versions, albeit rarely – isort comes to mind as a place where I saw one recently, maybe last year).
I would also note that having distro-package owned version into gracefully handles cases like a bad build/repackaging, simply by bumping the distro number.
I just spoke to the package manager maintainer, and this will not be considered for support. Responses from the rest of the community range from “why should we care” to “this is dead stupid”. In light of this, I am considering enacting a policy in our distro prohibiting the distribution of .post releases.
We have such a revision parameter, but is reserved for distro use only. There are other cases that warrant bumping that number, mostly rebuilds after a dependency breaks API. The crux remains that .postN are upstream releases. The metadata needs to align end-to-end since it is used directly for fetching/extracting the sdist before build and extracting the generated bdist for staging/packaging. (Ab)using the revision parameter at best results in only ever using the original final release, and at worst unable to bump revisions for forced rebuilds else fetching the sdist fails.
sudo is an isolated case where the upstream version is specified verbatim into the “port version” variable, which does not perform any conversion/mapping. Their version specifiers (currently 1.9.15p2) follow our semantic rules so no further processing is necessary.
Not sure if there will be a solution. The general prototype implementation I have is very hacky, ugly and introduces other contentious trade-offs wrt the rest of our framework (outside the package manager), and based on community responses, am considering prohibiting the distribution of .postN releases entirely.
I just want to note that you’re trying to bridge two communities – and I appreciate that you’re getting feedback from both sides which is less than ideal for you – but that these aren’t things we can meaningfully engage with in this forum.
Python packages have their own versioning scheme. It isn’t a strict subset of your distro’s versioning scheme. We can pitch ideas for how to resolve this, but most, maybe all, will require changes on the distro end. I don’t see an easy resolution. Even if we all agreed to deprecate post-versions today, and go through the pain of getting rid of them, I don’t see any reasonable way to expect that Python packages versions will remain a strict subset of your valid distro versions. What if we have some new, well-justified idea for expanding our version spec in 30 years (a long time in software)?
So will python give you the promise you really want? No. What do we do now? That’s the only productive path to pursue.
Since you say that this is a problem for “distros” in general, do you have actual data on the extent to which this is a problem in distros other than FreeBSD? Petr mentioned that RPM handles it fine. What about APT, Pacman, YAST, Homebrew, MacPorts, …?
We have such a revision parameter, but is reserved for distro use
only. There are other cases that warrant bumping that number,
mostly rebuilds after a dependency breaks API. The crux remains
that .postN are upstream releases. The metadata needs to align
end-to-end since it is used directly for fetching/extracting the
sdist before build and extracting the generated bdist for
staging/packaging. (Ab)using the revision parameter at best
results in only ever using the original final release, and at
worst unable to bump revisions for forced rebuilds else fetching
the sdist fails.
This is the crux of your dilemma, I think. The Python community is
hosting “upstream releases” on PyPI but also treats PyPI as “a
distribution” of consumable/installable packages, and so like your
distribution, needs a way to indicate that some versions are
basically the upstream release but slightly later, though not a full
new release. By consuming packages from PyPI in order to create your
own distribution’s packages, you’re essentially repackaging one
distribution into another. Cue references to Gödel’s incompleteness
theorems.
Arch/pacman does a manual override. This is exactly something completely undesirable for something systemic as promulgated in the version specifier standard.
The sources are not always fetched from PyPI. As long as the metadata (pyproject.toml mostly) specifies a package name and version, or something like setuptools_scm constructs a version dynamically, we have to match it, end of story. If the version field, specified or constructed, contains a .postN, that’s the version.
Gentoo/portage also does a manual override. Though, I’m not sure how far they go wrt end-to-end metadata alignment. I know Arch/pacman just wildcards the bdist file passed into installer so no metadata alignment at that end.
I am struggling in this thread with a lack of knowledge and context. I suspect others are as well.
You say you have to match the version number exactly here, as a logical conclusion. For me, without your familiarity with FreeBSD packaging, that feels like a big logical jump.
I don’t think I followed why sudo’s versions can be coerced correctly into package versions, but Python post releases cannot be so-coerced in the same way. You said something about both systems following the same semantic rules. Can you explain what you meant a little more?
There is a version number already reserved for the distro, but you’ve said it’s off-limits to update it with a post release. Can you explain why at all? It sounds to me like a perfect solution. Sure, it may mix updates to the source with updates to the package build into a single stream, but so what? It doesn’t need to be perfect on that account. What is the core problem with using this field?
There may be no solution here if FreeBSD is unwilling to budge, but I find that a little unlikely. There’s probably no solution without compromising on your goals or vision for this integration a bit though.
(And yes, I put it on FreeBSD being unwilling here. That I don’t think the Python community should remove features which have been in active use for ~10 years is more or less a given. I use dev releases and I’m not the only one.)
).
), you can start at