For VCS-backed versioning in particular, I have wondered in the past whether it would not be easier to turn that into static metadata when the sdist is prepared. AFAIK, no tool currently does this, but is there a fundamental reason?
PS: Of course, PKG-INFO already exists at that point, I wasn’t thinking clearly (or rather, I was thinking of pyproject.toml-using tools with which I ran into problems). Ignore me.
Not a fully formed thought.
Would it be possible to define some sentinel values for versions in VCS and for editables? Then to build special treatment for these values across the ecosystem. Would this help remove some dynamic metadata?
Sentinel values
Behaviour in the ecosystem
It would push the need to access SCM for version info to publishing tools like twine or some other tooling.
I think these two are actually fine to cache and hash if we get consensus on hashing a directory within the packaging ecosystem, the directory hash changing should be what surfaces “okay, you need to recalculate the dynamic metadata if you are a tool that cares about this” (and make checking this a function call, not an always checked thing so that only tools that need to know this pay for it)
The overall idea of a special value here, especially within importlib.metadata seems reasonable to convey to tools that expect static metadata the known cases that result in not having it.
I’m just more generally confused on where the offending “dynamic” version metadata is appearing. All the projects I’m familiar with which have automated versioning based on version control system state freeze that information at package build time, so the actual metadata inside the resulting sdist (PKG-INFO file) or wheel (METADATA file) is entirely static. It’s only “dynamic” from the perspective of the pyproject.toml file used by the build backend.
Is the concern purely about editable installs from a local file tree, and not actually about packages at all?
Yes. I think so. I think what we see here is the age old App vs Library dichotomy.
I think what happens here, is that library maintainers want to “lock” their dependencies for the benefit of their CI and co-maintainers, but without their project in the “lock” and without having to duplicate version info. So in these situations, from the perspective of the project maintainer, if they’ve got versioning based on SCM, they theoretically always have “dynamic” versions because it will change every commit. In my very loosely framed idea above, perhaps they could pin to an EDITABLE version, which would float and be supported by lockers and installers, but with appropriate warnings.
And then for the typical “application” developer case (e.g. django app, docker container etc), who is generally not publishing distributions to PyPI, but pulling in most of their dependencies from an index like PyPI, they don’t really seem to or need to lock their application code. What I see happening in these scenarios is generally that these folks lock their dependencies, install them into a venv and work on their applications.
That’s my understanding where the differences creep in at least. If my description isn’t clear enough, I think the treatment in the Poetry docs on the 2 operating modes might also help clarify: Basic usage | Documentation | Poetry - Python dependency management and packaging made easy
a) that “dichotomy” is a Python invented problem that does not need to exist and b) it’s unrelated to the problem here. Editable installs are a reality and not just for applications but also libraries.
The problem of dynamic metadata is also not unique to editable installs, the same problem exists with sdists as I have already explained a few times in this thread.
Yes, I think we align. I think in my mind, when I read “dynamic metadata” I actually only care about the problematic ones like “version” and “dependencies”. Which I think causes the most headaches from being dynamic in sdists. But you are right, there is a lot of dynamic metadata beyond “version” and “dependencies”.
Question to you then: if “version” and “dependencies” could no longer be dynamic do you think that would be enough? Or do you think everything needs to be static?
I tried that; it’s called PEP 665 and it was rejected due to community pushback thanks to the lack of sdist support (hence why PEP 751 now exists and tries to be more comprehensive in terms of the support from where could might come from).
I think the version part of this discussion is around source trees (which I just consider editable installs a form of), although as Armin pointed out a package can cheat if it’s an sdist as the version in PKG-INFO could be ignored by the build back-end when it writes out the wheel (but I would say that’s bad and it would be reasonable to make the packaging specs require metadata found in a PKG-INFO file must match what ends up in a wheel file if it doesn’t already). I think the key thing is tools use the version number for things like resolving, caching built wheels, etc., so not being able to get them cheaply and easily can be problematic when you rely on them for these sorts of thing.
But I think Armin is trying to talk about this in a broader context than just dynamic versions, although that the most common and broadly understood use of dynamic metadata.
Do you include version = attr: M2Crypto.__version__ into this category of dynamic metadata, or are we talking only about sources of information external to the project?
When your run something with a workflow manager (e.g. poetry run xxx or the uv/hatch equivalents, possibly even an IDE), it runs everything in an automatically managed virtual environment with your package editable-installed into it. On each invocation, it is expected to is detect changes to dependencies/package metadata that may require reconstruction of that virtual environment.
Regarding version = attr: M2Crypto.__version__, yes it absolutely is this kind of dynamic metadata. How is a workflow manager supposed to know which project file contains the __version__ definition? It’s most likely to be M2Crypto/__init__.py or src/M2Crypto/__init__.py but it could be anything. If a workflow manager doesn’t know which file to look at, it either has to let __version__ and importlib.metadata.version() go out of sync or assume that any change to any file invalidates the project’s current state and rerun something equivalent to pip install -e . every time, making the workflow manager so slow it ceases to be beneficial.
This is why I suggested using a filename+regex instead of an import path – it’s still dynamic but it’s explicit about which file to watch for changes that may change the version. I don’t think we necessarily have to eliminate dynamic metadata – just make it cheap and simple to evaluate and preferably reduce the need to invoke build backend to get to it.
Any mention of sdists are a bit of a red herring here. Sdists theoretically solve the problem for cases where the project can be packaged up once and consumed later. (That pre-evaluated metadata is unusable to pip/build if you unpack the sdist to get at the tests/docs but that’s a separate gripe.) Some local tool (workflow manager, build tool, SBOM generator, …) that also needs this metadata but expects a faster feedback loop than the cost of building an sdist every invocation is still at square one, using what they can access statically, getting some dynamic metadata by short-circuiting the build backends where feasible and writing the rest off as an unsolvable problem.
Editable installs are their own special problem here, because no-one has ever managed to agree on a clear definition of the required behaviour of an editable install. For instance, I was always under the impression that it was obvious that even with an editable install, if the developer makes any change that affects the metadata of the package, they must reinstall the project. And the responsibility for knowing what changes affect the metadata is firmly on the developer, rather than being something the editable mechanism will determine automatically. But the discussion here clearly includes people who feel that this is a problem to be fixed, rather than a fundamental part of the specification of an editable install.
If there’s one thing that I would love to see coming from this discussion, it would be someone taking up the (significant!) challenge of putting together a formal specification of the behaviour of an editable install. It sure as heck won’t be me, though.
Agreed. And that’s what we’ve been working towards for years now. The problem is that unless you totally eliminate dynamic metadata, tools that aspire to be correct have to deal with its existence - and doing so is hard[1]. That’s not to say it’s not worthwhile, just that incremental progress will never be enough for some people.
The problem here is basically setuptools, I’m afraid. Historically, setuptools users could write extensions that did literally anything they wanted with the build process. And while the setuptools maintainers have made immense progress in providing static and standards-based ways of defining project metadata, it’s simply not possible (as I understand it) to remove the customisation hooks, without breaking a huge part of their user base[2]. So people can still do bad things, even if they no longer need to.
If we were developing a packaging system from scratch[3], I’m sure we’d have chosen a static metadata approach. But we don’t have that luxury, and any packaging tool that wants to credibly claim to support “Python packaging” in general can’t dismiss the huge bulk of setuptools-based packages (many of which are likely invisible to us in private commercial development teams) that have barely even heard of pyproject.toml.
And costly - some of the performance difference between uv and pip is because uv makes simplifying assumptions that break in the face of dynamic metadata, whereas pip prefers correctness over performance ↩︎
Many of whom probably could use more modern, static approaches, but they don’t have the resources to rework their build processes, so that’s still “breakage”, like it or not ↩︎
Like Rust, npm, go, and all those other language systems that we look at enviously ↩︎
That’s a good question that helps to dive into the philosophical nature of the question “dynamic” vs. “static” and “editable” vs. “site-packages”.
In my eyes this construct, specifically, is “static” information, because it comes from a part of the package’s source code, which forms an inherent part of the software solution needed at runtime. When the version information comes from a Git repository this is considered “dynamic”, because the information is derived from the environment “hosting the source code” at build time. It’s still integrated as static information into the metadata, which in my mental model is the canonical truth for an installed package together with the unaltered source code.
We could record a cryptographic hash over the entire source code of the package that could be recalculated anytime to verify the validity of the installed package, if integrity is important.
In essence, I don’t fully grasp the fuss about “dynamic” as long as the build tools convert the setup to statically recorded package information. Talking about version number generation – let’s call it “automatic” versioning – I found this to be a tremendous time-saver compared to having to update the __version__ attribute – which I used to be a big fan of – in the package’s main __init__.py file. I use scm-setuptools nowadays. My mental model about the version number has not changed, though. It’s still “static” information for me, only more conveniently updated. And this is only possible, because I bow to having to manage my source code under version control to get this convenience.
The “editable” part of the discussion, in my strictly personal view, is a part of the less clean nature of Python, that leaves more room for freedom, though. (The comparison with Java’s Maven was great! It’s super-rigid.)
People who use editable installs either are package developers who work on various projects / packages / libraries at once and need or want to have live interaction with changing source code. Or they are “dirty hackers” (forgive the term) that like to mess with source code in production without source control and immutable infrastructure in place. The same breed of people who see Python as a “scripting” language. My colleagues at work still call the software we write “Python scripts” even though we’ve transformed everything to automatically built and neatly installed CLI applications.
Yes, Python is “for adults”, and adults oftentimes take stupid decisions. But that’s part of the freedom Python offers. Not sure if we want to change this. Though, packaging tools could be strict, because package should be a clean terrain, a place where you learn good software development practices. Just my two cents.
It is only static if that information is pure. Most of the time that information in itself is however computed (eg: I have seen packages invoke a subprocess to parse the git revision information).
At the end of the day I feel like most of this discussion completely misses the point. The point is to reduce the complexity for everybody working with packages. Instead what is progressively happening here is a massive amount of scope expansion to further justify the existence of dynamic metadata or to try to redraw the lines in different ways.
I want to re-iterate that the problem of dynamic metadata is entirely homegrown in Python. It’s a problem that (at least in this scale) is entirely non existing in a lot of ecosystems.
I won’t be giving up dynamic metadata without an adequate tool to replace it for the uses I have for it. (primarily, allowing users to run development versions and have access to that information without a ton of additional work)
You might ask why other ecosystems don’t end up with dynamic metadata, and in each of them, you’ll find capabilities we don’t have in a standard form that in multiple ways reduce the need of it, or that other consequences prevent it from being needed.
The versions of libraries in use can change in an environment and are expected to be accessible at runtime, and that version can’t be known in advance except with strict pinning that precludes the dynamic version in the first place. Practically every compiled language can ignore the consequences of this and doesn’t have to design around this.
The primary package index doesn’t allow VCS links or patch files, leading to users manually using an editable install to patch things.
There are more reasons besides this, but it’s not a fair comparison to point to other ecosystems that don’t have dynamic metadata without acknowledging differing things they do have which enable the underlying behaviors people want, or that language choices make some of it irrelevant for them
That’s just exchanging one word for another. In reality, “static” is an imprecise term, and we should accept that. In normal packaging discussions, “static” typically means “you do not need to invoke the build backend to determine the value”. I’m not at all sure what you mean by “static” in the context of this discussion, but I’m going to use that definition unless someone provides a different, but clearly stated, alternative.
Maybe it is. But it’s not going to go away unless we find a “static” way of addressing the issues people use it for.
We could - and you’re welcome to, if you want - build a new packaging toolset based around entirely static data. It may catch on. It’s certainly possible to use such a system - other ecosystems like npm and rust demonstrate this. But given the existence of the current ecosystem, would such a system be compelling enough to users to lead a sufficient majority of projects to switch? I doubt it, unless you come up with better answers for the people who find dynamic metadata important than “you shouldn’t do that”. I thought that was the point of this discussion, but maybe not - no-one who’s in favour of static metadata seems to be taking that approach, at least…
Alternatively, and with a lot fewer backward compatibility problems, gradually introduce more and more ways to handle common use cases with purely static metadata. That won’t eliminate dynamic metadata, but it might reduce its use to manageable levels. That’s essentially what most of the last 10 years of packaging ecosystem standardisation has been about. Maybe “reduce” isn’t good enough for you. But equally, “break a huge chunk of the ecosystem with a backward incompatible solution” isn’t good enough for the rest of us…
I wish this discussion was more productive. It would be really useful to get a better view of why people need dynamic metadata, and what their constraints are on changing to adopt solutions that are more static. It would be particularly interesting to hear from the parts of the setuptools user base that don’t generally get involved in packaging discussions, to find out just how much of setuptools’ extreme dynamism they actually need - but that may be too much to hope for.
But as long as we keep claiming that dynamic metadata is “not needed” without working with people who find it useful, to discover a static alternative that meets their needs, I fear this discussion will simply continue going round in circles 
Nope, ~mcepl/m2crypto (master): src/M2Crypto/__init__.py - sourcehut git … the point is only that I don’t want to maintain the same information in more than one place.
Calling it “pure” seems a bit loaded, and might not even have an agreed upon definition. Asking git for the revision is as pure as anything interacting the with filesystem can be, it has a deterministic outcome for a given state that does not have observable changes to the state of the program that would not exist if the call magically knew the answer without asking git for it.
It seems you’ve missed the point here, not those discussing it. You can’t call for brainstorming eliminating dynamic metadata and not have a discussion about how this will impact people’s existing workflows and what we would need in the ecosystem to enable those workflows that ends up as static metadata. The idea that this is scope expansion to you makes it seem like you’re just dismissing people’s existing workflows and why they have them, and the ideas they have that could even get closer to your ideal than where we are now.
I disagree. That dynamic metadata is used, and that it has uses is obvious and discussing it does not yield new information. It is the known status quo. What needs to be figured out is not a complete enumeration of all the things that will break, but to figure out how the common things that would break can be solved instead.
It’s not very hard to find arguments about why something cannot be changed. What is harder (but also more useful) is to figure out how we can transition from where we are, to where we should be.
Can you take a moment to reread what I wrote? The part you quoted is fine for that.
The discussion here has been examining the cases that dynamic metadata doesn’t have an obvious static answer for, and you’ve even had actual suggestions from multiple different people about how to handle it and limit the impact of it come up, yet you’ve dismissed it as scope expansion, and it doesn’t even seem like you’re taking the time to understand the things you’re directly responding to.