Dynamic versions in editable installations

I’m increasingly seeing the following idiom for single-sourcing versions:

from importlib.metadata import version
__version__ = version(__package__)
Edit: Click for original idiom

The following is what I often see, but the specific use of __import__ is not what I intended to discuss. Put an edited version above.

__version__ = __import__('importlib.metadata').metadata.version(__package__)

This makes total sense to me for installed packages, but I was curious about the interaction between dynamic versioning (e.g., based on git describe) and editable installations.

Right now, if I manage my version with versioneer, then I can use an editable installation and the package can self-report the version based on the current state of the repository. However, if I look at the importlib.metadata-viewable version, that’s frozen at the time that I ran pip install -e .. I don’t think other VCS-based version renderers have resolved this discrepancy, though they may make different choices about which to expose. For example, build-only renderer intended to be specified in pyproject.toml and not used as a runtime dependency or inject any dynamic code in the package would be frozen from the time of installation.

With PEP 660 coming out, I looked to see if there was any discussion of dynamic versioning, and was unable to find it, though this might be a failure of my searches. It does not seem to be explicitly provided for, in any event.

I’m curious if dynamic version metadata for editable installations has been considered, rejected, tabled, or something else.

Thanks!

Hi Chris, even with PEP 660, the metadata (including the version) will still be a static snapshot on the instant you do the editable installation.

PEP 660 specifies that the “editable” part just regards pure Python modules. Extensions, metadata (e.g. version, dependencies), data files, etc, are purposefully left out. The user is expected to “re-do” the editable installation for these changes to take effect.

1 Like

I don’t think that PEP-660 doesn’t address it, but leaves it as an implementation detail for the build backend. The backend could add an interpreter startup hook (pth file e.g.) that would update the metadata files, not requiring manual reinstallation by the user (just an interpreter restart, or not even that if the backend uses an import hook).

1 Like

I agree with @abravalheri although yes, the backend could do something like @bernatgabor suggests. Personally, I think it’s a really bad idea for metadata to vary dynamically, and I’d strongly discourage backends from trying to do anything like this.

3 Likes

I agree some metadata (such as project dependencies) should definitely not be updated by the backend, however, others such as c-extension files (may) could be automatically updated in the background. The problem with updating the version dynamically is that now your environment might suddenly become invalid, as a change in the version might mean the installer could have not resolved dependencies.

1 Like

Thanks all. I appreciate your perspective. Happy to move forward with the clear notion that dynamic version metadata is discouraged and I should not select my versioning tools with an eye to which might eventually support it.

Also there’s a problem definiting what “dynamic metadata” actually means. If it means the metadata can be updated automatically between an interpreter run to another without the user re-installing the package (i.e. in the sense that the Python modules are “dynamic” in an editable install), that makes sense. But dynamic in the sense that metadata can change and expect the changes to be picked up during one same interpreter run, that sounds like a terrible idea.

2 Likes

Yes, I meant the first of those options.

Sidenote: Is there a reason the __import__ builtin is used, instead of just a regular import statement or importlib.load_module, which __import__ is discouraged in favor of for non-highly-specialized use cases?

I think it’s because this keeps it a one-liner, otherwise, you’ll have a line for the import and one for the assignment.

I don’t know where you’ve seen this idiom but I’m sincerely hoping it does not take hold.

1 Like

Yes, I believe because it’s because it keeps it a one liner and does not put importlib in the package namespace.

Is there a (sensible) reason not to want importlib in the package
namespace?

I don’t know that there is. The only thing I can think of is that I’ve seen people import things like logging from the root of a package they use heavily instead of directly through the stdlib. The correct response would be to define __all__ but maybe doubling lines and adding new metadata feels uglier to some than one ugly line that can be skipped over once understood.

Apologies for introducing the idiom if that’s annoying people. The question was more about single sourcing than the specific character strings used to accomplish it.

I personally don’t like seeing packages and libraries do this. It adds extra overhead of an otherwise unnecessary import + call done at startup time in every process using the code that does complicated things to lookup a constant from an external location (elsewhere on the filesystem, etc). Why make all users pay that cost for no good reason?

If you have the concept of a version and want that to be available programmatically at runtime to your users, don’t put it in a __private__ dunder named attribute and populate the constant in code once, at installation time. Also document what guarantees your library makes about what type and format the value will always be, when and why it should be used, and what it means in your API docs.

.__version__ attributes are IMNSHO generally pointless.

4 Likes

Agreed. It’s straightforward for users of a package to use importlib.metadata.version, so what’s the point in the package calling it for them, “just in case”. Most users will never reference it, so it’s wasted effort.

Unfortunately, I think __version__ is a fairly ingrained habit for a lot of people. But it’s a holdover from the days before importlib.metadata.

If you’re going to insist on supplying a __version__ attribute, make it a constant string and don’t play costly games just to avoid typing it twice.

9 Likes

Some more notes for anyone wanting to provide a __version__ for backwards compatibility or any other reason.

  • If using setup.cfg, you can use version = attr: my_package.VERSION.
  • flit looks for a __version__ static attribute in the package entry point.
  • You can use a tool like tbump to keep version strings in sync across multiple files, including sphinx conf.py files.
1 Like

I’m not sure that they’re doing it for the users’ benefit - a lot of the time it’s CLI apps wanting to single-source their version to e.g. print in --version or to check for updates. I find that it’s increasingly uncommon that packages will have a __version__ unless they have some use for it.

1 Like

Fair point, but it’s equally straightforward for the package code to call importlib.metadata.version when needed as well - so why pre-calculate it? I’m still not seeing the benefit here (other than “because we’ve always done it that way”). With one proviso - if a project supports Python 3.7 or older I can see having a __version__ attribute to avoid a runtime dependency on the importlib.metadata backport. But that still doesn’t (IMO) justify complicated machinery simply to single-source the version.

But this is both off-topic and a matter of opinion at this point. The above is just my personal opinion.

If there is any plausible scenario where importlib.metadata.version(spam) could return a result other than a statically-specified spam.__version__ presuming both are run consecutively from the same instance of the same Python interpreter, and version = spam.__version__ is set in setup.cfg, then it would seem to be of value to retain it (so long as it is a static value, i.e. not just retrieving it from importlib). In particular, this would apply at least for editable installs and (more esoteric, but not totally implausible) if the package’s files were modified after install.

In particular, I’m most concerned when dealing with the interaction between package managers (OS and Python, pip and conda, etc), as I’ve in helping other users and testing myself, certainly confirmed multiple sets of circumstances (various combinations of uninstalling {pip/conda}-installed packages with {conda/pip}, path configuration issues, conda bugs, other user issues) where neither pip list nor conda list print the version of the package that’s actually imported (and spam.__version__ would correctly list).

I haven’t yet verified in which cases importlib.metadata.version(spam) inside Python doesn’t report the actual imported version (its certainly plausible that it would in some cases, but not in others, depending on the specific problem), and these cases naturally involve environments that are inconsistent, but this is my primary use case for a static __version__, as a source of truth on the version of the package that is actually getting imported, when troubleshooting these cases.

Also, as a sidenote, it seems __package__ is slated to be deprecated, at least per the discussion in python/cpython#77458 (BPO-33277) and python/cpython#89703 (BPO-445540), which would make more less-trivial to do without hardcoding the package name?

Err, don’t you mean setup.cfg?

1 Like