tl;dr
I want to generate releases tagged with a githash to ease rapid shared development. Think shared development of editable installs without having to rewrite all the tooling. Surely I’m not the only person who has ever wanted to do this. Are there any valid alternatives while remaining compliant with PEP-440?
I think I can solve my problem by simply ensuring that I only do this in conjunction with one of the supported suffixes (dev, alpha, etc) dev releases knowing full well that I lose the ability to publish releases at that level sans local version that can be reliably retrieved later.
Is this the only/best way to accomplish my goal?
Problem Statement
I’ve encountered the need to internally distribute development releases of libraries in order to ease developer workflows on interim work. For anyone who has come from say the java world, they are likely very familiar with the concept of snapshot releases. For anyone familiar with git, it should also be readily obvious how a single linearly increasing version scheme is problematic when there is a need for independent but concurrent in-progress work.
Ex, the need for a “dev” release from 2 different branches without knowledge of each other will always choose the exact same “next” version when using the usual schemes made available via say bumpversion.
I was initially able to get things working using something like “0.0.0.dev-c2bf818a7a6492524e3e74f78cd770ebe01d9ee0” under poetry, but that violates the semantic versioning terms dictated by PEP-440, so I tried to use local version identifiers.
That solution works when using poetry and “will install” using pip, but pip will also happily ignore an “==0.2.0” to install a “local version”. Note that poetry does not which is likely in violation of PEP-440.
> pip install my-lib==0.2.0
...
Successfully installed my-lib-0.2.0+dev.df6283508e018618914d269432c7ad3fc4aed1c5
Having now read the fine print carefully, I realize that the use of local version identifiers for this purpose does not match the stated purpose of local versions and results in incorrectly installed dependencies.
I did at least discover that pip seems to happily respect specified local versions.
❯ pip install 'my-lib==0.2.0+dev.6d606f3753d83df8fa6a8a080f3c07375f4beab0' | tail -1
Successfully installed my-lib-0.2.0+dev.6d606f3753d83df8fa6a8a080f3c07375f4beab0
❯ pip install 'my-lib==0.2.0+dev.df6283508e018618914d269432c7ad3fc4aed1c5' | tail -1
Successfully installed my-lib-0.2.0+dev.df6283508e018618914d269432c7ad3fc4aed1c5
Is there any way for me to safely create/use githash based releases in a pip-safe manner other than my stated example of choosing an arbitrary set of semantic versions which can be safely discarded?
Misc notes:
- My same approach seems to work fine with say Java/JS(pnpm, yarn, etc). The opinionated restrictions that forced me to try local versions only seems to be present in python as local versions appear to be the only place that I can inject a githash.
- My examples all use long githashes, but if it makes a different we were already thinking about switching to short githashes.
- pip itself does not appear to have any way to ignore local versions (flags on install). Probably for good reason.
- As mentioned above, poetry does not auto-select local versions. While helpful to me, it’s probably in violation of PEP-440.
- I’m not aware of any way to explicitly tag my githash-based local version as being a prerelease version.
- I’ve specifically avoided linear releases due to their problematic nature during concurrent development and the associated tooling headaches. I’ve done this in the past where the solution was to just keep running bumpversion until it succeeds, and it’s not great. I specifically wanted to avoid having to deal with extraneous bumpversion conflicts which is why my githash based approach updates the version in-place and never commits those specific version changes to source control. As it’s githash based, there’s no reason to commit bumpversion’s changes.
- I’ve specifically avoided direct linking to source control due to problematic tool support. Getting local tooling with say Docker as well as all of our CI/CD to work with source-control based dependencies requires a lot of extraneous credentials management that I’d really like to avoid such as injecting ssh keys, setup ssh agents, etc.
- I know this can be solved via a cleaner and more orderly progression of fully tested/specced library changes down the entire transitive dependency tree, but lets face it, we live in the real world and development progress can get messy.
- Circumventing this via editable installs can be an absolute nightmare. Sometimes you just really want to share a new version of a transitive dependency without forcing someone to do an error prone 35 step install using
git checkout
andpip install -e
.