Enigma’s Packaging Journey

To better manage package dependencies across ~35 repositories for our data processing pipeline, we ended up writing a tool that checks version constraints as part of CI. Those checks ensure that version constraints are consistent across all repositories. They also restrict the expressivity of Python’s constraints, bringing our dependencies closer to Go’s much simpler package constraints. I wrote up our journey as a three part blog series:

  • The first post describes the technical foundations of package management. This may be rather familiar.
  • The second post describes our tool.
  • The third post discusses social and economic aspects of package management.

My personal take-aways are two-fold:

  1. Writing small, custom validation tools is a huge win for enforcing an organization’s local conventions and needs. However, for that to be practical, the domain being checked must be reasonably simple. That is not the case for Python’s very expressive version constraints and NP-complete version satisfiability.
  2. Go’s design is so much simpler, featuring linear version selection (!), without giving up any critical expressivity. I’d love for Python to follow Go’s lead and start shedding much of the baggage of its current dependency management. It would make for a much more pleasant ecosystem — and much simpler packaging tools too :grimacing:.
2 Likes

Can you provide details of Go’s design? Or at least a link to something that someone with no Go experience could read to understand what you’re referring to? Also, I don’t know what you mean by “linear version selection”, and I’m not at all sure what you believe to be “critical” as regards expressivity.

As far as I am aware, pretty much everything in Python’s current design has been added because it was important (or critical, if you prefer :slightly_smiling_face:) to someone. How would we ensure that when we drop features, people who use those features have a viable migration path? (And “viable” of course includes “cost effective” - we’re requiring them to spend a non-trivial amount of time rewriting their dependency infrastructure, what would they gain from all of that effort?)

Don’t get me wrong, I’d love to see simplifications (extras and direct URLs are the ones that give me the most grief, personally). But we’re not in a position like Go, where we can get away with saying “you do it like this and that’s the end of it” (and honestly, I’m not sure I’d want to work in an ecosystem with that attitude…)

Note that you can use one single constraints file (literally talking
about the -c/–constraint option to pip install) to pin consistent
versions of direct and transitive dependencies across many different
projects with a diverse set of requirements. The OpenStack project
does this for hundreds of their packages, to make sure they’re all
tested with a guaranteed coinstallable set of requirements, while
allowing the individual packages to specify looser version ranges
for their separate requirements lists.

While they could synchronize copies of that constraints list into
all their different Git repositories, the way pip allows you to
specify a constraints file independent of a requirements list makes
it very convenient for keeping a central set of constraints in one
place and referring to that.

2 Likes

Do you have an example of what would that mean? Or would you been just referring to the dependency resolution algorithm complexity?

If I remember correctly go does static linking for dependencies right? I suppose that would be the major difference… (In Python all packages in the same virtual environment have necessarily to share dependencies, so it is much harder to satisfy).

Go has very minimal version constraints, it basically only supports >= and !=.

If you want to break backwards compatibility the official Go approved way, is you rename yourself. This narrows the dependency solver problem domain down significantly. It boils down to attempting to force that you never break compatibility, and two major versions of the same library can be imported and used independently, so you never have to do any real dependency solving.

Of course, there’s no tooling to actually enforce this, so projects semi regularly break it, and when they do it creates awful problems that the Go tooling has no real ability to fix. Other projects work around it, for instance kubernetes, at least at one time, I don’t know if they still do, pretends their X.Y version is 0.X.Y in the package manager to trick it into behaving more like a traditional package manager does.

Having worked in large Go projects, I don’t think it’s better. It’s different for sure. It avoids a lot of the pain points in traditional package ecosystems, it adds its own new pain points.

6 Likes

From the first post:

The apparent disregard in all this for established versioning conventions is striking. Making backwards incompatible changes in a new minor version (pip 20.3) or in a new patch version (Python 3.8.10) is a big no-no under semantic versioning and not helpful.

Neither pip nor CPython use semantic versioning.

pip uses calendar versioning.

CPython predates SemVer by decades and routinely makes breaking changes in minor versions. And the bundled pip isn’t part of the CPython API, rather there as a bootstrapping convenience (see PEP 453).

6 Likes

I agree that technically both projects use semantics for version numbers that are not semantic versioning. I also agree that officially the bundled pip is not part of CPython’s API. I wouldn’t want that if I was maintaining CPython. But the emergent behavior doesn’t seem helpful at all. It seems unreasonable to ask people to track which package (or other software component) uses which versioning convention (if any) and is a missed opportunity for having clear conventions. To put this differently, in the packaging world, less flexibility and more well-defined, shared semantics seems preferable.

@dstufft, thank you for the succinct summary of Go’s design.

If anyone is interested and willing to invest the time, Russ Cox’s blog posts on the design are very well-written and explain motivation and design in (gory) detail. Admittedly, the first time I encountered the posts I skimmed them and only absorbed a third or so of the technical points. But a few months later when I was looking for an alternative to the NP-complete default, I remembered and read them more carefully.

I do want to offer a somewhat different take on the “rename yourself” rule for backwards-incompatible changes: I strongly believe that the onus for making backwards-incompatible changes should be on the package maintainer more than on the users. After all, making the consumers do most of the work means that many more developers have to deal with the fallout from such changes. So the renaming rule doesn’t strike me as onerous. It also communicates a backwards-incompatible change really nicely.

I much agree that not checking the rules is a mistake. pip finally enforcing them was the starting point for our packaging journey after all. Besides that issue, what other pain points did you encounter?

In my opinion, semantic versioning imports pushes the bulk of the work onto the consumers, not onto the authors, and in a much more pathological way.

Regardless of what your strategy is, if you break compatibility going from N to N+1, consumers who want to upgrade from N to N+1, who are using the parts of your package that have broken compatibility will need to adapt. That’s a fundamental issue with breaking compatibility and no dep solving or import strategy is really going to solve it.

Changing the import name adds additional work on consumers, in this case even if they aren’t using the thing that broke backwards compatibility. Now if you don’t fall into the fundamental issue, this change is not particularly burdensome, it would just be changing the import name, but it’s something that every single user of your library would need to do to stay on the latest version.

This ends up being further exacerbated by the fact that the tooling doesn’t do anything to tell you that there is a new version (and in fact, I’m not sure that it can in every case, or maybe even most cases), so you’re likely going to be sitting on and old version, possibly that is no longer receiving updates, with no real warning about it.

I find the idea that the renaming rule isn’t too onerous something that doesn’t seem to be true in practice, otherwise you wouldn’t have some of the largest go projects purposely abusing the system to try and avoid having to do just that. I mentioned before that Kubernetes effectively lies about what version it is, it marks version 1.X as 0.1.X to work around the “2.x” problem with import versioning.

One of the purported goals of this system, is that just become one of your dependencies upgraded to a 2.x, that doesn’t mean all of them have to. Different major versions have different import names, and so they can live side by side, and this, according to the Go team, side steps the “diamond import” problem.

The problem is, it doesn’t actually sidestep the problem, it just changes it, and in the go-style version of the diamond import problem, it makes it far more likely to occur.

The problem boils down to anytime you have types from a library in your own library’s public signature. Everyone using those types has to agree on the same version, because the same type from two different versions is not equal to the same thing. This means that if you have any sort of inter-related dependencies, they all are forced to be on the same version anyways, and thanks to the semantic import versioning, now they can only be on a single major version, even if that major version didn’t change those type signatures at all. Now you have to do a large-scale migration across the entire ecosystem.

For example, I think it took one large monorepo that I’m aware of over a year to cope with the protobufs migration in Go, and most of that had nothing to do with the actual code changes in the migration, it was just trying to coordinate everyone upgrading at the same time.

I also think the other part of this equation, Go’s MVSS makes Go programs inherently less secure. I understand the idea behind it, but the practical outcome of that, is one of two things:

  • Libraries are going to set their lower bound to the most minimal version they can, in which case projects will not pick up new versions by default, and things will just “work”, but will often be hilariously out of date with tons of open vulns that are fixed in later versions.
  • Libraries are going to aggressive set the lower bounds to newer versions, in which case you’ve just reinvented the normal, not MVSS, except you’re introducing extra churn in the ecosystem by forcing libraries to put out additional releases just to bump their lower bounds.

Rus Cox’s argument about why this is fine, is that the people building the application can still upgrade to the latest version, and there’s a single command to do so, and that other ecosystems have lockfiles which accomplish the same thing.

I don’t find this argument particularly compelling. For one, defaults matter. The MVSS strategy ends up forcing a lockfile on people with many of them not realizing it, and not setting up the infrastructure to ensure that they’re maintaining that lockfile. These languages almost universally explicitly tell you not to use their lockfiles for anything except end user applications, but MVSS is almost like having a lockfile for every single library and application.

The final thing that I dislike about it, is that it assumes that there is a single definition of what backwards compatibility is, and what a breaking change is. There certainly are common themes about what definitely is a breaking change, but there’s a whole wide area of grey area that nobody really agrees on. Some projects will drop support for older versions of Go without bumping their major version. Is that a backwards compatible change? Some argue yes, some argue no.

This idea that it’s impossible to know for sure what is or isn’t a breaking change has led some people and projects to eschew the concept of SemVer altogether. I think that SemVer is still useful in a way to roughly communicate the projects intended level of breakage in a change, but that no change, no matter how small, can truly be assumed nonbreaking until you’ve tested it.

It flies entirely in the face of Hyrum’s Law and pretends that developers are perfectly able to decide if a change is breaking or not in every situation, and because of that removes the tools from the ecosystem for coping with when the developer gets it wrong. My relatively long experience in SWE leads me to believe that Hyrum’s Law is correct, and vgo’s assumptions are not.

Ultimately, there are problems with both systems, in almost diametrically opposed ways, so they end up taking different tradeoffs. I personally think the tradeoffs made by Go here are the worse set, but it’s perfectly reasonable to go the other way.

6 Likes