Implementation variants: rehashing and refocusing

[msarahan (Mike Sarahan)]

@ncoghlan

If a project wants to call their variants “foo”, “bar”, and “bob”, then that should be entirely possible, and the question of whether they’re mutually exclusive or not, or whether they mean the same thing in two different projects or not should be expressed in the metadata rather than having to be explicitly encoded in the tools consuming that metadata. To the machinery, they should just be opaque strings, the same way extras are.

I’m not sure I agree here. One of the main advantages to having mutual exclusivity and such encoded in some central location (like a solver “plugin” or “helper” for a variant is that you don’t need to spread that logic around in every package. The “run_exports” feature in conda-build, where runtime dependencies get added automatically based on build-time usage, is a relevant, if simplistic, example. The problem in Conda-land was that it was really hard to know what correct compatibility bounds should be. Every package had to guess or cargo-cult sane constraints for their runtime libraries. “run_exports” was created to give the power and responsibility of defining sane compatibility bounds to the shared library package author, and allow the consumers to just trust that the authors understand the package compatibility bounds.

I may be misunderstanding what you mean by “machinery” here. Where do you draw the line regarding “machinery” and whatever is interpreting the metadata?

@ncoghlan

But A should also have a way to declare that “A variant 1” depends specifically on “C variant 5” while “A variant 2” depends specifically on “C variant 6” (and in some cases, the names of those variants will match, either because one of those projects copied the other, or they’re both following recommendations from another source, like a selector library)

This delocalization of variant meaning makes my head hurt. I’ve been thinking that there would be common key/value kinds of things, and that we’d have something like

given variant “cuda=12”, obtain package A and B where they both have that variant, if they have the “cuda” key at all.

I do see the value in having the package/function from Oscar’s idea, and I see that with that design, different packages/functions could respond differently to a particular key/value combination, but I still think the key/value unification is important.

This also gets back to the idea of putting a hash into filenames as the filename differentiator, since key/value combinations would not be especially workable due to length and syntax complications. Here’s conda’s docs on the topic: Build variants — conda-build 0.0.0.dev0+placeholder documentation

I should note that my hope was that such a scheme could eventually replace aggregate approximations, such as manylinux, and instead encode all compatibility information in a hash_input.json file, then captured in the filename as a hash. This does not preclude putting some human-readable variant identifiers in the filename, it just avoids relying on those identifiers for package/variant resolution.

[pf_moore (Paul Moore)]

@ncoghlan

Wheels that share the same variant name (including anonymous default variants) would then be required to declare the same dependencies rather than leaving that as a common, but technically incorrect, client assumption.

You’re misunderstanding how clients currently work.

There is no assumption that all wheels have the same dependencies. What there is, is an expectation^[1] that an installer can legitimately choose any wheel that is compatible. Dependencies are only checked for the selected wheel, not because we assume they are the same, but quite the opposite - because we don’t care whether they are the same. Dependencies are just a way for the wheel to implement the required behaviour.

---

1. not an assumption, as it’s fundamental to what a wheel is ↩︎