There hasn’t (so far) been a standard for naming and comparing optimized binaries; most systems just call things by the ISA family (e.g.
ppc64le), without much more information.
We’ve pulled a library out of Spack that could be used to label optimized wheels. We’re calling it archspec:
The library does several things that will be of interest to packagers:
- Detects the microarchitecture of your machine (i.e., not just
- Compares microarchitectures for compatibility. You can say things like
skylake > haswellto test whether a
skylakemachine can run
haswell-optimized binaries (you’ll get
- Query the features available on a particular microarchitecture. You can ask, e.g.
'sse3' in haswellor
'neon' in thunderx2
- Ask what compiler flags to use to get different compilers to output binaries for a specific target. e.g., you can ask what flags to use on
gcc, at a particular version, to get a binary for
The library also defines a set of canonical and hopefully familiar names for microarchitectures. The list currently looks like this (from spack output):
$ spack arch --known-targets Generic architectures (families) aarch64 arm ppc ppc64 ppc64le ppcle sparc sparc64 x86 x86_64 GenuineIntel - x86 i686 pentium2 pentium3 pentium4 prescott GenuineIntel - x86_64 nocona nehalem sandybridge haswell skylake skylake_avx512 cascadelake core2 westmere ivybridge broadwell mic_knl cannonlake icelake AuthenticAMD - x86_64 k10 bulldozer zen piledriver zen2 steamroller excavator IBM - ppc64 power7 power8 power9 IBM - ppc64le power8le power9le Cavium - aarch64 thunderx2 Fujitsu - aarch64 a64fx
We use this library in Spack to ensure:
- that every binary package is built for a specific target
- that we can use a particular binary package on a given host machine.
The detection logic and library bindings are currently for Python, but the domain knowledge (features, arch names, etc.) is all in a generic json file with a schema, which we hope can enable people to easily build other language bindings. Currently the library supports macOS and Linux (Windows help would be great).
We’d love to get more contributions to keep the data up to date, and we’re hoping that if this takes off, it’ll enable people to more easily distribute optimized binary packages and containers.
Comments/suggestions/contributions welcome! For more info, see this talk from FOSDEM.