I’m trying to publish a script that will be used as an addon for Blender, which requires me to locate wheels for all the packages that I am importing, and for every platform my script could potentially run on. So while pip wheel allosaurus -w ./wheels downloads all the wheels I need for my dev machine (which is Windows), I also need to figure out how to download the wheels for macosx and manylinux too.
The wheels I need to download also have dependencies, so I need to download those too. And the dependencies are a mix of binary and source wheels (allosaurus relies on scipy and torch, but also a lot of source only wheels). So a command like pip download allosaurus --dest ./wheels-mac --only-binary=:all: --python-version=3.11 --platform=macosx_11_0_arm64 does not work since it only downloads a very out of date version of allosaurus.
I’d like to write a script that a user can execute that will download all the necessary wheels and put them in a directory. I’d like to give it just the list of the top level wheels that I need and have it recursively find both source and binary wheels as necessary and put everything in that directory so that the user can then bundle them with my script.
Is there a way to do this? The only thing I’ve found so far that kind of works is running pip wheel allosaurus -w ./wheels separately on Windows, Mac and Linux machines and then copying their results to my final folder. This has obvious problems for anyone who does not have one or more of those machines and who wants to package my script. Even if I could just generate a list of all the wheels I need to get, that would be a big help.
You are looking for two things. One is cross platform resolver that determines which wheels to get per platform. The other is way to download/fetch then. Pip supports second part but not cross platform. Uv supports cross platform resolution but I don’t think has second part. So I’m unaware of an existing tool that solves this. There exists a uv issue open for second part and similarly I think pip has an issue open for the first part. Either helping with that issue or using uv to get dependency list and then using separate tool/script to get wheels could work. You might even be able to use uv to get dependency list and then use pip to get wheels as I think pip may have a flag to override wheel selection (or you could mock wheel/abi related tags you get).
Easy, you already have it, it is pip wheel allosaurus, or am I missing something?
I am not sure I understand what you are trying to achieve. Feels like this paragraph contradicts the previous paragraph. Should the user download the wheels, or should it be done by you (or the packagers)?
I do not know about the full use case. I do not know how Blender addons usually deal with these things, but you are probably not the first one trying to solve this.
The reason why that doesn’t work is because SciPy doesn’t support macosx_11_0_arm64 due to some pretty nasty kernel bug in macOS. If you use --platform=macosx_12_0_arm64 then it does what I think you want.
Also @sinoroc this specific issue is discussed in pip issues that have been open for years. The answer is I’m unaware of any working solution for this today. The underlying goal is fetch all wheels for library across different platforms. pip download/wheel X does it for current platform/environment. Requesting wheels for different platform is what’s missing for pip and I think requires custom tool or wait for that pip/uv issue to be resolved.
edit: If you have 0 environment markers affecting resolution and are lucky then pip download/wheel + those flags can work. Only way I know to check that though is run resolution in each platform, confirm they pick same dependencies, and hope that stays true.
I’ve been looking through the uv docs and watching a few videos, and can’t find anything about listing dependencies. uv actually looks like a virtual python environment. What I need to do is just find all the crossplatforms wheels I need - do you know if there is a uv command line which will do this?
Resolution | uv is a way to find dependencies across different platforms. That won’t give you wheels, but you can build script on top of that to find wheels or use pip wheel/download with --no-dependencies on top of that.
@blackears I am really not sure that what you are asking for makes sense. Maybe it does, but you have not managed to convey a request that does make sense at least to me. If I try to piece things together from your question in something that does make sense, then my answer is that it is not feasible at least not without a whole bunch of coding from your side.
One rough draft of an idea that might work, is that you somehow get a tool like PDM, Poetry, or uv to generate a “lock file” for the dependencies of your code. In this lock file, there should be a whole bunch of URLs to wheel (and sdist?) files. You could write a script that reads the lock file and downloads all of those wheels and sdists. You could try to make your script smart enough to download those files in as many buckets as necessary: one bucket for Windows 64 bits CPython 3.12, one bucket for Linux 32 bits PyPy 3.8, and so on… Maybe if you are lucky you won’t need that many buckets, because all the dependencies are pure Python and compatible with all versions of Python from 3.8 to 3.12 or whatever, but otherwise you will need indeed a bunch of buckets. I am afraid that things are not as simple as just Mac or Linux or Windows.