Where you specify either base or win_base, depending on the OS you’re installing it on. I was wondering whether it is legal and acceptable to instead define it as follows:
extras_require={
'base': base_deps + (['pypiwin32'] if win else []),
'full': full_deps + (['pypiwin32', 'xxx'] if win else []),
}
That way, you always just specify base and it gets the OS dependent packages?
The assumption here is that we don’t cross-compile a wheel, if it’s even possible, so that the metadata of the wheel will match the one it is generated on and for so the correct base list would be in the wheel. And if installing from source, it would naturally know the OS it is running on and return the correct list.
The standard/recommended and declarative way (i.e., not dependent on running any code in setup.py) to specify a dependency that is conditional on running under Windows is with the marker sys_platform == "win32". In your case, this would look like:
Also, for anyone looking for the complete accepted version and a formatted table with extra information, the env markers are defined in PEP 508. PEP 496 is the drafted version that was rejected, but the markers were carried over to PEP 508.
Packaging isn’t my main area of knowledge, so I wasn’t aware of the marker standard until @pf_moore’s response prompted a bit of further research.
This was really helpful, but now I have a followup question in the vain of the original:
Similar to the optional runtime dependencies (base and full), we have corresponding optional dependencies when building from source using setup.py. E.g. if you use sdl2 at run-time (kivy_deps.sdl2), you need a dev version of sdl2 at build-time (kivy_deps.sdl2_dev) to provide headers etc.
One way to handle it is to offer base, base_src, full, and full_src and you additionally list the latter when building from a source distribution which installs the dev deps. I don’t see any provision for using markers in extras_require to detect whether we’re installing a wheel or sdist.
So I’d need to fallback to do some runtime code in setup.py to detect if we’re building a wheel or compiling from source and provide different values for extras_require. My question is whether my plan is “safe” or if there’s a better way to do this. Because it feels a little “wrong” to me to execute code to conditionally change the listed deps - probably because I’m not sure of all the contexts where setup.py could be run.
I don’t understand. There is no difference in the process of compiling from source and building a wheel. Building a wheel is done by compiling from source.
If you mean installing from source, then yes, setup.py install is separate from setup.py bdist_wheel. But you shouldn’t compile differently - certainly pip assumes that it’s OK to build a wheel and install it when asked to do an install, and may not use setup.py install at all (at the moment, it still uses setup.py install for non-PEP 517 projects, but that’s considered “legacy” and may change).
If what you mean is that you want to use markers in build-time dependencies, then you should be able to do this in pyproject.toml (which is the standard way of declaring build-time dependencies nowadays).
I’m fairly sure there is a better way to do what you want, but it’s hard to tell what that would be from your description. Could you clarify what you’re trying to do, that means “installing a wheel” vs “installing a sdist” have to be treated differently?
To be clear on my perspective:
“Installing a wheel” means extracting files from a zip, and putting them in the correct locations in a filesystem. No compilation required, ever.
“Installing a sdist” means building the project (in the same way as you would to build the wheel) and either moving files from their build location to the target locations, or packing the built files into a wheel and then installing that wheel (the latter is what packaging is moving towards in a post-PEP 517 world).
From that perspective, I don’t understand the problem you’re having, hence my request for clarification.
Sorry for not being clear, but my meaning was exactly the two situations you’re describing in the two bullet-points: installing a wheel vs a sdist.
It’s not exactly clear to me how you’d use pyproject.toml for what I need so I’ll describe a more concrete example of what I’m doing. Say we use sdl2 on windows only, linked through cython. At runtime we need sdl2 binariesmy_deps.sdl2, at compile-time we also need the headers from my_deps.sdl2_dev.
Then, if installing a sdist we’d do pip install proj[base,base_src], otherwise just pip install proj[base]. But, I instead was looking to do something like:
So we can always just do pip install proj[base]. I don’t see how pyproject.toml would help for the setuptools backend here? But I’m not familiar with it.
You’d just put my_deps.sdl2_dev; sys_platform == "win32" in the requires field of pyproject.toml, surely? That would cause pip to install the dev package in the build environment when building a wheel as part of pip install`, and then the wheel could be installed without needing the compile-time dependency to be present at runtime.
All of this would happen automatically whenever you did a pip install, with an existing wheel being used if possible, or a wheel being built on the target system, and then used for the install, if there’s no pre-existing wheel.
Right, but the issue is that sdl2 is optional. You’d only need sdl2’s headers at compile time if you intend to use sdl2. We have other dependencies, such as gstreamer which is also optional and the dev wheel is fairly large. Your solution would install all these dev packages when a wheel for proj is not available and you get the sdist (e.g. after py3.8 is released before we can upload a wheel to pypi of proj for 3.8).
Ideally though, if you don’t plan to use gstreamer or sdl2, the dev version of these deps should not be installed when you build from a sdist. That’s why I’m trying to find a way to specify them through extras_require so you can say pip install proj[full] and then it’d get just gstreamer if there’s a wheel, otherwise it’d also get gstreamer dev to be able to build the project with gstreamer support. And if you don’t plan to use gstreamer at all, you’d just do pip install proj[base] and you’d just get the minimal required deps.
But with your solution you’d always get all deps dev packages when a wheel is not (yet) available on pypi and a user tries to install the package. And I didn’t see setuptools pyproject.toml support something like extras_require?
So going back to my original solution, I’d do something like:
If we take your optional sdl2 dependency as an example, can you make a package proj-sdl2 with an unconditional dependency on sdl2 for runtime and the sdl2_dev in require in pyproject.toml and put all the required code there? Then in proj you can have extras_require={"sdl2": ["proj-sdl2"]}. This means only users that install proj[sdl2] will pay the cost and only if they download the sdist.
This also avoids another issue: if you have a project with conditional dependencies there’s no way to tell whether they were in effect for an existing wheel. This applies to wheels in PyPI and wheels cached on user machines (which pip does automatically). When a user installs proj the first time pip will likely build and cache a wheel, then every other time they try to install the same version it will just pick up that existing wheel and whatever optional things happened to be enabled at the time.
While that’s certainly a good idea and I have thought about this approach in the past - it is something we probably should do in the long term. I’m afraid we can’t really do that currently because it would take much much more effort than we currently have available to make all our dependencies stand-alone projects (we have a few of them).
Currently they are intertwined with our build system, we’d need to move all that out to new independent projects, one for each dependency and make sure they still work and have a cython interface with releases etc. And then we’d need to maintain these dependencies when people want to use them for other projects, open bug reports etc. We just don’t have the developer time to do that. Right now, these deps are simply dlls and headers/libs that we reference directly from our project.
But what you mentioned about cached wheels makes me think that when someone builds from a sdist, we should install all our deps, because for the most part they’ll use a wheel, and the few cases where they compile from source it may be better to simply compile with all the dev packages installed.
Right now, these deps are simply dlls and headers/libs that we reference directly from our project.
It can stay the same with the setup above. The fact that proj[sdl2] installs a separate proj-sdl2 project is an implementation detail; the source for that package could be in the same project as proj. There would need to be separate files for the Python build (e.g. setup.py or pyproject.toml) and the build would need to generate separate artifacts, but it does not need to fragment the support or development endpoint.
it may be better to simply compile with all the dev packages installed
That is the safest approach in my opinion. If you’re already planning to create pre-built wheels then it should be pretty rare that it’s actually needed.
)