Streamlining Dev Setup on Windows

Lately I have been involved (at work) in an Electron project, and was made aware of windows-build-tools. I am not particularly knowledgable about Node packaging, but from what I’ve learned, it is a dummy package that, upon installing (globally), prepare the target Windows machine to compile native Node modules.

This gets me start wondering whether an equivalent for Python (probably under the PyPA umbrella?) would be useful. It should be quite doable since Visual Studio installers since 2017 (used by Python 3.5+) are freely available, and both highly customisable and scriptable. Python 2.7 support is also possible thanks to Microsoft’s dedicated toolchain, but I don’t think it is worhwhile to persue in year 2019.

(Speaking of Python 2.7, Node’s build toolchain seems to depend on it as well. I wonder whether they have a migration plan moving forward.)

How this should be done, however, is another problem. The simplest approach would be similar to windows-build-tools, i.e. publishing a dummy package to PyPI, and perform whatever tasks required in a custom I am not very fond of this idea, however, since IMO it is a very bad idea to download and run arbitrary code, in general. Some other ideas from the top of my head:

  • A dedicated pip subcommand?
  • A build backend hook in setuptools, or a standalone project? This adds an additional probability a the package author can add appropriate configuration in pyproject.toml to fully automate the process.

I am quite interested in looking into this topic myself, since Windows setup is really a pain when teaching newcomers. Windows is so often (unfairly, IMO) viewed as inferior when a user needs to manually install additional hundreds MBs of things to the machine, while other platforms seem to “just work” (not because they don’t need to, but have a more straightforward way to install them before the user notices).

1 Like

This sounds like a great idea!

We could have a “windows-build-tools” package that only installs on Windows, that uses a dedicated PEP 517 backend that would (conditionally?) install the required tooling and possibly an importable package as a documented way to get information about the tooling it’s installed. pip 19.0 has PEP 517 support so that’s definitely not a far future thing.

We could definitely have a project under the PyPA umbrella though I think it’s better to get it up and running reasonably well before we move it into the pypa umbrella.

Future pip integration is also something that we could discuss once we have it working. (I, personally, would really like to make pip a more complete tool with publishing, building and installing capabilities)

I would much rather package distributors publish wheels. That way there’s at least a little bit of obligation on their side to make things work, and it’s very clear when they are not guaranteeing Windows support at all.

There are plenty of free CI systems that will do the builds. I’d rather integrate a “getting started” workflow into PyPI, possibly including some sort of webhook that can trigger a build when an sdist is published (where the package owner owns the build, not PyPI). Anything that makes it easier for package maintainers to release source-only packages and push the burden onto every single end user is a misstep, in my opinion.

I totally agree, but maybe this can be considered in a different way.

I use Windows for Python development myself (as well as other platforms), but it’s still take me a lot of effort if you ask me to create a minimal Python development setup right now. It’s not obvious what version of Visual Studio I should install, or what components I should select.

On macOS I can xcode-select --install; on Ubuntu python-dev provides everything I need. For Windows, however, I need to open the browser, search for VS2017 Build Tools, download, run it, and decide what components I want. It would be immensely helpful if there is a more streamlined path to do this.

Agreed, and that’s a general problem with getting a dev environment on Windows that I’m helping work on solving (via my day job at Microsoft).

The best I can offer right now is that installing Visual Studio 2017/2019 Community and selecting Python and then Python Native Development will get you the IDE and its debugging tools as well as the compilers for 3.5 onwards. (Given the target audience is package developers, the IDE makes sense, and really isn’t that big a deal given the multi-GB download and install of compilers and system headers/libraries.)

It’s not a good look for me to add recommendations in blatantly PSF-endorsed locations such as the main site, but if someone more independent wants to do it I’m happy to validate and occasionally update/fix the info. Getting it in front of package developers rather than normal users is the tricky part - my guess is 95%+ of the millions of downloads each month don’t require these tools.

1 Like

This is part of the reason I said that the work should be done independently first – putting it in something (like PyPUG) would be a discussion that has to take stuff like this into account. I’m glad you brought this up early.

Implementing things independently is a much lower friction task and a pre-requisite for inclusion anyway, so I didn’t want to block that idea. :slight_smile:

I’m more than happy to defer to Steve’s judgement here since he’s a lot more involved in work related to this topic, than I am.

OTOH, there is a guide planned for PyPUG about “setting up your baseline python environment”; which I hope would make this situation a little less confusing to deal with for end users. Create a new guide for setting up a baseline Python environment · Issue #396 · pypa/ · GitHub

1 Like

I’ve started to try out things, and here’s my experiement setting up a headless minimal build environment:

I’m having some problems figuring out what components/payloads I should install, however. The environment currently gives me error: Unable to find vcvarsall.bat when I try to build the module.

If you manually select workloads until it works, you should be able to export the configuration from the installer as a JSON file that you should be able to provide to the installer. I haven’t actually tried it though.

Wow, that’s very handy! Thanks!

Ah, it turns out I’m hitting bpo-35699 (distutils cannot find Build Tools 2017 since 3.7.2). I switched to 3.7.1 and things work as expected (after adding the missing components).

Hey @uranusjr, were you able to spend some time working on this?

I’ve since figured out the correct minimal required components required to build CPython extensions. The followings work for 3.6 or later (except 3.7.2, as mentioned above):


Progress stalled a bit afterwards since Chinese New Year and I took my vacation to the Philippines :stuck_out_tongue: I have since returned, but am still picking up slack at work, and it’ll probably take me a couple of weeks to get back into gear.

Here’s a part of my todo list about immediate hurdles:

  • The CLI behaviour of vs_BuildTools.exe does not seem consistent for some reason. The Dockerfile I use to test it sometimes fail seemingly randomly. There are reports online with similar issues.
  • The installer itself requires a update (KB3151800?) update to run on older versions of Windows.
  • How do we integrate if the user already has existing VS installations (Build Tools or otherwise)? Node’s windows-build-tools ignores any existing installations and manage its own, but I think that’s sad.