Remove bdist_wininst command

Packaging
1

Hi,

After deprecating the distutils bdist_wininst command in Python 3.8 and fixing setuptools to handle a missing bdist_wininst command, I now propose the remove the command:
https://bugs.python.org/issue42802

In July 2019, it was said that only wheel packages should be accepted on PyPI:

People are free to distribute wininst installer programs outside PyPI, but it’s a pain to maintain the “wininst” code base (the installer itself and the distutils command), there are 19 open bdist_wininst issues and most are inactive for between 1 year 1/2 and 10 years.

Do you see any good reason to keep the bdist_wininst command?

Victor

2 Likes
2

I don’t see any good reasons to be maintaining this support, given that nearly all of Python Packaging tooling is moving toward standardising around wheels and, IIUC, no automated installer supports installing using this format from a package index.

I’m on board for removing this unless someone has a valid use case that simply cannot be satisfied without this, and that someone also steps up to actively maintain this and work on existing issues. :slight_smile:

2 Likes
3

:rofl: :rofl: :rofl:

2 Likes
4

Yeah, time to remove it. Hopefully setuptools has stopped unconditionally importing it already, but if not, it probably makes sense to replace it with a nearly empty stub file, as I’ve already done for some of the release (look under PC/layout). That solves the maintenance burden, but only actually breaks people who try and use it, rather than importing something that wants to monkeypatch it.

3 Likes
5

Copy of my message on the issue.

FYI last year, I fixed setuptools v45.2 (released in February 2020) when bdist_wininst is missing:

But I didn’t try setuptools with bpo-42802: Remove distutils bdist_wininst command by vstinner · Pull Request #24043 · python/cpython · GitHub to double check.

2 Likes
6

We should be clear that this will completely break extensions which rely on the features it supports and which aren’t supported by wheels. pywin32 is one such package - many pywin32 things work from wheels, but many things (such as services, COM servers, etc) do not.

FWIW, bdist_wininst is currently broken on 3.8+, and a PR has languished for over 3 months - but people can work around that by applying the patch to the python version that builds the distribution.

Honestly, just because “most” packages don’t use bdist_wininst doesn’t mean that it’s OK to break packages that do when there’s no possible way for those packages to “modernize” and keep the existing functionality. I don’t see how the fact that there are open issues for bdist_wininst justifies killing it entirely without any way to replace the lost functionality (and as above, the fact that issues remain open aren’t just because they don’t have a PR)

1 Like
7

Oops - I meant to link to Issue 41771: bdist_wininst doesn't execute postinstall script - Python tracker

1 Like
8

Technically, is it possible to maintain bdist_wininst outside CPython code base? I don’t know if setuptools/distutils allows “pluggins”.

I’m not surprised that there is at least one project which still rely on bdist_wininst. But the question is if CPython core developers (which is a small team) should continue to maintain this code base. It’s a matter of maintenance burden and who is going to maintain the code.

Moreover, wininst seems to be very specific. Why not using MSI which is better integrated with Windows for example?

Would you mind to elaborate how the fact that pywin32 would be distributed a wheel package (which is basically a ZIP file) prevents to use services and COM servers on Windows?

1 Like
9

pywin32 only distributes wheel packages on PyPI: pywin32 · PyPI

The README says:

It sounds like an acceptable tradeoff, no?

2 Likes
10

The primary reason for dropping bdist_wininst and bdist_msi right now is that they don’t support virtual environments, and it seems unlikely that they ever reasonably could (they can’t really support multiple installs on the same machine, for example). Packaging in general is moving toward self-contained, side-effect free installation, rather than system-wide (or user-wide) installs.

There is also absolutely no reason why they need to live in core, and it would be entirely reasonable (once the deprecation is formalised) for someone to take it on as a standalone project, even retaining the same setup.py commands. That would allow the few projects that benefit from them to keep producing and distributing them, as well as actually improving them. But as long as they live in the core repo, that is much too hard.

Ultimately, this kind of cleanup process is going to force those who continue using them out of habit to evaluate whether they really want to be using it or not. If they really do, there are options for continued maintenance. But if it’s cheaper/easier to move to something else, that is also a totally valid solution. Leaving it unmaintained while we repeatedly release “new” versions is not viable long-term.

2 Likes
11

I ran the setuptools test suite on a Python without bdist_wininst. I had to remove setuptools/command/bdist_wininst.py file to be able to run tests. Result: 1 failed, 665 passed, 17 skipped, 5 xfailed, 1 xpassed, 3 errors.

Errors and warnings:

  • test_distutils_stdlib
  • test_distutils_local_with_setuptools
  • test_distutils_local
  • test_bdist_wininst_warning

I understand that almost everything works in setuptools, unless a few specific tests which expect bdist_wininst command to exist. It sounds doable to fix 4 tests.

1 Like
12

I removed the command: bpo-42802: Remove distutils bdist_wininst command (GH-24043) · python/cpython@0e2a0f7 · GitHub

Follow-up issue in setuptools: distutils bdist_wininst command has been removed in Python 3.10 · Issue #2527 · pypa/setuptools · GitHub

2 Likes
13

I agree with Steve that the bdist_wininst maintenance can continue as a third party module. It would be easy to contribute to a third party project than CPython itself.

1 Like
14

Well, I started to look at the backlog of issues for wininst as soon as the top post went out. I ran into unrelated issues running the test suite, so last weekend was consumed by that. I was looking forward to getting into the actual issues this weekend, but apparently, that is something that isn’t wanted anymore.

< rant >
At what point, in a volunteer project, is 1 week deemed more than enough time for non-core developers to analyze and resolve issues? On one of the python mailing lists, I remember that 30 days was considered a polite amount of time for responses in a distributed collaboration model, like Python’s development is.

I’ve been contributing to Python, in some form, on and off now for 20+ years. I started the first win64 buildbot nearly a decade ago. My Real Life doesn’t allot me as much time as I was once able to commit, but I do still enjoy as least following the discussions. I am lucky to be able to dedicate a few hours a weekend (sometimes even up to 8!) to all development tasks I have going on.

This particular occasion isn’t all that aggreges, but it seems to be the straw that broke the camel’s back. It seems that unless you can dedicate a ton of time, mostly immediately, your contributions are not wanted in this development community. I would think that for a small team, which apparently wants to grow, alienating old hats with robust understanding of parts (or most) of the Python codebase in favor of rookies with loads of free time isn’t necessarily a good thing. It’s not that rookies cannot grow into pros and fresh blood is truly needed to maintain project health, but to chase away contributors that do not need any training time seems wasteful.
< /rant >

I guess this is just my frustration of seeing things continually being eliminated without true replacements in place. The “once its gone, then they’ll make something better” strategy doesn’t cut it, IMO. It should be, "we have something that improves upon thing, let’s remove the old thing".

If you think that the current state of packaging has suitable replacements for what is provided by the wininst installers, please do enlighten me. A 1-1 listing of features would suffice.

FWIW, one of my first contributions to Python was to improve the extensibility of Distutils as our company one of the first adopters (anyone remember the make Setup fun-ness?)

1 Like
15

I was also very surprised to see the code removed as it did not look like there was consensus on this move and, at least, it would have made sense to fix the setuptools tests before removing the code, not leaving setuptools in a broken state. I don’t understand what is the rush in removing code that does not do any harm, other than magically solving a bunch of old bugs as invalid.

Anyhow, it should be easy to resurrect the bdist_wininst code into its own little package. If you like, I can bootstrap the effort and hand over the package to you once it implements the status quo ante. However, I will not be able to do more than check that the code runs. What do you thing of setuptoools_wininst as package name?

1 Like
16

I was wondering about this before reading your comment. It definitely makes sense IMO, although I cannot speak about the hand over part since I dont have any real-world usage of the command myself either. It would make perfect sense for anyone assuming maintenance of the package to propose it joining PyPA as well, if they wish to.

Another off-topic thought I have on this: Would it be a good idea for other distutils commands to be moved out of stdlib without too much effort?

1 Like
17

All the commands have already moved into setuptools… so… yes? The main thing we gain from removing wininst from the next release is fewer people having their installs blocked by bad AV.

I agree it was a quick move from this post to final result, but then this post is not the first time it’s come up. So rather than saying you need to be actively working on stuff every week (a simple “I’m looking at it” post doesn’t take a week, BTW, and would have been enough), we are really lacking any kind of central coordination or even a central discussion location that some people aren’t ignoring for any of a variety of reasons. That fracturing is probably our biggest problem right now, which I think is why most of us are trying to just twiddle little things at the edges and avoid involving too many people.

The fact that I’m posting this in the packaging category and not a core dev category kinda emphasises my point, I think.

2 Likes
18

Hi,

About the timeline:

  • There are 19 open issues on wininst: 13 issues are between 4 and 10 years old. The fact that wininst is not maintained is not a new fact.
  • 2016: PEP 527 was approved, it disallows to upload files other than source (sdist: tarball, ZIP) and wheel packages (bdist_wheel) to PyPI.
  • July 2019: I created Deprecate bdist_wininst discussion to propose to deprecate distutils bdist_wininst command, there was a consensus to deprecate it.
  • January 2021: I created this discussion to actually remove the deprecated command. I saw that people who work on packaging still agree to remove the command, so I removed it.

For me, there is now a clear consensus that wheel binary packages is the way to go according to developers who work on Python packaging.

The What’s New Python 3.10 entry clearly suggests to replace bdist_wininst with wheel. Some wininst features are not supported by wheel packages. It has been discussed that pywin32 requires to run a manual step to support Windows Services. IMO it’s an acceptable trade-off compared to wheel package advantages (ex: support virtual environments).

It has also been said that it’s perfectly fine if wininst installer and distutils bdist_wininst command development continues as a third-party project. People are also free to offer download of wininst EXE installers on their website (but not on PyPI: PEP 527).

I removed bdist_wininst to reduce the Python maintenance burden.

1 Like
19

It’s also worth noting that while wheels don’t offer a means to automatically run post-install commands, that is a limitation that has been discussed and acknowledged in the past. It’s possible for an update to the wheel format to be proposed which adds such a capability, but to my knowledge, no-one has done so. There are a number of problems which any such proposal would have to address, because one of the key principles of the existing packaging ecosystem is to move away from running arbitrary code at install time, but that’s something that can be discussed as part of any proposal.

So in short, yes wheels are not currently a complete feature-by-feature replacement for wininst, but no-one has so far seemed interested in fixing that.

1 Like
20

People who work on PyPI are fighting against typosquatting packages: malwares which steal secrets like SSH private keys. There are 1 to 50 packages that should be manually removed. It’s a lot of work. Automating the detection of malwares is an even larger project, it seems like some people are working on that.

I like the fact that wheel packages cannot run arbitrary (Python) code. For me, it’s a great argument for stronger security. I would even suggest to require a command line option to opt-in for source installation (run “legacy” setup.py).

pip is a generic package manager. Companies can build more specialized installers for their own needs. But I would recommend against making pip too complicated, and focus on the most common use cases. pip cannot solve all problems. The team working on pip is too small.

4 Likes