Remove bdist_wininst command

I ran the setuptools test suite on a Python without bdist_wininst. I had to remove setuptools/command/bdist_wininst.py file to be able to run tests. Result: 1 failed, 665 passed, 17 skipped, 5 xfailed, 1 xpassed, 3 errors.

Errors and warnings:

  • test_distutils_stdlib
  • test_distutils_local_with_setuptools
  • test_distutils_local
  • test_bdist_wininst_warning

I understand that almost everything works in setuptools, unless a few specific tests which expect bdist_wininst command to exist. It sounds doable to fix 4 tests.

1 Like

I removed the command: bpo-42802: Remove distutils bdist_wininst command (GH-24043) · python/cpython@0e2a0f7 · GitHub

Follow-up issue in setuptools: distutils bdist_wininst command has been removed in Python 3.10 · Issue #2527 · pypa/setuptools · GitHub

2 Likes

I agree with Steve that the bdist_wininst maintenance can continue as a third party module. It would be easy to contribute to a third party project than CPython itself.

1 Like

Well, I started to look at the backlog of issues for wininst as soon as the top post went out. I ran into unrelated issues running the test suite, so last weekend was consumed by that. I was looking forward to getting into the actual issues this weekend, but apparently, that is something that isn’t wanted anymore.

< rant >
At what point, in a volunteer project, is 1 week deemed more than enough time for non-core developers to analyze and resolve issues? On one of the python mailing lists, I remember that 30 days was considered a polite amount of time for responses in a distributed collaboration model, like Python’s development is.

I’ve been contributing to Python, in some form, on and off now for 20+ years. I started the first win64 buildbot nearly a decade ago. My Real Life doesn’t allot me as much time as I was once able to commit, but I do still enjoy as least following the discussions. I am lucky to be able to dedicate a few hours a weekend (sometimes even up to 8!) to all development tasks I have going on.

This particular occasion isn’t all that aggreges, but it seems to be the straw that broke the camel’s back. It seems that unless you can dedicate a ton of time, mostly immediately, your contributions are not wanted in this development community. I would think that for a small team, which apparently wants to grow, alienating old hats with robust understanding of parts (or most) of the Python codebase in favor of rookies with loads of free time isn’t necessarily a good thing. It’s not that rookies cannot grow into pros and fresh blood is truly needed to maintain project health, but to chase away contributors that do not need any training time seems wasteful.
< /rant >

I guess this is just my frustration of seeing things continually being eliminated without true replacements in place. The “once its gone, then they’ll make something better” strategy doesn’t cut it, IMO. It should be, "we have something that improves upon thing, let’s remove the old thing".

If you think that the current state of packaging has suitable replacements for what is provided by the wininst installers, please do enlighten me. A 1-1 listing of features would suffice.

FWIW, one of my first contributions to Python was to improve the extensibility of Distutils as our company one of the first adopters (anyone remember the make Setup fun-ness?)

1 Like

I was also very surprised to see the code removed as it did not look like there was consensus on this move and, at least, it would have made sense to fix the setuptools tests before removing the code, not leaving setuptools in a broken state. I don’t understand what is the rush in removing code that does not do any harm, other than magically solving a bunch of old bugs as invalid.

Anyhow, it should be easy to resurrect the bdist_wininst code into its own little package. If you like, I can bootstrap the effort and hand over the package to you once it implements the status quo ante. However, I will not be able to do more than check that the code runs. What do you thing of setuptoools_wininst as package name?

1 Like

I was wondering about this before reading your comment. It definitely makes sense IMO, although I cannot speak about the hand over part since I dont have any real-world usage of the command myself either. It would make perfect sense for anyone assuming maintenance of the package to propose it joining PyPA as well, if they wish to.

Another off-topic thought I have on this: Would it be a good idea for other distutils commands to be moved out of stdlib without too much effort?

1 Like

All the commands have already moved into setuptools… so… yes? The main thing we gain from removing wininst from the next release is fewer people having their installs blocked by bad AV.

I agree it was a quick move from this post to final result, but then this post is not the first time it’s come up. So rather than saying you need to be actively working on stuff every week (a simple “I’m looking at it” post doesn’t take a week, BTW, and would have been enough), we are really lacking any kind of central coordination or even a central discussion location that some people aren’t ignoring for any of a variety of reasons. That fracturing is probably our biggest problem right now, which I think is why most of us are trying to just twiddle little things at the edges and avoid involving too many people.

The fact that I’m posting this in the packaging category and not a core dev category kinda emphasises my point, I think.

2 Likes

Hi,

About the timeline:

  • There are 19 open issues on wininst: 13 issues are between 4 and 10 years old. The fact that wininst is not maintained is not a new fact.
  • 2016: PEP 527 was approved, it disallows to upload files other than source (sdist: tarball, ZIP) and wheel packages (bdist_wheel) to PyPI.
  • July 2019: I created Deprecate bdist_wininst discussion to propose to deprecate distutils bdist_wininst command, there was a consensus to deprecate it.
  • January 2021: I created this discussion to actually remove the deprecated command. I saw that people who work on packaging still agree to remove the command, so I removed it.

For me, there is now a clear consensus that wheel binary packages is the way to go according to developers who work on Python packaging.

The What’s New Python 3.10 entry clearly suggests to replace bdist_wininst with wheel. Some wininst features are not supported by wheel packages. It has been discussed that pywin32 requires to run a manual step to support Windows Services. IMO it’s an acceptable trade-off compared to wheel package advantages (ex: support virtual environments).

It has also been said that it’s perfectly fine if wininst installer and distutils bdist_wininst command development continues as a third-party project. People are also free to offer download of wininst EXE installers on their website (but not on PyPI: PEP 527).

I removed bdist_wininst to reduce the Python maintenance burden.

1 Like

It’s also worth noting that while wheels don’t offer a means to automatically run post-install commands, that is a limitation that has been discussed and acknowledged in the past. It’s possible for an update to the wheel format to be proposed which adds such a capability, but to my knowledge, no-one has done so. There are a number of problems which any such proposal would have to address, because one of the key principles of the existing packaging ecosystem is to move away from running arbitrary code at install time, but that’s something that can be discussed as part of any proposal.

So in short, yes wheels are not currently a complete feature-by-feature replacement for wininst, but no-one has so far seemed interested in fixing that.

1 Like

People who work on PyPI are fighting against typosquatting packages: malwares which steal secrets like SSH private keys. There are 1 to 50 packages that should be manually removed. It’s a lot of work. Automating the detection of malwares is an even larger project, it seems like some people are working on that.

I like the fact that wheel packages cannot run arbitrary (Python) code. For me, it’s a great argument for stronger security. I would even suggest to require a command line option to opt-in for source installation (run “legacy” setup.py).

pip is a generic package manager. Companies can build more specialized installers for their own needs. But I would recommend against making pip too complicated, and focus on the most common use cases. pip cannot solve all problems. The team working on pip is too small.

4 Likes

So if there were no open issues, what then is the reason for removal? I believe that I am set up now to be able to fix all of these. Locally, I have resolved the biggest obstacle, being able to test built installers without user interaction.

.tar.xz files cannot be uploaded either, should we remove lzma, too? :wink: As you state later, these files can be hosted elsewhere, so I do not think this provides any rationale.

That is true, however you yourself state (Issue 37481: Deprecate bdist_wininst: use bdist_wheel instead - Python tracker) that at least 2 years should pass (dated 2019-07-04) and “it’s not too painful to maintain a few more years.”

Fine, then I offer my services to maintain the bdist_wininst codebase.

1 Like

Ah, sometimes else: there is PEP 632 -- Deprecate distutils module | Python.org plan (still a draft) to deprecate distutils. So if someone wants to continue maintaining wininst, I suggest to either contribute to setuptools, or to develop a setuptools plugin somehow.

3 Likes

Point still stands, then I offer to take on the mantle of the entirety of distutils.

Things should be much easier now that setuptools has its own private copy instead of monkey-patching willy-nilly throughout the distutils codebase.

1 Like

If you’re volunteering to maintain distutils going forward, I’d suggest making a post on the thread you’re quoting from, since that’s way broader than just maintaining bdist_wininst in a separate location.


Note that just because Steve will withdraw the PEP, does not mean that some other person might not pick it back up and take it forward.

Various things that distutils does are straight-up non-functional/significantly degraded/incompatible compared to the rest of Python Packaging tooling. IMO letting distutils “die” is the best way to deal with a lot of user confusion and issues that have plagued us for a really long time. And that’s even if there’s some volunteer dev availability to maintain distutils (which will still be significantly less compared to the volunteer dev availability across third party build backends and build frontends).

3 Likes

I’ll also echo this – I’d rather have bdist_wininst in setuptools (core, or as a plugin) than to have it be improved upon in CPython’s distutils copy.

We’ve been making nearly all the improvements related to distutils in setuptools for years now (for various reasons that I hope everyone reading this post is familiar with), and I’d prefer that we continue with that, instead of taking a completely different approach for this.

5 Likes

Isn’t that exactly the monoculture that is routinely rallied against here? Shouldn’t we have competition in the build tools space? It is unfortunate that there is significant overlap between setuptools and distutils, but that doesn’t mean that they cannot coexist and drive improvements in each other.

Next weekeend, I plan to tackle as many BPO issues as I can and look into implementing a PEP 517 interface for distutils.

1 Like

We do, but nobody can compete with “already installed on every machine”, which makes distutils the expected winner in every head-to-head. Moving its functionality to setuptools (and eventually removing the preinstallation of setuptools) removes that advantage and helps encourage users to investigate further.

This problem is exacerbated by our community’s tendancy to value “first party” over all other criteria.

BTW, the distutils PEP has already gone to the steering council (don’t have the link handy on my phone but it’s on their github repo issue tracker). You probably need to post on that saying that you’ll take over maintenance if they reject it, and they can weigh it against the other concerns.

2 Likes

Well, if we want equal competition, then (as @steve.dower said) distutils should be moved out of the stdlib - on the one hand so it doesn’t have an unfair “comes with Python” advantage, and on the other hand, so that it isn’t unfairly limited by Python’s development cycles.

Given that setuptools is essentially a version of distutils with many enhancements that users have wanted over the years and distutils never provided, I fear that you’re going to end up either reimplementing a lot of setuptools, or looking very underpowered in comparison with setuptools. But it’s your choice, I guess…

For a start that will mean implementing an equivalent of bdist_wheel for distutils. At the moment, distutils can’t build wheels at all, which is why everyone uses setuptools…

But if you want to enable people to use distutils without setuptools, then yes, a PEP 517 interface is essential. We’re (slowly) removing all the ways in which the packaging ecosystem special-cases setuptools, in favour of standardised interfaces like PEP 517. We’re not there yet, so there are still some areas where setuptools has preferential treatment (editable installs, for example) but the goal is 100% to make any build back-end that implements the standards an “equal citizen”.

One other thing you should probably look at is manylinux support - I don’t know to what extent code changes were needed in setuptools to handle manylinux, but I’m pretty certain that any which were won’t have been backported to distutils.

2 Likes

Wait, what? I’m directing you there to avoid you having to redo years of work done in setuptools. How’s that promoting a monoculture? :man_shrugging:t2:

distutils is legacy code that has been ignored for multiple years now and all the development effort that would be needed to bring distutils up-to-date with the current packaging standards and mechanisms has been done already: that project is called setuptools.

I don’t believe distutils should be considered a competitor of setuptools – distutils is the legacy codebase that setuptools is built upon, and newer versions of setuptools don’t need it to exist in the Python standard library anymore. Notice that setuptools (on import / install) also takes control of the distutils import names.

We’ve been telling users for years to stop using distutils, because setuptools fixes many of distutils’ flaws and improves upon it in even more ways.

[snip] Others have already talked about the “comes with Python” and its implications, so I won’t touch upon that further.

4 Likes

PEP 632 -- Deprecate distutils module | Python.org has been approved by the SC. I suggest you to fork distutils as “distutils3” and make it super attractive. Thanks to pyproject.toml, people will be able to select it rather than setuptools. There is no need to make it into the stdlib.

Having something in the stdlib leads somehow to a monoculture. Removing distutils from the stdlib should help the competition, no? Because now all distutils alternatives are at the same level.

Note: there was a distutils2 project in the past, but it was abandoned after 1 or 2 years.

2 Likes