Windows Store package to become installer/launcher?

Ideas
61

Despite the follow-ups, yes it was a bit sarcastic. There are actually some really easy ways to make it work for pip that have been deemed unacceptable by the team, as well as some more recent OS additions that can make it cleaner (apart from needing ctypes, semi-private APIs and a recent OS).

Doing it for CPython is harder, since it’s not just the entry-point executable that’s needed, but every dynamic library that might get loaded in the process. The only OS support that really exists here are the installer APIs, which require a reboot.

But the point is that the Python manager really can’t rely on a user-accessible or modifiable runtime without constraining the user in ways that are inappropriate. The answer is to use a private runtime, and hijack a command from the user’s runtime so that it runs with the private one and can manage the user’s one (whether that hijacked command is python -m manage, python install or whether we make users discover/learn a new command is the discussion).

62

I thought we could maybe use python -manage, but someone has already registered anage. :slight_smile: We could use python --manage as an alias for python -m _manage though. install is also still available.

1 Like
63

I’ve found that the automatic python app execution alias in windows is more of a hassle and likely to confuse users than a help, and that the windows store versions end up worse than installing it normally because of various limitations of windows store apps. a change that at least makes the windows store package an installer would help, but is there any way we can just ask Microsoft to not do this, or have the alias point to py if it exists, or prompt the user to install python from python.org if not?

3 Likes
64

I’m all for moving towards a more powerful manager/launcher in general, but the various constraints in this situation make me unsure if it would really be an improvement from the user perspective.

I guess the main question I have is: what benefits are there to using the Windows Store version of Python? The only one I can really see from reading the thread is “it is suggested when you try to run Python without having it installed” and maybe “it’s easy to do through a GUI people are familiar with on Windows”. But both those advantages only have to do with the install process, and will disappear once someone has Python installed. And on the hand it sounds like the Windows store install has various pitfalls compared to a normal Python install.

Given that, I agree with @Liz that it seems preferable to move in a different direction. Either have “run-python-before-it’s-installed” prompt to install a Windows Store app that is not Python but is instead a manager of some kind (and that manager would then install Python, blocking the install prompt), or, as a last resort, just put warnings on Python.org explicitly recommending that people not use the Windows Store installer.

65

If they’re unwilling to do this, can’t we just ask them to remove the alias, such that people look up the correct way to install Python?

2 Likes
66

The only real option is to get the alias entirely removed, and so users will go back to getting “Command not found” errors. For reference, about 30,000 people daily would get that error (charts below), and about 1/3 of those are choosing to install, which is a very high conversion rate. So that’s a lot of real Python users who would revert to complete failure with only the top Google result to help them. I personally don’t want to be responsible for causing that many failures.

I can’t share specific numbers, but the vast vast VAST majority of Python-related issues reported in Visual Studio are due to the python.org installer being run. The reliability is just too low, and Microsoft knows it, and so it won’t be recommended anywhere automatically. A link to python.org in docs is probably the best we’d get[1].

Replacing the installer with something more reliable is the point of the Store install. It’s the newer installer technology that is supported, reliable, and efficient. If we don’t use it, we write something entirely from scratch, and deal with the many many issues afflicting installers ourselves. I’m not signing myself up for that, but if we get a serious (10 year maintainer[2]) offer from someone willing to do it, we can consider it.

We could distribute the more modern ourselves from python.org if we wanted, instead of just the Store, but frankly the compatibility issues with existing libraries are why we don’t. We need some option for people who need a fallback, and it’s taken a while for enough of the common libraries to catch up. I personally haven’t encountered any code[3] that doesn’t work with the Store install in a couple of years, but I see enough angst about it that I’d rather not make it the default. A managed install as I’m proposing here wouldn’t have those technical issues,[4] it’d just have security/reliability risks instead (e.g. upgrades would probably have to go back to being clean installs, and there’s no way to prevent users irreversibly breaking their own runtime or the ability to repair it).

All of this stuff matters way more for the 95-99% of users who aren’t on here to present their position than it does for anyone involved in this forum - we are literally the Python experts, and can handle stuff like portable installs and global site-packages without wondering why everything just broke[5]. Most users are not going to cope well with these things happening, and most of those users are on Windows, so I feel a real responsibility to make sure Python works well for them and continues working no matter what they inadvertently try to do to it. Which to me, means:

  • don’t give them the python.org installer
  • don’t blindly modify their PATH
  • don’t make them choose a Python version (until they know they need to)
  • don’t make them need administrator privileges
  • give them safe in-place updates, preferably automatic
  • give them a big “Repair” button that works
  • give them a big “Remove” button that works
  • give enterprises the ability to approve our installs/builds

The “Windows” campaign ID is only from the built-in redirector. It makes up about 70% of all traffic for Python 3.12 (where it’s currently pointing).

image
image1687×633 37.8 KB

image
image1706×636 34.7 KB

  1. Without me having to make dishonest representations to the people who control these things. Which frankly, I’m never going to do. ↩︎

  2. And for the record, the last “10 year maintainer” we received lasted about 2 years, so expect me to be incredibly skeptical. ↩︎

  3. Bad documentation all the time, but I don’t count that. That always existed, and will always exist. ↩︎

  4. Notably, it wouldn’t run under the isolation that Store Python currently gets. So search path/DLL hijacking becomes a risk again, but at least the libraries that never fixed their need for DLL hijacking will work… ↩︎

  5. Though we still do that to ourselves as well :slight_smile: ↩︎

4 Likes
67

Thanks for the info, it seems like a clearer benefit now if I take this all at face value (I am).

I understand that you can’t give numbers, but can you explain what you mean by this? It almost sounds like you’re saying python.org’s installer has a fundamental reliability issue, and if so, shouldn’t that be what we are fixing? (I’ve never experienced this, and this statement is surprising to me)

1 Like
68

It’s inherited from MSI/Windows Installer’s fundamental reliability issue, and the large amount of feature creep that went into it over the last couple of decades. That installer format can do a lot of things, and most aren’t commonly used anymore (if they ever were). A lot of what it can do is via generic scripting interfaces, rather than OS-level integration (e.g. environment variable management isn’t nice, it’s a pretty decent sized hack). The entire system is semi-deprecated in Windows, and it’s so widely used that touching anything at all is pretty much riskier than leaving people with their workarounds.

But frankly, the main issue with the installer is aggressive virus scanners that simply block parts of it without warning or feedback. So installs just one day do not work for unclear reasons, but aren’t cleaned up properly because the virus scanner didn’t handle it well. This is somewhat unavoidable, but because MSI tracks everything independently from “is the actual file on disk”, you get stuck very easily.[1]

The newer installer technology is MSIX, which is what the Windows Store uses. It’s essentially a zip file with some metadata, and the installation is managed by the OS in a much more reliable way. Some of its features are basically deprecated (e.g. console commands aren’t getting fixed or improved anymore, which I’m still fighting the team over), but otherwise it’s the better option.

If we were to go to something else, it would essentially be a self-extracting ZIP that does its own metadata handling and tries its best to upgrade/repair/uninstall. Now, that is what I’m proposing, but following a manager model (where the user first installs a tool to download/extract/manage Python installs) rather than an installer model (where the installer also installs the uninstaller/manager for itself).

It’s going to have some of the same issues, and create new ones for us, but I think the trade-off should be in our favour apart from when everyone gets mad about the design :slight_smile: Which is why I’ve resisted making small changes to installation (which will make people mad without fixing any issues) until I have enough vision to make one big change (which will make people mad that it changed but hopefully happy that it’s now better).

Edit: Here’s a GitHub query if you want to read through reports of issues with the installer.

  1. The database says it’s installed, regardless of whether the file is there, and so reinstalling doesn’t necessarily bring it back. ↩︎

1 Like
69

This seems like a reasonable scoping of it now that I’m following the issues you see. To rephrase my original question for the new information, while keeping the underlying intent of asking: If and when we get to this point, would it be reasonable for the PSF to own this store entry, for the Microsoft default app execution alias to point to it, and to point users to this over the python.org installer?

My concern here is that we have too many ways already that non-technical users end up with a python installation or portable environment, and that I’ve experienced issues with the windows store limitations, there are clearly problems with the status quo I wasn’t aware of, and it seems to be neither “official” (user perspective, loosely; PSF or OS provided) of these are the most reliable, that instead being one provided by hatch, pdm, or uv through a 3rd party build of python that the user might not even know isn’t just setting up python the same way they would, but automated.

Is there a path to doing this in a way that cuts down on the number of ways people get python when they aren’t aware of how python is redistributed? Do we have a path for windows with this that could work for environment management tools too? Is this a place where we should be working with the other existing builders and installers to attempt to bring what they have done into a more unified use?

1 Like
70

I hope so, since this is already the case. We just don’t direct people already on python.org to the Store, for reasons I gave above.[1]

I’m hopeful this could be that one way. I don’t have any issue with other environment managers access the same packages we would use, and Windows builds are portable by default (it’s only the ways we try to be POSIX-like that makes them non-portable), so they’d likely go for it. Most of indygreg’s patches are for static builds, so we’d take them, but we don’t need them. (The PyBI builds Nathaniel did a while back were just our builds repackaged, but unmodified.)

  1. And also some people complained that we were advertising a commercial product, but I think they just misunderstood that the Store app is fully controlled by python-dev… technically owned by the PSF, but only really in their role as trademark owner and not in any practical manner. ↩︎

1 Like
71

One question I’m out of touch on. Is access to the Windows Store now a given in corporate environments? When I was working, it wasn’t, but applications like Python were allowed, although they weren’t supported or formally approved by the IT department. As a result, downloading the python.org installer and installing Python that way was common for teams that wanted to use Python internally, but installing Python from the MS store could easily have been impossible.

My personal impression (of a limited, but to my knowledge typical of the business I was in, group of users) is that there is a significant amount of use of Python in a “business support” role - data analysis where Excel would have traditionally been used, automation of internal business processes, development of local webapps or monitoring systems, etc. Much of that user base can’t rely on corporate IT support, but can rely on an “if you don’t break stuff, we’ll leave you alone” attitude.

Maybe uv is a sufficient alternative install route for those groups. But I’d much rather there was an option to install from python.org (even if it’s just “download this .msix file and install it using Add-AppxPackage”).

1 Like
72

Yeah, I’m not sure how things have changed more recently. Initially, the idea was that enterprises would have a “private” Store where they’ve [bulk] added apps for their people to use. In this world, side-loading (“download this .msix”) was banned by default, because it’s exactly that shadow IT that the people who are paying say they want to avoid.

As of Windows 11, I’m pretty sure side-loading is enabled for everyone by default provided the MSIX is properly signed (which ours are). I assume this extends into Windows Enterprise (if anyone wants to test this, you should be able to download the sideloadable 3.13.0 MSIXs from here[1] and double-click python-3.13.0-amd64.msix).

Future work might include making a regular installer (probably with no configuration options) containing whatever launcher/manager we build, but I don’t want to start there. It’ll scare too many people if we say we’re replacing the old installer completely - I only want to improve the Store experience right now, and people who can’t access the Store won’t see anything change (yet).

  1. Apologies for the size, the only unauthenticated download option is “all the MSIX files at once”. Only the file I named is expected to work - the others probably won’t, but we don’t care. ↩︎

73

I just want to make sure more perspectives are seen on this:

In a project I work with regularly, people often come to our help channel with a problem, and someone more familiar with it than I, responds: “Did you install Python from the Windows Store?” And they say “Yes”. “Uninstall that and get the installer from python.org”
And they do that and their problem is solved.

This is something I’ve seen over and over and over again.
So we shouldn’t take it on one person’s reporting that the python.org installer is less reliable.

(I also don’t think you should take my reporting as reliability information. It’s just a reminder that there are more perspectives.)

6 Likes
74

That “one person” has (controlled) access to all of Microsoft’s telemetry for every Windows machine in the world[1], and is the first point of contact for installer related issues reported to CPython. I’m not exactly just chiming in as a one-off user :wink:

Could you characterise the problem? In my (fairly extensive) experience, these problems are typically the project doing something that isn’t compatible, isn’t a good practice, or makes specific assumptions about the user’s environment that nobody is guaranteeing. We aren’t obliged to modify Python to maintain these problems, and users aren’t obliged to modify their system in order to maintain these problems.

I’m well aware. With the detail you’ve provided here, trust me that your perspective has been fully taken into account. It’s one that I’ve heard many times over the last decade.

But now that you’re here, I’d love to hear if you have any additional detail that might be new to me. For example, if the type of problems that are being solved by switching installer are one that I haven’t heard about before, then that would be very interesting.

  1. If they didn’t disable it. ↩︎

1 Like
75

Just want to add some info related to this topic.

We’ve started to use “pre-commit” recently in our developer team. Nothing special on macOS. Then some windows users reported

“unable to create process using py.exe -3”

, after some investigation, I found it’s because using windows store version, which does not ship “py.exe” by design: What is py.exe?

Re-install the official site version fixed this, and it’s clearly to see “py.exe” listed on the installer UI.

After checking the post(which I got during searching for what is “py.exe” and what does it do), I think you’re now going to favor msstore Python over the official site distro. Which I have no problem with, I myself will always use msstore version if there is, like PowerShell, autoupdate make it so convenient.

But, it seems some python feature needs to be improved together with your change proposal, I mean “zipapp”, checking the section of “#specifying-the-interpreter” in its doc.

Simply download Release pre-commit v4.0.1 · pre-commit/pre-commit · GitHub and change .pyz to .zip then extract.
when opening the python.exe with notepad(it’s my newly installed macOS VM, not working PC, so tools are limited), you see clearly where this “py.exe -3” coming from:

image
image1148×1115 47 KB

76

For anyone arriving late to this thread, the discussion is continuing at Pre-PEP: A Python Installation Manager for Windows with a more fleshed out proposal.

1 Like