Pre-PEP: A Python Installation Manager for Windows

Ideas
41

Could pymanage additionally offer a GUI?
E.g. Gooey aspires to “Turn (almost) any Python 3 Console Program into a GUI application with one line”.
Could the store’s start button then start that GUI app?

42

Oh no, in my case it’s definitely a mess! :laughing:

For contrast I rarely get to use py. I’m constantly switching back and forth between Windows, MacOS and Ubuntu so in this case py isn’t part of a standard install. Sometimes I’m on Windows and remember it exists.

I had settled on using pyenv and pyenv-win to manage my Python installs until uv came along and largely replaced those almost entirely due to providing easily installable Python binaries without needing to wait 10+ minutes for them to compile[1].

I do still use pyenv to manage my basic python command and to install/build prereleases. I’m fairly sure the Windows port uses the python.org installers so if those were to be replaced I imagine pyenv-win would eventually have to switch to using whatever bundles this manager uses.

That’s mostly my point. People will try to do this and get annoyed when it doesn’t work. I think it would be better for users if instead they learn to use pymanager install which wouldn’t have this issue. I think it’s just going to be more confusing if you can access the same commands in multiple different places.

My hope would be that pymanager install... would register its managed installs in the Windows registry so the existing py launcher (or a clone with the same behaviour) would continue to work as-is alongside these new installs, or be updated to recognise them if that’s not possible for some reason[2].

So the workflow change would be instead of going to python.org or the Windows store to install or update Python you’d use pymanager install 3.14 which you could then run with python3.14.exe ... if you had added the folder to PATH or py -3.14 ... if you had not.

  1. On Linux. The other speed-ups are nice, but wouldn’t have made me switch. ↩︎

  2. I thought about writing a script to register uv python installs, but with the rate they’re going I assume they’ll get around to that at some point if they haven’t already. ↩︎

43

I’d have to test things, but it would be reasonable to show sys.argv[0] as “the thing the user typed to launch pip”. I think for a few older versions, possibly all of them, that may not be correct, and of course if people launched via pip.exe then you don’t want to show that.

Ultimately, in my opinion, things other than Python should just say “run python ... and see (core docs) to figure out if you need to replace python with something else” (but worded better).

Potentially, one day, if someone builds one.

It does. I didn’t demo it (because I don’t have the existing launcher installed). PyManager also supports the -0/-0p arguments from the py launcher, so Paul’s muscle memory can relax.

Nobody has complained about it yet, but you could also just pymanager -3.14 ... and it would install and run it if needed (currently you could also python -3.14 or py -3.14 and exactly the same thing would occur, but those are controversial).

44

Thanks for the demo video, it’s looking very nice!

I like that, once the installer is available, you can just type python and it will run it. And it’s good you get the aliases (after updating the PATH).

It makes me want a similar installer for macOS and Linux builds too. And if so, we should design the interface to work the same (or as similar as possible) across all of them, so we can have a single set of instructions for people, whether this is via python or pymanager or another name.

Along with handy tools like @glyph’s MOPUp macOS updater and Astral’s uv install python, it would be great to make similar things available for official installs.

4 Likes
45

That doesn’t work in the docs. And I think we tried that and it didn’t work in runtime messages either :slightly_frowning_face: The point is, people want to be told precisely what to type and have it work. That’s an unachievable ideal, of course, but py -m pip ... has worked in the majority of cases for years now.

What’s important is that this needs to be covered in “How to teach this”. Because it’s the biggest compatibility problem in the whole proposal, and at the moment it feels like there’s little being done about it beyond “people will get used to it”.

I’m harping on about this because everything else about the proposal looks good to me, in case it’s not obvious. In particular, I agree with @hugovk’s point that we should be looking at this as (in the long term) being the interface that we’ll provide on all platforms, and not just have it be a Windows thing.

5 Likes
46

Aaah, Python 3.13 from pymanager didn’t show up for me because I already had it in the registry from the regular python.org installer, which leads to it not being added when pymanager installs it, which leads to this discrepency between your py.exe and the regular py.exe

From PyManager:

C:\Users\ducks\AppData\Local\Microsoft\WindowsApps\py.exe --list
 -V:3.13.1 *      Python 3.13.1
 -V:3.13          Python 3.13 (64-bit)
 -V:3.10.11       Python 3.10.11

From the current installer:

λ c:\Users\ducks\AppData\Local\Programs\Python\Launcher\py.exe --list
 -V:3.13 *        Python 3.13 (64-bit)
 -V:3.10          Python 3.10.11

Yes I did notice this, I think it’s actually good for py to have this in the same way uv will automatically install python if you request a version you don’t currently have. It should probably be defeatable in a similar way to how you can prevent uv from automatically installing python and require the use of the install command directly.

1 Like
47

Thanks for the shout-out! MOPUp is an incredibly simple script; more of an encoded social convention than any meaningful automation. It didn’t seem appropriate to contribute. However, if there is interest in doing something like this, I’d be happy to help out with some of the work of getting it included.

1 Like
48

Then it couldn’t be python install 3.15. A new command like pymanager install could work or perhaps even an adaptation of the existing py could work but not python install. Having the basic python executable on PATH be an install manager would not work on MacOS, Linux or other unices.

49

Why not? I don’t see why it’s any harder than it is on Windows. Although I’d be fine if pymanager was the generic interface, and then what we do with python and py can be something we decide later…

50

It seems we’re deciding now that it can’t be anything other than any of the existing 4-5 things they may already be.

I’m proposing that we decide to align on a single command that does something useful, and that’s being rejected on the basis that it’s too hard to change. It’s never going to get easier. Deferring a decision is basically deciding on status quo forever…

1 Like
51

The difference is that Windows does not have a system Python but actually having checked the available OSes I see that they have all migrated to python3 so the name python is potentially available for reclamation. Except that python is still the command to run a program in a venv.

It might make sense when considering only Windows to want to reuse the same python.exe as both an install manager and the command to run things but only because it is hard to add another .exe that could hold the same privileged position as the one that is already there. If there were any plan to redesign this basic CLI interface for installing and running Python on all platforms then we would definitely not want to use the same executable name for the installation manager as the command to run things. The basic reason is that you want the installation manager to be accessible from a venv but always live outside any venv (much like uv is/does).

If the goal is to reunify all of this then that is great but it would be a huge shame in that case to make something now that works for Windows and then find that it does not end up being adopted across the other platforms. The py launcher was a great idea but it was presented for Windows and then remained a Windows thing even though it could just as easily have been something cross platform. If we want something that is going to be adopted across all platforms then it needs to be presented and evaluated as such rather than being presented as part of a Windows-only improvement.

52

I mean, there’s no real reason that PyManager couldn’t detect a virtual environment just from the environment variable, or that the executable put in the venv couldn’t recognise the subcommands/options and pass them to PyManager instead of the originally associated Python. It would just be a change to core, and wouldn’t be backported to existing releases (so one of these “after 5 years we won’t even remember it” problems).

53

It’s just all backwards to do it this way. Do we expect that a uv-created venv would have a python that passes these commands back to uv?

I do really like the idea of reunifying the “commands that do Python stuff” but I think if that is a goal then it should be approached from a cross-platform perspective. Anyone who is interested in that goal should be concerned about the possibility that it ends up misdesigned as a Windows-only feature rather than being designed for adoption on all platforms from the outset.

54

I already tried and failed to advocate for essentially dropping the intermediate executable/link in a venv (see PEP 582). People like there being a separate venv command that has the same name :man_shrugging: I’m just trying to accommodate them.

I’m open to feedback on the design if it looks too Windows-centric, but I’m not going to expand the scope on this. The most important things for me are to stop having to publish new apps each year for each new Python version, and to avoid churning Windows users more than once. I’ve been resisting otherwise reasonable requests for things like adding python3.exe because while it helps some cases, it churns many more, and it doesn’t actually fix things enough that we can promise no more churn (for a while).

I believe this proposal will not need to churn for quite a while. Yes, it’ll take some adapting, but we literally can’t improve anything about installation without causing that. People still complain that the latest installer blocks Windows 7! So this is one churn, with a 2 year deprecation period where the old installers are still available, and then we live with it for the next 10+ years. I’d really like to use the churn to get out of a lot of our current mess, rather than to ingrain it further.

55

I understand that you don’t want to expand the scope and I don’t think that is unreasonable. I just want it to be clear to anyone else that if their goal is to make something that becomes the standard way that things work across all platforms then this is not the way to go about it. I am sure that this proposal will lead to many improvements on Windows but I don’t expect that the design here will be adopted on other platforms (much like the py launcher).

1 Like
56

(I chucked a big chunk of this post, because it was unmanagably long. It’s still too long, but I don’t have time to condense it any further, so my apologies).

What happened to backward compatibility? Why can’t we improve things the way we do in all other areas of Python, by carefully balancing compatibility against benefits? Your arguments feel like you’re saying we have to do something incompatible, so all bets are off and we might as well take the hit and redesign things completely. That’s the sort of Python 2 → Python 3 transition that we agreed we wanted to avoid in the future.

I’d like to see the individual aspects of this proposal justified in terms of the benefits they provide being weighed against the breakage they cause. Not all lumped together in a “we’re breaking things, so let’s get it all over in one go” all-or-nothing approach. As I see it, we have:

  1. Adding a new Python manager (the pymanage command). This seems like a pure win. There’s essentially no backward compatibility issue, and significant benefits.
  2. Retiring the py launcher, or reimplementing it. Reimplementing it is fine (and essentially transparent) as long as there’s no functionality change - but feels pointless. Changing functionality feels like gratuitous breakage to me, and I’m unable to understand what benefit there would be, beyond simple things like deprecating features that we consider little-used and low-impact. Retiring could be a significant breakage for some cases/workflows. Even so, I’m sort of OK with doing it, as long as there are clear benefits documented in the PEP. And I think it needs a better transition plan than the proposal currently gives.
  3. Refocusing on python as the “correct” way of running the interpreter on Windows, rather than py. To an extent, this has been imposed on us by the fact that the existing Store Python doesn’t include the launcher. This is still a compatibility breakage (see my earlier post on how it breaks the pip documentation) and the benefit seems somewhat vague, as it’s more of an after-the-fact acknowledgement that we want to retire the launcher than a positive choice. But I think it needs to be a definite policy choice, as there’s work that needs to be done to undo the messaging that we’ve been promoting for years that the py launcher is the correct way to run Python on Windows. And the PEP needs to document how that process will happen - it’s not going to be a smooth transition if we just leave it to chance. Maybe the best way to do this would be with a separate informational PEP, much like PEP 394. In fact, maybe we should include Unix systems in the same PEP, reclaiming the python name as the canonical way to invoke the main system Python on all platforms.
  4. Building manager functionality into the python command. This is the place where there’s a significant level of disagreement. IMO, it would be far cleaner not to do this, and I honestly don’t understand the arguments around why this is needed. The “user story” of a new user typing “python” at the command line doesn’t feel convincing to me - if the user is comfortable using the command line to invoke Python, why is it so hard to understand that the first invocation triggers the installer, and once that installer has run, python will run the installed version and pymanage is how you manage the installed interpreters? And if there’s technical issues around how MSIX installers and/or the store work, then they haven’t been explained clearly enough, as I can’t see why that wouldn’t work in a practical sense.

I don’t think anyone is saying that “nothing must change”. In fact, my sense is that most people are pretty enthusiastic about the new functionality, with the sole exception of the decision to include pymanage functionality into the main python command (and the weird architectural consequences of that - python actually runs pymanage, which in turn runs the actual interpreter).

One final technical point. Removing pymanage functionality from python would require sacrificing the ability to do python -3.12. That’s a bit of a shame - I like the current option of running py -3.12 - but realistically, if we’re going back to “python is the official command” rather than the launcher, isn’t the correct way of doing this just to invoke python3.12 anyway? Or, to put this another way, I don’t see python -3.12 as a compelling reason for having pymanage functionality in the python command.

6 Likes
57

We balance compatibility on our supported scenarios, but as soon as packaging/distribution/installation gets involved, we have hundreds of technically unsupported scenarios. Many of these are directly contradictory, and therefore something has to break, either by design when we do it, or when the user causes it.

We don’t support “user has modified their own PATH”, for example, and so if we want a change that impacts users who have modified their PATH manually, we aren’t compelled to treat it as a compatibility issue.[1]

So yes, we might as well take the hit and redesign things. Of the items you list, you’ve missed all the little changes that if we made them on their own would constitute a breaking change (e.g. adding python3.exe, or fixing service account installs, or changing the order of py.exe’s results, or adding installs to PATH by default, or making free-threaded binaries install properly). But I didn’t list them in the PEP either, because they’re rationalisations, not reasons.

For the four things you do list (which I appreciate - most people can’t so clearly articulate the things they care about):

  • Yes a manager is great, but anyone can do it. When I shared my prototype at work, the most common response was “just tell people to use uv”. If we’re going to do something in core, then we should do something that only core can do. Otherwise we commit scope creep on behalf of the entire core team, adding something for them to maintain without taking anything else away.
  • I don’t particularly care to do anything much with the py launcher. It can spin out and be a standalone project just fine, it’s already basically independent from CPython and doesn’t need to be part of regular releases. If we release it on its own, then it’s available. The question really is, given PyManager, should/why should users also use py?[2]
  • Refocusing on python is the thing that only core can do. If we want to encourage all Python users to use a different command, we should spell it uv and save ourselves a ton of work :wink: But the more serious point is that the lack of focus already exists. You keep referring to pip’s documentation, but go check out the range of other options already used in other projects - there is no consistency, and we don’t help achieve it by offering so many subtly incompatible choices. Narrowing it down to one requires transition time, but ends up with much better consistency.
  • Going back to all those user surveys, you’ll see that most users expected it all to be under a single command. So you may say it feels cleaner, but the broader community disagrees. Again, plenty of people are already happy to spell “python.exe” as uv run, so why should we confuse them with multiple commands and having to decide when to use each?

Not necessarily. It would be easy to check the executable name and then print an error only for subcommands, but let the -3/-V: options through. If you’re running an unspecified python (you wouldn’t explicitly specify PyManager’s python, because it isn’t part of an actual runtime - it is never returned for sys.executable, for example) then you might even be okay with the shebang handling as well.

But it’s not a technical point - it’s a policy point. If python.exe on Windows is only permitted (by us) to directly launch Py_Main(argc, argv) with no other functionality, well, that’s already broken, but it’s also purely about policy, which is what I’ll happily tell people when they ask why the proposal didn’t work out. (And to be clear, this is a big “if”. So far, only MAL has seemed to speak on behalf of the core dev POV in this thread, though past discussions have indeed shown that the core team are against subcommands on python, though to be honest it felt more like an “if you have to ask, just don’t do it” discussion rather than a thoughtful dive into user experience).

  1. Between major versions, and ignoring other reasons it may be a compatibility change e.g. changing the runtime’s installed layout, which is only permitted during alphas. ↩︎

  2. Slight aside: changing py’s installation mechanics to work with PyManager is also a breaking change. We can’t just ship it as-is, users will be broken unless they take action or until they come to a clean machine. ↩︎

1 Like
58

I wish this point was given more consideration, as it’s easy to see that this (coupled with an obligatory link to that xkcd strip about standards) will be the most common response in the wider community too. Didn’t the survey a year ago show that the community wants “one tool to rule them all”? I understand there’s a million obstacles on that road but if there’s an opportunity to “take the hit and redesign things”, shipping Python with uv as an installation manager (on top of everything else it so painlessly does) would be IMO a much more exciting development.

3 Likes
59

This looks cool and I’m thrilled to see development in this direction! That said, I share some of the concerns expressed by others.

Overall I like this because it’s more towards a manager-first approach. But it’s not totally that because the manager is still python, at least sort of. I get that that’s because the main goal is not the manager-first-ness, but rather to mitigate the awkwardness associated with installing Python on Windows specifically. But I think it would be a shame if only Windows got a delicious manager, and also I think making the UX this different between Windows and other platforms might create additional confusion that would make it harder to get users on board with an official cross-platform manager later.

I know you mentioned at some point some constraints on what can be done with the store and the python shim that launches it and so on, but I can’t find them now. My first thought is that the flow I would vote for is something like:

  1. Typing python gives something like Try typing pymanager instead.
  2. Pymanager is a shim that, if the real Pymanager isn’t installed, installs it.
  3. The user interacts only with Pymanager to install and manage Python versions (and ideally libraries as well).
  4. The user in general doesn’t type python except after some kind of activation command, or after explicitly setting up some kind of auto-activated environment.

Probably this isn’t possible. But I think trying to stuff everything into python.exe to provide all the functionality is going to have too many negative side effects.

Also, I’m a bit leery of bending over too far to support Microsoft’s quirks about stuff like providing the Python shim when Python isn’t installed. What happens if they decide to get rid of that later, or add additional shims whose existence confuses users, or loosen the restrictions on what the shims can do such that some of these gymnastics become unnecessary?

I think one of the most useful things core can do is just authoritatively tell people not to use the plain core Python. :slight_smile: If that means just having the shim say “Install uv instead”, then okay.[1]

They, do but that doesn’t mean the command has to be spelled python. What I’m worried about is confusion between what arguments and options wind up getting passed to Python vs. to the manager.

Okay, but if they’re happy to spell it that way, then why not lean into that? The thing is that with uv, you can tell that uv --something blah is passing something and blah to uv, whereas uv run --something blah is passing that stuff to uv run. But if python --help is giving you the Python help and python help is giving you the manager help, that distinction is not so clear.[2] Because the Python executable already has a decent-sized surface area of command-line usages (which are widely relied on), it’s hard to squeeze in a parallel set of subcommands and such there without creating some ambiguity. It’s easier to do it by having the new manager stuff on the outside and leaving the “real” Python as the subcommand that can take all the arguments we know and love.

So in some sense the thing I think most needs to change is the idea that the default entrypoint people use to interact with Python is typing python, which is the thing you were at pains to preserve. :sweat_smile: So if in the demo all the places where you actually ran Python were spelled pymanager run ... and the ones where you were using manager commands were pymanager ... then I’d be like, yes, ship it now!

But, like you say, at that point it becomes hard to distinguish from existing tools. So I wonder if it wouldn’t just be better to save the trouble and just straight up tell people to use uv or conda.

  1. Obviously I’d have it say “Install mamba instead” but just a clear consistent message would be an improvement. ↩︎

  2. In the demo, for instance, alongside the subcommands it lists a -v verbose option — but “plain vanilla” Python already has a -v option. So is this another option with the same name? If I do python -v is that the manager’s -v or Python’s -v? ↩︎

1 Like
60

I’ll bundle up all the comments about uv (and conda, and yes, conda is effectively equivalent to uv here) and reply to them all at once.

uv and conda are downstream of us (we are upstream), and they are non-authoritative for anyone’s system (unlike your OS provider). We don’t control them, and they aren’t responsible for coherence among all the software on their system. So the proposal is for upstream to stop making binary releases and let other people do them, and we wasah our hands and say “we hope these guys don’t charge you/collapse/mess up”.

I’m not entirely against that - Linux has been working fine on a sources-only basis forever - but if you want to use compatibility as any sort of reason at all then this is a much more disruptive route.

Additionally, uv is maintained by a for-profit company, and conda used to be, which apparently python.org are not allowed to endorse. We even had the link to our own Python app in the Windows Store taken down because it was apparently too strong an endorsement (which I opposed, but lost). These are legal arguments, to be had amongst the PSF’s officers/counsel, but the current precedent is “don’t recommend any businesses”, which makes it impossible for us to point at uv, even if it were technically responsible.

Why isn’t “installing an app from the Store” a good enough activation command?

Ultimately what isn’t possible is the shim. Python is the only one that’ll get to be in the OS, and eventually it’ll probably get taken away (it’ll definitely get taken away if it stops doing numbers or we ask to have it taken away). New shims are going into a PowerToys Command Not Found module, which requires users to have installed an extra product, upgraded to non-standard PowerShell, and activated the module, and even then it has to come through WinGet who decide their own installs and don’t particularly care to listen to us about using our own installer.

So basically, the users we care about are not getting another pre-installed shim, no doubt in large part because of the same behaviour changes we saw for Python, and ours would frankly be worse if we were moved into the replacement.

So the questions on behaviour I’d like to see actual consensus on:

  • after installing PyManager, should python install and launch Python on first run? Or should it give an(other) error pointing to pymanager and/or activating a venv?
  • should running pymanager install and launch Python on first run? Or should it give an(other) error telling users what all the commands are?
  • If we decide python points to the other command, then after installing a global runtime, should PyManager’s python launch that runtime? Or should it point to pymanager/venv still? (to encourage users to move to a single command unless they’ve explicitly activated)
  • If we decide python launches global runtimes, should it support -V: and shebang processing for users to select? Or should we only allow those on pymanager?
  • If we decide python can have version selection, should it also support auto-install? Or should we error in that case and require manually using pymanager?

Whatever set of answers you choose here will directly impact the experience of every user, largely in terms of the number of failed commands they’ll run. If you’re taking this seriously (you don’t have to, but I am), work through your own proposed scenario and see whether you feel successful or whether you feel beaten up until you’ve done exactly what the tool wants you to do.

2 Likes