Pre-PEP: A Python Installation Manager for Windows

Ideas
1

Following on from the discussion on Windows Store package to become installer/launcher?, I have written up my proposal more completely in PEP format and built a proof-of-concept.

Draft text: pymanager/pep-draft.md at pep · zooba/pymanager · GitHub

Diff since the original post: Comparing post-1..pep · zooba/pymanager · GitHub

Prototype implementation: GitHub - zooba/pymanager: Draft repository for a Python install manager app

Pre-built prototype: https://cpythonpackages.blob.core.windows.net/pkg/python-manager-0.1a2.msix

And to be clear, this is not ready to release, and already lags slightly behind the PEP text. It’s here for interest - let’s discuss the text, please. Still not ready for release, but the build feels pretty good to me, so feel free to try it and provide feedback.

Also, the text is still lacking cross-references, many citations[1], external links[2] and the like. I’ll get to those, but I think there’s enough there to be worth discussing even without camera-ready text.

I have some idea of what the major controversies will be, but please, raise any concerns or suggestions you have :slight_smile: I’ll read it all, and I look forward to finding out what others have seen that I’ve missed.

  1. As I’ve tried to be clear, most of the evidence is anecdotal and/or private. If/when I get time I’ll go and search the issue tracker for examples of the issues, but I simply can’t link to or quote every in-person discussion I’ve had about installers over the last ten years. I’ve tried to be fair about that in the text. ↩︎

  2. Stuff like when I mention uv but haven’t linked anywhere. Because it’s really hard to find without a convenient link :wink: ↩︎

5 Likes
2

OK, so I’ve read the proposal. It looks great, overall. I’ve collected my thoughts here - sorry about the length, I thought one post covering everything would be better than a bunch of individual comments.

In the “Background” section, there’s a discussion of intended uses of the various packages. While that’s great, I think we should also discuss (to the extent that we can) the actual use. In particular, my experience is that many people use the embedded distribution as a standalone distribution, and struggle because its design doesn’t lend itself to that usage (specifically, the fact that it’s hard to use pip to manage packages). My belief is that this is either because the nuget distribution is significantly less discoverable, or because users aren’t comfortable relying on the nuget package manager.

My hope is that the proposed installation manager will address these sorts of misunderstanding, but I think we need to discuss them in any case, or the PEP looks like it’s been designed based on what we wish was the current situation, rather than on what that situation actually is.

Under the install subcommand, the PEP says:

Passing --upgrade with no tags will attempt to replace all installs with newer compatible versions, though as this may be destructive we reserve the right to disable this command and require tags be listed explicitly.

What does that mean exactly? When and how would we disable the command? And who is “we” in this context? IMO, the PEP should pick one option. Changing the option would probably require a PEP anyway, and such a PEP is no harder to write if the initial spec made a definite choice. (If the intention here is to say that we could change the behaviour without requiring a follow-up PEP, then my initial reaction is to disagree, but if that’s what you want you should say so explicitly, and justify that statement).

You say that python install --target is intended to cover embedding cases. Does that mean it will disable pip the way the embedded distribution does, or will it be a fully functional installation like the nuget one? IMO people will expect it to be fully functional, and having to hack the _python*.pth file to enable pip would be surprising (as would the “flat” directory structure).

If you want an option to install an “embedded” distribution, I’d suggest being explicit and using --embed, reserving --target for a “standalone” but otherwise normal distribution.

I’m inclined to say that we should support using the distributions outside of PyManager. We can add provisos (such as we expect any reported problems to be reproducible using python install --target), but IMO the ambiguity involved in “we don’t support this but you’re welcome to use it” will discourage use, leading to unnecessary duplication of effort, as well as lack of clarity for package maintainers. For example, if someone reports a problem with pip, when used in a Python installation that was downloaded from python.org by hatch, what do we do? Telling them to reproduce the issue with the official python.org build isn’t reasonable, as this is an official build. Telling them to talk to hatch isn’t much help - what could hatch do if the core devs won’t support them?

In “Interaction with venv”:

Based on experience with the py.exe launcher, where it was found that users expected py to launch an active virtual environment, we would argue that PyManager as proposed only behaves correctly because it uses python as the main alias.

I’m not convinced the behaviour is correct. If a user types python help pathlib when the system Python is active, they get the help for pathlib. Why wouldn’t they expect that command to work even when a virtualenv is activated? The technical reasons why this won’t happen are fine, and there’s no point repeating them, but technical reasons don’t alter what people expect, however much we might like them to. Maybe we’ll have to say “tough, it’s not going to be possible to change this”, but we shouldn’t ignore the fact that people won’t like it.

The “Environment Variables” section discusses entry point executables (like nox.exe) installed by pip. I don’t completely follow that discussion. My understanding is that the “Scripts” directory (where those executables are installed) will be distinct from the location of python*.exe. I don’t see this as an issue, as that’s the case currently. Equally, while it’s annoying that we still can’t solve the problem of that directory being on PATH by default, I understand the technical reasons behind it. It is a regression for users who currently set “Add Python to PATH”, though, and should be explicitly noted as such. Personally, I use pipx and virtual environments, so I don’t care much myself, but I know users who would be frustrated if they installed nox and then found that the nox command didn’t work for them.

(I didn’t read the “configuration” and “index schema” sections too closely - they look comprehensive and I trust they will be reviewed in more detail as we get down to more low-level matters).

I’m sad that script dependencies won’t be supported, but I agree it’s a significant chunk of extra functionality, and too much to include right now.

I’m concerned about the shebang processing. There are various launchers with different rules for shebang handling (in particular py and the wrapper used to create entry point executables). While consistency isn’t necessary, gratuitous inconsistencies will cause confusion and make it harder to document behaviours. In particular, the form #!C:\Full\Path\To\python.exe is a form that’s used in various places and not supporting it will likely cause problems, especially for people who have traditionally used the py launcher and are switching to the new python.exe.

Speaking of which, what will be the fate of the py launcher if this proposal is accepted? There is a lot of documentation that tells people to use py on Windows (the pip docs being one). If the launcher is no longer available in the official Python installer, those docs will need to be changed again :slightly_frowning_face: Furthermore, people will have scripts and utilities that use py as the Python command. Yes, with the existence of the store Python, you can’t just assume that py is present, but there’s a general acceptance (in my experience) that the Store version of Python has some “quirks”, and this is easily accepted as one of them. We’re now promoting the Store Python experience as the official behaviour, and I think that’s going to be a big change for a lot of people.

You do mention this in “Backward Compatibility”:

Users who currently rely on the traditional installer will find themselves having to switch to a significantly different workflow. This will particularly impact those who have scripted downloads and installs. The deprecation period of two releases will allow time to transition, and the traditional installer will have additional output and warnings added to direct users to the newer options.

but frankly, I think that even with the deprecation period, you’re being very optimistic if you think this will be easy. People who support multiple Python versions in particular will have a lot of migration work to do. And changes to workflows and basic commands (like py) are the hardest things to change in larger organisations. I’m surprised you’re not more concerned about this yourself - do your users include an unusually high proportion of people using the store Python? In my experience, it’s still currently a minority of users who use it.

“How to teach this” should probably include a discussion of how to teach this to users who are currently using the python.org installer and the py launcher. The current section is focused only on teaching the new behaviour to someone unfamiliar with Python, which is unlikely to be the biggest affected group. The discussion should also cover how we counter misinformation resulting from people searching the web and finding discussions on the py launcher and other information relating to the current installer.

Overall, if this proposal was working from a clean slate, with noexisting installation mechanism, I’d enthusiastically support it. However, we don’t have the luxury of a clean slate, and I’m concerned that the transition will not be as smooth as the PEP suggests. In particular, I’m closely involved with the packaging ecosystem, and we have historically received a lot of complaints about disruptive changes and lack of stability. Whether we like it or not, this change will be seen as yet another disruptive packaging change[1], and as such will dig deeply into our already-overspent churn budget. It would be immensely helpful if this PEP acknowledged that issue and did more to help mitigate it.

  1. and from bitter experience, we know that pointing out that this was done by the core devs, not the packaging community, won’t help in the slightest :slightly_frowning_face: ↩︎

3 Likes
3

I’ve been thinking about the subcommand problem, and in particular whether the main interpreter could reasonably do something with those subcommands when they don’t correspond to a script in the current working directory.

What I came up with is the following:

  • add a new stdlib module called python
  • the python -m python CLI implements the subcommands described in this PEP (but disallows making any changes to the currently running interpreter)
  • the main interpreter binary delegates the known set of subcommands to that module (similar to the way it hands off non-trivial command invocations to runpy) if they don’t correspond to a file in the current working directory

Guidance to Linux distros would be for them to patch the new stdlib module if they felt the need (although we’d encourage them not to break it completely, just as we encourage them not to break ensurepip and venv)

4

Thanks for the detailed feedback (and yes, one big post is a great place to start). I’ve added a decent amount of text as a result, so I won’t respond to each individual point. Diff since the original post (also added to OP)

Agreed, and added (though brief - I want to avoid too much speculation in those sections).

Fully functional. All it’ll do is extract the ZIP somewhere else and skip generating any registry keys/aliases specified for the package. You won’t see it in python list and can’t remove it with python uninstall.

The embedded distro can get its own tags in the index. I’ve already tested this and it’s great.

The distros are as supported as any after install. What’s not supported is using our index.json to locate them. It’s going to be hard enough to make any changes to the index (location/format/etc.) as it is, and we don’t need to complicate that further by making it an official public API (or take on the burden of uptime, etc.).

But once you’ve got the direct URL to the ZIP, it’s as supported as ever, which usually means we’ll send your questions back to the tool that you installed it with, or ask you to confirm that your issue occurs when you use our installer.

I’m actually backing off on the help subcommand, after implementing it and not really feeling it. I’d probably use it, because I go to our docs first, but I suspect AI helpers are going to win this battle.

I noted that expectations may change, which would lead to stdlib changes in the venv module. There’s nothing we can do from the PyManager side.

It turns out that so is requires-python, because I’ve not even considered installing from version ranges :man_facepalming: (which don’t actually have any place in the Company/Tag structure). I’ll have to think about it - I suspect we’ll probably end up at “if you specify a range, you’ll only ever get PythonCore runtimes”.

My next best offer is to not support it at all! It’s the bane of my existence in this area, and if there isn’t a relatively simple rule to make it work (safely and securely) within the system, I’d prefer to tell people to get another tool that processes them. The standard approach to this on Windows is to create an adjacent .bat or .cmd file with the same name, and then rely on PATHEXT choosing those by default rather than your .py file.

I’m still thinking about this one. We could expose py as another alias of PyManager just fine, the problem is that if you ever install the old launcher, it’ll always shadow - it’s PATH setup intends to bring it to the front. And I think that shadowing is going to be very annoying during the transition.

I definitely don’t want to add all the PyManager commands into py. Though runtimes installed by PyManager will set up the registry keys needed for py (and other tools) to find them, so the old py will still work for launching runtimes. Perhaps that’s enough?

I wouldn’t be opposed to spinning the current launcher out into its own repo/release (hopefully with its own maintainers…). I don’t want to replicate all of its shebang handling, or the venv detection logic, but others may want to keep that. It also wouldn’t prevent us from providing py as a PyManager alias.

Just for clarity, the experience/behaviour here is just around py.exe not being available? Distributing PyManager through the Store doesn’t actually imbue the installed runtimes with those quirks (which I noted at one point, because some modern OS features rely on those “quirks”). In particular, the sandboxing and unmodifiable installs shouldn’t apply to the runtimes themselves.

We can make as many older versions available as we like. I’ve got a script to generate an index from our Nuget packages, which gets me all the way back to 3.5 (though without tkinter or the test suite in any of them), and it’s not impossible to recreate the packages for them. Most of the packages are still on our Azure Pipelines instance from when we built them.

No, I have an unusually high proportion of people who just use whatever Python responds to python[3] on the image they’re using (or they’re using Anaconda) :slight_smile: From a $work POV, I’m likely to try and roll this out regardless, with a custom index.json that pulls from our internal builds instead of public ones. I’m not in any position to enforce certain workflows on our teams, but given the choice between the hacked together Powershell scripts and a py-manage install ... command, I’m confident I know which way they’ll go.

I’m still thinking hard on this section. Your thoughts are very valuable, but I have nothing extra added to the text right now (it’ll also be influenced by earlier decisions).

You’ll be very glad I’ve been resisting all sorts of changes for the last few years on this basis then! We could have had a complete transitional mess if everything had just gone through.

I’ll try and sound less optimistic about the transition period, though I think it’s important to remember that nobody on Windows gets automatic updates between Python releases. Anyone currently using an install of Python isn’t at all forced into the new model, they must choose to do it. (That may not be true once they’re in the new model, which is why I’ve wanted to make sure the design is solid before making any changes at all to installation.)

Any proposals for things that may help mitigate the impact (or perhaps you meant the perception of churn)?

5

My view on the subcommand issue isn’t really changed by the discussion in the ‘Rejected Ideas’ section.

My expectation (hope?) would be that the python.exe would essentially delegate to a subcommand of pymanager similar to how the python command if you’re using pyenv delegates to pyenv exec python[1] and lets pyenv figure out what it actually needs to launch.

The default behaviour to install/launch the default/latest Python would remain so there isn’t the issue of there being no python.exe command, you just wouldn’t access the install/uninstall/list subcommands via the python executable.

Part of the reason for this is so users don’t get frustrated by having install/uninstall/list suddenly stop working when they’ve activated a venv, but it also allows for adding other useful subcommands without fear of clashing with things users might expect to work with python.exe.

Similarly I’d expect invoking pymanager.exe with no arguments[2] to give information on the available subcommands and not launch Python itself. This gives another way to find out what the commands are, but also I just wouldn’t expect launching the manager to launch Python - I don’t see a reason for it to do that outside of a subcommand.

  1. And all the other managed commands: python3, python313, pypy etc delegate similarly. ↩︎

  2. Or with a --help option or help subcommand. ↩︎

1 Like
6

I assume you mean “when in a venv”, and python.exe is the redirector (which looks for a pyvenv.cfg file to decide where the real python.exe is), as opposed to the PyManager alias, which is the same executable as pymanager.exe just with an alternate name?

Yeah, this is a real risk. But it has to be balanced with users who want them to stop working in a venv. Letting them stop working is easier for compatibility, and as yet I don’t think there’s any other evidence to strengthen either side.

Things would be drastically more complicated if the install subcommand here worked on packages. But it doesn’t. So “once you’ve activated your venv you have indicated that you don’t intend to install/choose another Python runtime” seems like an explanation we can run with.

Yeah, I’ve been thinking about this one. I think it makes sense for that particular alias, but not for any of the others (including the “launch without a console” aliases). This might be easy enough to do by checking argv[0]. I don’t really want to have too many almost-equivalent-but-not-quite executables being built - pymanager is really just there to bypass an active virtual environment (or other PATH modifications), I really don’t expect it to become a common command.

7

No I mean that when not in a venv.

The python.exe would function roughly as you have described but without access to install/list/uninstall subcommands which should be handled by using pymanager directly meaning there’s no change in how they are used whether in or out of a venv and there’s also less chance for surprise that the commands don’t work on other distributions of Python.

I was trying to suggest a way of doing this would be to move the launching of python from pymanager.exe to a subcommand and have python.exe launch that instead of being a simple alias.

[Edit: Better phrasing]

8

Ah, I see what you mean. There are technical reasons to minimise the “distance” between what the user runs and what process ultimately launches, certainly in “number of intermediate processes launched”, but it’s no doubt possible to have the same executable reject certain commands based on how it was launched.

That would, however, be admitting defeat for the long term, and so the python command would always be the special case and not the default one (despite being the common way to get the manager in the first place). It might make the initial transition easier, but I think we’re in a better place to have one obvious command at the end of it.

In past discussions, the behaviour of the python command was at least as controversial as the presence of the subcommands. And I really don’t think there’s a way to improve the user experience without giving the command the ability to install a runtime. Once that’s in there, taking away the ability to configure the install seems wrong.

Also, it’s future (potential) work, but I’d kinda like a run subcommand to be able to implement any/all standardised script metadata. That’s not really appropriate for the normal python command, which is really just trying to bypass PATH and find the runtime you meant.

9

Ah, OK. That’s something that the PEP should make clearer. Although I don’t really understand your reservation here, and frankly I think it’s a mistake to try to keep this private:

  • The schema itself is documented in this spec, and versioned. The capability (which you have stated you plan on using yourself) to host a private index relies on this, so we have the need and the mechanisms in place to manage change.
  • Uptime must be guaranteed, as if the index is down, python install --target . 3.12 will fail. Are we willing to break people using that in their CI jobs?
  • The URL for the index will need to be stable, as otherwise how will the manager find the index?

I don’t think there’s any additional risk from letting consumers other than the manager use the index. Quite the contrary - by documenting it, we can state explicit terms of use. Legitimate users will respect our requirements, and malicious users won’t be blocked by the fact that they have to look at the code to find the URL…

OK. But it’s not just about help. People will run python install --target . 3.11 to install a standalone copy of Python, and activating a venv will break that expectation. Maybe we need to make the canonical command into py-manager install --target . 3.11, with a note that python does the same when you’re not in a venv. But that comes back to the confusion over “is python the manager or the interpreter”…

I may be over-thinking this one - people may find it less confusing than I expect. But at a minimum, we should add a note to “how to teach this” saying “we hope that the fact that python in an activated virtual environment doesn’t run the manager won’t be too confusing”… :slightly_smiling_face:

No, my point is that people (me, at least :slightly_smiling_face:) have scripts, workflows, and muscle memory that rely on the py executable and its current behaviour. Removing it, or making it do something different, will break things.

For example this code in pipx relies on py -3.11 -c "import sys; print(sys.executable)" printing the full path to the user’s Python 3.11 interpreter. If the launcher is no longer part of a default Python install, that’s going to break.

Which reminds me - it’s not clearly stated in the PEP but I assume python install 3.11 will create the necessary registry entries to declare that Python 3.11 is available on the machine as per PEP 514? Because if not, then we should be formally deprecating PEP 514 as a way of discovering what Python interpreters are available[1].

Also, what will happen to the file associations for .py and .pyw files? Will they now go through python rather than the launcher? Because the behaviour of those associations is defined by the behaviour of the launcher, which is defined by PEP 397. Being able to double click on a .pyw file with an initial line of #!C:\Users\Gustav\AppData\Local\Programs\Python\Python313\pythonw.exe and have it run with that specific Python interpreter is guaranteed by that PEP, and I’m sure it’s behaviour that many people will rely on.

This may be an option for console scripts, although I have extremely strong views regarding the fact that using .bat or .cmd files is full of pitfalls and should never be done unless you’re prepared for the consequences[2]. The OS is simply so hostile to using anything but an .exe file to implement a CLI application that there’s no good answer here.

I’ll drop this debate for now. It would be too easy for it to become unproductively confrontational. But the key points here are:

  1. The py launcher’s existence and behaviour is guaranteed by at least PEP 397. Dropping it implies dropping support for that PEP.
  2. The .py and .pyw associations are also part of the provisions of PEP 397. While the .py association is arguably flawed because of how PATHEXT works in practice, the .pyw association is necessary to support double clicking Python GUI scripts.
  3. I don’t know for certain, but I would imagine that shebang support is used by people wanting their .pyw script to be run under a specific interpreter. Maybe requires-python would be enough to cover this, but I can imagine a script being tied to a venv containing its own dependencies.

Ultimately, it’s already hard enough to use Python scripts as “applications” that work reliably - I’m an expert in these matters, and I’ve never found a solution that didn’t break somehow[3]. But the py launcher is the nearest we have at the moment to a solution here, and dropping it without offering something better is a regression, IMO.

It wasn’t. I had misunderstood that the installed runtimes would share the “quirks”. Now that you’ve clarified, I see that isn’t the case, and actually PyManager will remove those quirks from store-installed Python. So that’s a benefit, and should be made clearer in the PEP.

OK. I, on the other hand, deal with an unusually high proportion of people who don’t have a Python command, and use py exclusively. This is largely because I took your advice that setting PATH in the installer was a bad idea, and told them not to do that but to use the launcher. So you can blame yourself for that problem :slightly_smiling_face:

More generally, people (again, largely driven by your advice on PATH) have been promoting py as the “correct” way to run Python on Windows for years now. The pipx install instructions for example suggest py. The Python on Windows FAQ tells you to run py. All of that documentation will need updating, and the people who have learned from it will have to re-learn new habits.

I’m not trying to say this isn’t a worthwhile thing to do - overall, I think it is. But I think you’re glossing over a significant change that’s going to impact a lot of people, and that concerns me.

But nobody will be immune to the period of confused messaging that will follow this change. Users staying on older Python versions will have to reinterpret instructions to “run python” as meaning to “run py” - and we know from experience that many find this confusing. New users who start with PyManager will need to know about py, even if only as “the old way of doing things” and that it (sort of) just means “run Python” (except when it doesn’t…)

I did mostly mean the perception - although IMO this is actual churn, and calling it “perception” is part of the problem.

One thing that I think would help a lot - although at the risk of framing this as “packaging vs core devs”, which I do not want to do, but the division of responsibilities is very visible here - is for the core devs to put effort into publicising this change, making it a highlight of Python 3.14 (or whatever version it goes into) focusing on the benefits and how the new experience is improved, while providing transition advice for users and explicit advice to tool developers on “how to explain to your users how to run Python on Windows” - with links to “official” documentation hosted on python.org that discusses the transition, the reasons for it, and the benefits it will bring.

I know that’s extra work (and quite a lot of it!) but I think it’s important.

When I was talking above about the index being “supported”, I was thinking about the redirector of another example of something that we’ve done, but not really documented, and has ended up being shrouded in mystery while still being critical for some people (notably people working on environment managers) to understand.

I can’t speak for David, but I certainly don’t distinguish between “python.exe in a venv” and “python.exe elsewhere” in the way you assume here - because there’s nothing documented to suggest that I should need to do so.

I’m keen that we don’t make the same mistake with the index that we did with the redirector :slightly_frowning_face:

For you, “Python in a venv is different” is very clear, because you know about the redirector. But you may be the only person for whom this is true. For everyone else, “Python in a venv” is just a Python interpreter like any other, it’s just using an isolated environment.

  1. It’s arguably becoming less and less meaningful anyway, as things like standalone installs of Python are becoming more common, but tools like pipx, tox, nox and virtualenv do still use it. ↩︎

  2. I’ve lost track of the number of times I’ve had to debug issues caused by the fact that .bat files don’t nest properly. Java uses this mechanism all the time, and it sucks. ↩︎

  3. You could argue that I’m too much of an expert, and I see flaws that others don’t care about. I’m not going to dispute that… ↩︎

10

Valid reasons, yeah. Okay, I’ll remove those mentions, but will see what flows out of that - if it’s going to turn into e.g. having to publish and maintain helper libraries then I might back out of it again! (My main reservation is that the amount of work I want to do is to build and maintain the manager, and update our builds. I don’t want this to turn into something of Warehouse scale.)

This is one of the things that’s already in that section - not quite the words you have, but it does say:

Now, rather than the install manager, python will always launch your virtual environment, and any packages you install are only available while this environment is active. To get access to the manager again, you can deactivate the environment, or use py-manage <command>.

Is that insufficient? (Obviously that section has a lot of text to come!)

It’s clearly stated now. The entries will be created, though they aren’t consumed by PyManager. Keeping py around and independently installable (now also suggested in the text) might be the best option for preserving those workflows.

They can be associated with python.exe and pythonw.exe which alias to the manager. If you’ve already got py.exe installed then the next time you trigger one Windows will pop up its dialog letting you choose how you want to launch it.

Pretty sure people who really understand shebang lines say the same things about those :wink: Neither is a great solution, but it’s the ones the platforms have and support. We don’t need to jump through hundreds of hoops to port shebangs across platforms, but I do want to be able to enable some basic portability (which is why I’m less concerned about shebangs with Windows paths in them - that’s not portability, that’s a different feature).

I think I can also be considered an expert here, and I’ve found a few solutions that are just fine. The prototype PyManager, for example, is a reliable application that’s mostly written in Python.

The challenge is entirely in using just the script and some random install of Python. That is risking everything, and unless you have the document compatibility of an app like Excel, it’s always going to be huge. Reliable Python apps need you to be totally certain about your entire stack.

Yeah, most of the challenges faced here are because of stuff I did myself :slight_smile: Happy to take the responsibility, and proposing what I hope is an overall workable solution is my penance (along with handling the transition pains).

Yeah, it gets worse for a period, but the messaging is already confused. At least by releasing something we can do the big PR push you suggest and explain the before/after situations better. It’s hard right now to find excuses to get out there and clarify, because it just comes across as “you’re doing it wrong”, to which everyone replies “but it works (sometimes)” and ignores us. I know this because I’ve been trying it over the years.

I assume it’ll mostly land on myself, though of course anyone is welcome to help get the word out. I’m hoping that 3.14 is the first release that will offer it as an option (and stop increasing the number of independent Python apps in the Store), but the old options will remain available for a couple of releases (unless another major security issue arises in the traditional installer and we have no choice but to pull it…).

I mean, you know that I dislike the venv model (hence PEP 582). I’m doing the best I can to work within it without disrupting what people didn’t want disrupted by having a single Python install and executable-less projects.

11

I’m unsure how many people add python to path on windows, but count me in the number of people who both never did and advised others not to because the py launcher on windows has historically been “the most well-behaved option”

I’m concerned that this is adding yet-another-way, when it seems like the ideal might be just making the py-launcher Microsoft store aware, available from the Microsoft store, and capable of managing python versions

1 Like
12

My only real expectation of the difference (as a user at least) is in which interpreter is launched when I enter python in a terminal. I’m not too worried about how it gets there as long as it arrives at the appropriate interpreter.

Sure, it doesn’t need to explicitly launch the other executable just share the same Python discovery logic.

I don’t really consider this “admitting defeat”. I’m not sure what victory would count as if I did! :slight_smile:

My view of things is that the python command should be for launching the “context-appropriate” interpreter, and that with PyManager I’d expect it to be whatever that considered the default[1]. pymanager.exe would then be the tool to use for installing/listing/updating/removing python installs and selecting what its own python.exe launches.

I’m not convinced it’s necessary for python.exe to be able to install/remove/list python installs outside of the case where it is used and pymanager.exe has no installs available[2].

I believe the initial goal here is to make more sense and improve a current way rather than add yet another way. With the eventual goal being to reduce the number of ways by combining the work needed for both the python.org Windows installer and the Microsoft store.

Doing things with the py launcher doesn’t solve the issue that entering python in a terminal gets you the python ‘app’ from the Microsoft store and it’s then logical that using python again after install should launch the version you just installed. Users expect the python command to do something and the current situation with a separate Windows store app for each major Python release isn’t ideal.

  1. In the same way that in my case, currently it’s whatever pyenv-win has as the first ‘global’ entry. ↩︎

  2. Such as when first installed and used, when it should (presumably) retrieve the latest version of Python from an index. ↩︎

2 Likes
13

If it’s anything more than “this is the URL, the PEP defines what’s present there, and we’ll ensure it’s reliably available” then I’ll support you in refusing to go beyond that :slightly_smiling_face:

That explains in technical terms how to work around the fact that python no longer invokes the manager. I think it would be worth acknowledging that this is a workaround, and there will be people (we can’t be sure how many) who will be uncomfortable with the fact that python install stops working. It’s more about perceptions than implementation, and as such probably ought to be in the release notes or documentation rather than in the PEP, but the “How to Teach This” section is where we should list the things we need to cover in the docs.

Cool. But that means that changes to shebang handling are breaking changes. Possibly at a sufficiently small level that we’re OK with the breakage, but the PEP is where we should document what will break, no matter how small. If you think the breakage is justified, just say so (ideally, with evidence, of course!)

Hey, I understand shebangs! Yes, they suck too (especially on Windows, where the kernel doesn’t support them). The failure modes are different, though, and that’s the point here - you can’t assume that bat files are a replacement because the difference in limitations might matter. That’s why changing shebang implementation is also problematic.

Without data, it’s impossible to be sure. All we can do is minimise the differences (which is what I’m advocating) and hope for the best.

Please share! (Probably in a separate thread, or offline).

I’m happy to help, but in order to do so we have to be sure we have a unified message to give. And right now, it feels like we still have too many questions.

I see this change as effectively making the Microsoft Store version of Python into the official distribution channel for Python on Windows. With python.org hosting a copy of the distribution file that people can download for manual/offline installation. Most people won’t see the fact that the actual interpreter is downloaded from python.org as the important point here. And I think this is a sufficiently large change in perception that I think we should get a consensus from the SC and the core dev community that this is the message we want to present. Because I’m sure there will be people who frame this as “Microsoft taking over”, and we won’t be in a good position to counter this if we’re debating internally over nuances.

2 Likes
14

I’ve spent a decent amount of time over the holidays making the implementation more robust, robust enough that I’m willing to share a preview.

My hope is mainly to test the usage experience - I want people to see how it feels, and provide feedback after using it for a bit. Since the original PEP text, I’ve added that it will detect and launch other installs (found via PEP 514), and supports a py.exe entrypoint, as well as the legacy -0 and -0p options, though it’ll still prefer its own installs - you have to specify. You may need to uninstall an existing “Python launcher” (from Installed apps) to use this.[1]

The only not-so-obvious command I’ll mention here is python list --online <version> to see what packages are available. But a big part of this is to find out how usable it is without help. There will be documentation, of course, but we know that people poke around before they look up docs, and the further they get the happier they will be. See the PEP text for the current best documentation.

The main caveat for now is that it’s using a bundled feed (i.e. it’s not online) that references the packages from Nuget (see the PEP for more info on these) - eventually it will reference its own packages that include things like Tkinter.

If you’ve installed from the Store before, you may need to “Manage app execution aliases” to enable the new install. No avoiding this, unfortunately - it’s how the OS is designed - but there have been so few issues about switching version that I suspect people either figure it out or don’t care.

The MSI installer is not recommended for most users - if you know how to use msiexec without looking it up, it might be for you. Everyone else should be using the MSIX (see the PEP for more info on this). I’ll try and help if it breaks your machine, but I can’t offer any more than that. Installing runtimes will modify your registry and Start menu, which can’t be undone if something breaks. It shouldn’t delete anything it didn’t create.

Sources: Release 0.1a2 · zooba/pymanager · GitHub
MSIX: https://cpythonpackages.blob.core.windows.net/pkg/python-manager-0.1a2.msix
MSI: https://cpythonpackages.blob.core.windows.net/pkg/python-manager-0.1a1.msi (not recommended, use the MSIX)
(Edit 08Jan25: Updated the MSIX to 0.1a2)

Packages and binaries are signed with my own certificate, so trust them as much as you trust me. Code signing is not optional for MSIX packages. The runtimes that will be installed (with the bundled feed - this is, of course, entirely configurable) are the official PSF-signed releases from Nuget.

I will delete this download when I have enough feedback, or if it turns out to be broken. If you are here looking for it and there are no links, scroll down in case I posted another.

  1. This is one of the reasons I’m not so keen on trying to directly replace py.exe. ↩︎

3 Likes
15

I’ve made a range of updates to the draft PEP text, and intend to go back through the recent posts to see if there was anything that I’ve missed (it’s possible I’m still not being as gentle with the “changes cause change” messaging as I may need to), but wanted to signpost the current state and encourage people to treat it as near complete.

There’s a new test build at https://cpythonpackages.blob.core.windows.net/pkg/python-manager-0.1a2.msix if anyone wants to try it out. The python help command should be the only one you need to start exploring the options, and of course just running python will get you into a working (recent) runtime. (The PEP text and current sources are slightly ahead of this build, but nothing majorly different IIRC.)

I’m going to prepare a video demo as well, since I expect that’s going to be more useful for most people to decide whether it’s worth the churn. (If you want a video on the current install process, there are hundreds on YouTube :wink: )

What I’m really keen to hear is feedback from people with unusual install situations, such as networkless/air-gap or pre-installed deployments where it’s not possible for a python install ... command to reach our server, or it’s not possible for a user/script to do an install even if the files are available. I have a general feel for a lot of these scenarios, but would love to be able to confirm that a few real ones will work or figure out how to make sure they’ll work.

16

And a video demo for those who can’t/won’t install the package. No edits - it’s really this fast, and I really made that many typos :smiley:

Edit: Features not in this demo

  • detects existing installs (just like py launcher)
  • has a py alias
  • basic shebang processing (e.g. python script.py checks script.py for a python3.x-like command)
  • filter python list with ranges (e.g. python list >3.10)
  • format python list output (e.g. python list --format jsonl)
  • configuration files/environment variables/etc.
5 Likes
17

I’m a windows user and that looks amazing, thank you for working on it.

2 Likes
18

This looks very nice, but I still don’t like having “python” launch the manager. Why not simply name the command this “pymanager” ? (after all, the popup already asks for installing the “Python Install Manager”)

This would get rid off most of the concerns voiced in this topic, AFAICT.

And it would also resolve a tangent from a completely different angle: calling something “python” which isn’t the Python interpreter (or a reasonably close derivative), does clash with the trademark policy we have on the wordmark, since the main point of having a TM policy is to avoid user confusion.

1 Like
19

I’ll caveat this first with, this is the first time I’ve directly used a .msix installer and I don’t generally use the Windows store often. This may just be a quirk of the system but I did have some issues trying to find the application after it had been installed.

The installer itself doesn’t really tell you much, I’d like it if it told you what aliases it had created and if any had specifically not been created due to the “app execution aliases” settings. I had to dig into that as somehow all of the non-windowed aliases were disabled[1]. The fact that the application that’s launched when the install finishes is called py-manage.exe in some inaccessible program files location did also add to the confusion a little as I was searching for that and not PyManager.exe.

Once I’d got past that hurdle things seem mostly fine as you show in the video. I still think the PyManager and python commands should do different things and that putting the management features into the python and py commands confuses things more than it helps.

A few examples of weirdness/bugs.

  • python --help and python help now give different info.
    • One gives the standard python help and the other the installer help.
    • Similarly for py help/--help and pymanager help/--help.
  • py has different output for list and --list.
  • If a venv is activated, if someone’s got used to the commands working with python they all stop working until they deactivate the venv.
  • Running pymanager just launches python, which I don’t think is actually that useful.
  • The py alias provided here doesn’t respect virtual environments and launches the base install.

Currently to manage python installs I use pyenv and it mostly gets these things “right”[2].

python is replaced with a shim, it redirects to whatever version of Python you have as your current default in pyenv. The exact mechanism isn’t important but this is all it does, you can’t access any of the install management features through the python command.

Running pyenv with no arguments tells you what all of the management commands are without needing to remember if it’s help or --help[3].

pyenv uses global/local/shell commands and settings to determine what python and other aliases do. I think the multiple configurations are potentially unnecessary for PyManager but I do think it’s useful to be able to easily change the default. It’s fairly likely for instance that I’d want to install 3.14 and have it available on release, but not make it the default until some other packages support it.

I hope that’s some useful feedback, in general I’d be supportive of the idea if it switched to a clear separation between the python command and the tool used to manage both that command and the installs.

Suggestions on that line would be that if someone just uses python with no arguments, to print a line before launching the REPL showing that it’s being launched by pymanager and that they can use the pymanager command to install/update/modify installs.

You could also make it so entering pymanager in a Python repl launched this way explained to users that they needed to use pymanager from the command line and not the repl, but that might be going too far?

  1. I was fairly sure I’d disabled python.exe at some point when setting up to avoid the Windows store redirection, but I was surprised that PyManager.exe also ended up disabled presumably as a result of this. ↩︎

  2. Of course here ‘right’ is subjective. ↩︎

  3. It’s help ↩︎

4 Likes
20

I don’t know about “unusual” but I have a high proportion of people who are new to Python and are told to install Anaconda because it brings in all the needed packages, provides GUI for launching Jupyter etc.

When the previous python.exe store launcher was added it sounded like a good idea to me until I had 200 students installing Anaconda and then trying to run python script.py. Previous instructions or installer settings for setting PATH no longer worked I think because python.exe was in a privileged earlier part of PATH and was hard to override. Then students who already had Anaconda were installing the store Python on top and then getting confused about their multiple Pythons with different installed packages.

We evaluated the possibility of just using the store Python but there were various issues with it. I don’t remember exactly but something like:

  • The store Python was not a “normal” Python installation somehow and couldn’t do certain things.
  • We wanted students to be able to use numpy/matplotlib etc before needing to learn about anything like pip/venv.

The sequence in which we typically want to teach these things is like:

  • Basic Python
  • Scientific stuff (numpy etc)
  • Learn terminal and make CLI programs
  • Learn pip and packages

Note that the terminal part comes after spending some time learning Python via GUI (e.g. Jupyter, Spyder, …). At the point when Python, numpy etc are first installed students are not assumed to know how to run commands in a terminal. After they can use CLIs they can learn pip, git etc.

How does this new python.exe impact people using Anaconda?

Does it provide a better alternative to Anaconda than the current store Pythons?

3 Likes