PEP 773: A Python Installation Manager for Windows

[pitrou (Antoine Pitrou)]

@steve.dower

No, NuGet is specifically for unregistered installs, so after it extracts it you need to go into <dir>\python-freethreaded.3.13.1\tools\python.exe directly.

And what is <dir> here?

[steve.dower]

@pitrou

And what is <dir> here

Your own choice of directory. It’s in the nuget install command as well as the python.exe command.

[jamesdow21 (James Dow)]

@steve.dower

@jamesdow21

TOML would be a really nice choice (which PEP 518 explains far better than I could) over JSON or YAML

JSON has more reliable string quoting and escaping rules than TOML, which matters a lot when dealing with paths on Windows. I do wish Python’s parser would allow comments in JSON, and if someone were to add it then PyManager would also support it, but as far as tradeoffs go it seemed like the right one. There’s nothing preventing a switch to TOML configuration files (support both formats) in the future, but I expect custom configurations to be fairly rare here, there aren’t that many knobs.

Exactly the type of reason to prefer JSON that I wasn’t aware of. Like I said, definitely not a dealbreaker to use it over TOML.

Might be worthwhile to add a sentence or two explaining that choice to the configuration section or potentially a new section down in “Rejected Ideas”^[1]

Especially because any later proposal to add support for TOML could then reference that more explicitly than having to search back through this thread for details of why JSON was chosen in the first place.

I’m personally quite a fan of seeing those types of clear and concise details about trade-offs when picking something for a design, because I feel like whenever I stumble across them as I’ve been learning over the years, they have been good teaching examples about how to make and document good design choices for my own projects

small edit to add: I also want to add a thank you shout-out to all the people who have written so many good proposals over the years and contributed to discussions here. I’ve learned a ton from so many great people here and it still feels really surreal to occasionally be involved in discussions with people whose names I know so well from all of the work they’ve done

---

1. Considering the full scope of the PEP, it’s a very minor detail, but using a different file format for config is an “Idea” that was “Rejected” ↩︎

[barry (Barry Warsaw)]

@steve.dower

JSON has more reliable string quoting and escaping rules than TOML

My general rule of thumb is to use JSON for computers and TOML for humans. JSON is not a human friendly format, but it’s great for interchanging data between software.

[bwoodsend (Brénainn Woodsend)]

I’m a bit squeamish about no longer being able to disable MAX_PATH. With virtual environments being installed into the user’s temporary directory, I find the limit gets crossed pretty easily. And last I checked, there’s no special handling in pip to catch/explain the resultant OS error.

[bwoodsend (Brénainn Woodsend)]

Can you clarify the story with automatic updates? It looks like PyManager autoupdates – is this disable-able? What about the Python installations? Would 3.12.0 ever implicitly update to 3.12.8 or 3.13.0?

[steve.dower]

@bwoodsend

I’m a bit squeamish about no longer being able to disable MAX_PATH.

Well, you just don’t have a convenient button (which requires admin, which mean vast numbers of people can’t actually click it), but you can still disable it (I assume you noticed that you linked to the instructions for how to disable it without using the installer, but perhaps not everyone else clicked through).

I’m using the lack of button to apply pressure to the people in a position to change the default setting. I can nearly prove that 100% of users who have enabled this option did it because of Python (or more recently, Git).

@bwoodsend

there’s no special handling in pip to catch/explain the resultant OS error

The error just reports as “file not found” (I think because the buffer gets truncated somewhere in the OS, though I haven’t tested to see if it’s vulnerable to opening the wrong path), and the fix for many users is still “avoid this library that nests unnecessarily deeply” rather than anything else.

@bwoodsend

It looks like PyManager autoupdates – is this disable-able?

Yes, it’s an OS feature, and can be disabled per-app in the “App settings” page.

@bwoodsend

What about the Python installations? Would 3.12.0 ever implicitly update to 3.12.8

Right now, no, but manually running py install --update will do it (and if you set up a scheduled task to do it automatically then it can be automatic).

@bwoodsend

Would 3.12.0 ever implicitly update to … 3.13.0?

Never.

[steve.dower]

@barry

My general rule of thumb is to use JSON for computers and TOML for humans.

That’s fine, but unfortunately, both are fairly annoying when it comes to backslashes, and backslashes are critical to Windows configuration.

At least JSON only has exactly one convention, even if it means you always need to double up backslashes and escape double quotes. Not following the convention is a parse issue that any JSON parser/editor will tell you about.

TOML gives you the choice to use almost JSON formatting but you can’t use certain quotes, or to use more convenient formatting but you can’t use single quotes (which are valid in paths), and in some cases the escaped characters include the escape and others they’ll treat it as an escape. (YAML offers even more options…)

And we’re talking about overriding a small set of defaults, where typically the default value is going to be fine (or conveniently overridden from the command line, such as the default format for py list). Writing configuration isn’t a core scenario here (compared to, say, pyproject.toml).

[MegaIng (Cornelius Krupp)]

@bwoodsend

I’m a bit squeamish about no longer being able to disable MAX_PATH. With virtual environments being installed into the user’s temporary directory, I find the limit gets crossed pretty easily. And last I checked, there’s no special handling in pip to catch/explain the resultant OS error.

If pymanager can’t do this automatically, this is IMO a dealbreaker for using this to replace the user-facing installer. Often users on home systems are admins and therefore the installer would just be able to make this change - no longer having this be part of the installer will cause additional and unnecessary issues, questions, bug reports, … (unless microsoft comes to senses at some point and decides to switch this option by default in some update)

This is just going to affect CPython, but all python packages (no, not just package installers/managers). The current installer at least gives users a very clear option “press this button to fix problems” that, in the worst case, packager authors can direct users to. Asking endusers to modify the registry themselves severely increases the barrier in a way that is going to prevent this fix from being used.

@steve.dower

TOML gives you the choice to use almost JSON formatting but you can’t use certain quotes, or to use more convenient formatting but you can’t use single quotes (which are valid in paths), and in some cases the escaped characters include the escape and others they’ll treat it as an escape. (YAML offers even more options…)

(I do want to point out that the choices are very similar to what python itself gives it users, just using ' instead of r". I also don’t think giving users more choices to express themself however it’s convenient is a good argument against a format… I am also not sure what you mean with “almost JSON formatting but you can’t use certain quotes”. AFAICT the syntax for single " strings is identical? But I don’t care that much about this.)

[steve.dower]

@MegaIng

If pymanager can’t do this automatically, this is IMO a dealbreaker for using this to replace the user-facing installer.

The most it can do is have an extra option to one of the proposed commands that has to be run as administrator, which is as much or more documentation than what’s currently there. There are no longer any buttons that we control in the install UI - it’s just a CLI tool in a standard OS package.

@MegaIng

I also don’t think giving users more choices to express themself however it’s convenient is a good argument against a format

I already said we can add more formats in the future. The PEP is not a constraint on the design of the installer (unlike most PEPs recently, so I get why people would assume this), it’s just setting out the proposal at this stage in enough detail that it’s possible to decide without having to seek out further context.

[MegaIng (Cornelius Krupp)]

@steve.dower

The most it can do is have an extra option to one of the proposed commands that has to be run as administrator, which is as much or more documentation than what’s currently there. There are no longer any buttons that we control in the install UI - it’s just a CLI tool in a standard OS package.

Right, to be convenient the manager would need to have an elevator to get admin privileges and would need to suggest it at some point during an install process - which seems difficult to do UX wise and annoying to implement. But maybe an option that does this would be good anyway? It would be

• Launch a terminal with admin privileges (right click start symbol), run pymanager enable-max-long-path.

vs

• Open the registry editor, go to the path ..., don’t edit anything else, edit the key ... (or maybe create it if it’s missing)

It’s probably not a huge difference, but the former feels simpler and less scary.

@steve.dower

I already said we can add more formats in the future. The PEP is not a constraint on the design of the installer (unlike most PEPs recently, so I get why people would assume this), it’s just setting out the proposal at this stage in enough detail that it’s possible to decide without having to seek out further context.

Yeah, fair enough.

[pf_moore (Paul Moore)]

@MegaIng

Launch a terminal with admin privileges (right click start symbol), run pymanager enable-max-long-path

It would be pretty trivial to write a small script to disable MAX_PATH (it’s just a simple registry change), so this can easily be “Launch a terminal with admin privileges and run python no-max-path.py”. The no-max-path.py script could be in the docs somewhere, I guess, or someone motivated could publish it as something like a gist (or even as a package on PyPI, if you feel like over-engineering it )

Discoverability is worse, conceded, but at some point we have to accept that it’s not Python’s job to tell people how to administer their Windows systems.

Having said that, I wouldn’t object if Steve felt that adding a pymanager no-max-path subcommand (which had to be run as admin) was worth it.

[steve.dower]

@pf_moore

Having said that, I wouldn’t object if Steve felt that adding a pymanager no-max-path subcommand (which had to be run as admin) was worth it.

I suspect some kind of additional “mess with my system” command is going to be inevitable - it could also do the PATH modification (without admin) that people are going to demand.

My main hesitation is that I dislike making changes that we can’t undo. The MAX_PATH change at least only affects processes that declare they support it, while the PATH change surprises other apps (though hopefully it’s better controlled now that we won’t put random DLLs in there).

Discoverability would be to change the “Add this directory to your PATH for global commands: <directory>” message to “Run this command to modify your system: py install --configure-system” (I’m trying to minimise subcommands, and also there’s no reason it can’t do those changes while also installing runtimes).

[bwoodsend (Brénainn Woodsend)]

@steve.dower

@bwoodsend

I’m a bit squeamish about no longer being able to disable MAX_PATH.

Well, you just don’t have a convenient button (which requires admin, which mean vast numbers of people can’t actually click it), but you can still disable it (I assume you noticed that you linked to the instructions for how to disable it without using the installer, but perhaps not everyone else clicked through).

It’s less about convenience and more about awareness. No one is going to look at a strange file not found error in a failed pip install and think I wonder if there’s a magic registry setting which is documented at the bottom of the pymanager guide. That prompt was the first place I learned of MAX_PATH.

But overall I agree that there’s not much pymanager can do. I’d still suggest we put some try/except/helpful error message guarding into pip though. In terms of awareness, that should be almost as good.

[AA-Turner (Adam Turner)]

Would it be going too far for pymanager to try writing and reading a 261-character-long path in %TEMP%, perhaps during the install step? That would mean that it could print the ‘modify PATH’ message at the same time as forewarning the user that they will likely have problems with path length, unless LongPathsEnabled is set.

A

[steve.dower]

@AA-Turner

Would it be going too far for pymanager to try writing and reading a 261-character-long path in %TEMP%, perhaps during the install step?

It can read the setting easily enough. Though I’m only assuming that if the default is changed that it’d be changed by setting the setting (if it were changed some other way, reading the setting wouldn’t work and testing is the only way).

[jack1142 (Jakub Kuczys)]

@steve.dower

You mean for the install manager itself?

Yes.

@steve.dower

When installed from the MSIX, PyManager will have its own automatic updates, which involves there being an appinstaller file somewhere on python.org that references the latest release. That may be sufficient for someone who wants an API? But since Windows already uses the API for updates, it’s only really a benefit for the initial install, and for that you don’t actually need the MSIX - you can point the OS at the appinstaller file itself and it’ll get the right one.

If I could use that to download an MSIX file of the latest version (at the time I fetched the appinstaller file), it should be sufficient. I’ll have to play with it but it seems like that would be the case.
The idea here is to bundle MSIX with the Chocolatey package - this is a preferred option when the license of the software allows redistribution of its binaries as it means there’s no need to download anything during package installation.

[steve.dower]

@jack1142

The idea here is to bundle MSIX with the Chocolatey package

Yeah, that’s the idea, and why the MSIX will be available to download from the python.org download page. It’s just the lack of API for finding the current URL that you want.

My testing appinstaller file is at https://cpythonpackages.blob.core.windows.net/pkg/pymanager.appinstaller if you want to see how it looks. I expect the only thing that’ll change is the URL.

[steve.dower]

@DavidCEllis

Things I ran into in brief testing this morning:

Just letting you know that I’ve fixed/improved most of what you’ve shared here - if you’ve still got it installed then you should get the new version automatically today, or it’s on GitHub. (Note: quick follow up update)

@DavidCEllis

Would it be possible for pymanager to detect if its copy of py is shadowed on PATH and warn the user, also possibly listing the commands with pymanager instead of py where appropriate

The second part yes, it’s done in the update. I’m thinking to add warnings to the old launcher for py list/py install/py uninstall/py help (when no script file by that name exists) to try and help with shadowing.

I want to minimise the messages that turn on/off by some system state, since if the state is intentional, then the message just gets in the way. Using pymanager as a command is the intended way to keep the legacy py command intact, and so you wouldn’t want more messages (more helpful errors, yes, but those have to go into the legacy py.exe).

[ichard26 (Richard Si)]

@bwoodsend

I’d still suggest we put some try/except/helpful error message guarding into pip though. In terms of awareness, that should be almost as good.

(puts on pip maintainer hat)

I’m not on a Windows machine so I can’t verify this easily, but pip does try to detect that the path is too long and raise a more suitable error in this case:

src/pip/_internal/commands/install.py

If this isn’t working, please file a bug report. I wouldn’t be surprised if something has broken this in the last four years since this was added to pip.