PEP 773: A Python Installation Manager for Windows

PEPs
242

This is actually part of the incentive for the change. Beginners have the incorrect intuition that installing a mutable (since pip needs to be able to install packages) into a read-only location (which is what installing as admin does) is a sensible thing to do then bombard other projects with bogus bug reports that they can’t install their packages because of permission errors.

That’s just Windows. os.getcwd() is nothing more than your current working directory — for applications launched through the desktop, the current working directory doesn’t make any sense but has to be set to something so Windows chooses the executable’s location[1] and macOS and Linux use the user’s home directory. In either case, it’s meaningless and not something code should use. IDEs like IDLE may change it to something more natural but it’s still broken code that relies on it.

Using __file__ was always the correct way to locate the current Python script. Depending on what you actually want that location for, that’s almost always still the correct thing to do even with pyinstaller.

  1. Presumably you’re using the py launcher and it’s installed in C:\Windows\System32. ↩︎

2 Likes
243

But my point is that the behavior of this exact same version of Python (3.14.2) is different when installed via executable (or at least, if installed “for all users” in the traditional Programs directory). When installed the “old way”, os.getcwd() always gives the directory of the script file, if it exists. I understand why it might default to another location if the script does not exist (for example, if working in the interpreter directly) and that’s perfectly fine. But why stop giving the location of the script? It seems to me the “old way” was quite sensible.

As I already pointed out above, the problem is that __file__ doesn’t work once you’ve frozen the code into an executable because it just points you to a temporary directory. Ironically, in that case, os.getcwd() continues to work just fine as expected. And I’m not actually generally using os.getcwd() explicitly. Usually I’m just pulling in files from a neighboring directory. Parsing a folder full of excel files, for example. The point is that the working directory used to be the script’s directory in 100% of use cases, so I could just use relative file paths rather than needing to tell the script explicitly where it was. That doesn’t work anymore UNLESS I run the script from an IDE or I’ve frozen it into an executable, which feels weirdly inconsistent.

As an aside, it may not surprise you to learn that I’m an engineer, not a CS-educated developer. By lunchtime I’ve done ten things you’d probably consider heretical practice. My focus is on usage and outcomes, and I just find it frustrating that things that have worked perfectly fine for the last decade or more are suddenly not working fine, and what I’m told is the “correct” way works in fewer situations than the “incorrect” way. And to hear that this is all apparently in the name of “user-friendliness” feels a bit backwards in the face of my experience with it.

With the plan to phase out the exe installer, it just feels like you’ve put a timer on my enjoyment of the language. I don’t begrudge your desire to have some install manager for development purposes, and I’m sure there are other people who will like it, but I really wish you’d just keep putting out the exe installer alongside it.

244

I’ve personally found __file__ (at least without a fallback) to be bad practice unless you can guarantee the context of where your code is being run from. There’s lots of contexts where it doesn’t exist and you get an exception referencing it, e.g. in REPLs, notebooks, embedded, different interpreters, etc.

1 Like
245

Only by chance, and not by design. The behaviour of Windows isn’t something we can specify in Python.

If you want to always have the current working directory set to the location of the script, then use one of the OS features to ensure that (such as a shortcut/shell link, or a batch file with a cd command).

I thought you were wanting to find the location of the script? Which will indeed be in a temporary directory in this case.

Perhaps you should explain the actual thing that you want (probably by giving concrete examples of what you want to use it for), so that we can offer you options that are intentionally designed to work. I think you’ve just been getting lucky for a long time, probably by only worrying about a very narrow range of uses.

Yeah, I get how this seems a bit on the nose. All I can say is that the specific things you’ve noticed weren’t the target of the “user friendly” changes - they’re just the side effects. The things that weren’t considered user-friendly were the unreliable [un]installers, the limited support for multiple simultaneous versions, the lack of programmatic/scriptable updates, and the security vulnerabilities in the installer.

We weren’t going “oh look, the common behaviour is to start in the script directory, so we need to fix that” - we actually didn’t consider that at all, because it’s not something we can control (except through the most indirect methods to induce the OS to do it, and even then we can’t guarantee anything).

Anyone is welcome to do this, even just repackaging our builds as released through the install manager. We’re choosing not to commit our volunteer effort towards it (and even if we took on a new volunteer to do it, that would still leave a burden on the entire team if that volunteer goes AWOL, and that’s the burden we don’t want).

2 Likes
246

I really really don’t think I’m doing anything even remotely unusual or unpythonic, but you’re right that I seem to have caused confusion by talking about finding the working directory, even if I think that’s the most direct symptom. Let me try again by providing a simple, replicable example.

Create a directory containing

foo.py
bar.txt

bar.txt contains whatever you want (how about “hello world!”), foo.py consists of:
with open('bar.txt', 'r') as fin:
input(fin.read()) # input() prevents the console window from closing immediately

This is an extremely basic, fundamental, and I think pythonic piece of code.

Using the executable Python installer, installed what I would call the normal way (“for all users”)…
IDLE: works
Windows Explorer, double click on script: works
Run executable generated by pyinstaller: works

Using the Python install manager, default options…
IDLE: works
Windows Explorer, double click on script: FAILS due to being unable to find bar.txt
Run executable generated by pyinstaller: works

I simply cannot believe this is either desirable or inevitable; that sort of elegant simplicity is what made Python a Good Thing. I would guess it’s a failure on the part of the py launcher to provide the terminal with its location (see my previous comments) but at the end of the day, the moment I have to start hard-coding file locations or using unreliable means of having the script locate its absolute path just to access a neighboring file is the moment that Python has lost its sauce.

1 Like
247

Steve will be able to explain better, but this is because the new py launcher is a modern Windows Store application. That’s necessary to allow Python to be installed from the Windows Store, and via the modern (and supported) .msix installer format. Unfortunately, one of the ways in which Windows behaves differently for Store apps is in terms of its behaviour when double clicking on a file associated with the app - the CWD is the Windows system directory. This is essentially because Windows doesn’t consider the CWD to be something that GUI applications (which are what you run when double clicking something, in Microsoft’s world view) should be relying on.

That’s why we’re saying this is an operating system behaviour, not something we can influence.

Of course, we could have continued to build the launcher as a traditional application, and carried on distributing the (no longer supported) MSI installers, but as the release team is all volunteers, we have to choose the approach that’s sustainable for us.

As Steve said, if there’s sufficient demand, someone could choose to repackage what we distribute in the old format. But it won’t be us, because we don’t have the resources to support that format for free.

248

Yeah, this is roughly what I expected - but let’s consider why open("bar.txt") doesn’t obviously mean “bar.txt adjacent to the current script”:

import sys
with open(sys.argv[1], 'r') as fin:
    input(fin.read())

---

$> python path\to\my-script.py bar.txt

This is essentially the same as your example - in both cases, it runs open('bar.txt', 'r') - but I think it’s pretty obvious in this case that it means the bar.txt where the user is running rather than adjacent to the script. The current working directory and the filename are both user-provided, so they can be used together automatically, but the script directory is not considered to be user-provided.

In this case, you’d use __file__ to get the script location[1] and make sure you read the file from there:

import pathlib
script_root = pathlib.Path(__file__).parent

with open(script_root / 'bar.txt', 'r') as fin:
    input(fin.read())

Now, we’re no longer relying on the current working directory at all, but are always going to get the file from our script’s location. You can substitute the string constant for sys.argv[1] here as well and it will still use the user’s argument, just in a different location (you probably wouldn’t do this because it’s confusing in other ways, but it would work).

Most importantly, by explicitly saying “we want to open this file from the script location”, your script is going to work under practically any launch conditions, with no reliance on the OS doing anything specific.

And to be clear, this isn’t the only place it behaves differently, it just happens to be an obvious one. You also get different CWD behaviour if you launch as admin, or through symlinks or shortcuts, or through scheduled tasks. Fundamentally, the CWD is only really meaningful from inside a terminal, where it’s clearly under the control of the user. Programs launched in practically any other way are not obviously tied to a CWD, and so you can’t really assume what it’ll be set to when they start.

  1. I’m sure others will argue about this, but for a script it’s totally fine. If you’re building a general purpose, reusable, redistributable library with embedded data, there are other tools. But for a simple script like this, __file__ is just fine. ↩︎

3 Likes
249

To me this seems highly inelegant. Obviously you feel otherwise, but you’ve just doubled the length of the script in order to maintain current behavior, and (I keep trying to emphasize this), what you wrote won’t work if I freeze the script with pyinstaller. In that case, __file__ points to a temporary directory and using it will not allow the script to locate nearby unfrozen files.

Let’s look at your own docs. For an obvious example let’s go to the beginner’s guide ( 7. Input and Output — Python 3.14.2 documentation ). The entire file usage methodology implicitly assumes that the script’s location is the working directory. There’s never any discussion of needing any of this extraneous location pinpointing nonsense, and no beginner is going to figure it out. Now, you can’t even use relative file paths.

We’re talking about altering a behavior that’s been the bedrock of file access for as long as I can remember. Even in C I don’t have to figure out the absolute path to my location just to access a file in the same directory.

And again, as regards new users, you’re going to generate a LOT of confusion for every student (and the instructor, at least initially) when the code they’ve written in IDLE, that’s been taught for the last decade and has always worked, suddenly fails when they double click the file. Then they implement your suggested fix and find that it throws an error when they freeze it. Headaches all around.

Surely, surely, there’s a way for this to be taken care of under the hood with the py launcher. I really don’t buy that this is an OS limitation when both IDLE and pyinstaller can do it without issue on the OS in question.

2 Likes
250

That’s on PyInstaller, frankly. If they don’t include all the files you need, or they don’t put them in the right place, then how are you ever going to find it? Perhaps they have a way to handle it, I don’t actually know.

Every example here is at the interactive prompt, which is indeed going to use the current working directory. None of them use a script.

The section that describes scripts doesn’t mention the working directory, so adding some mention or explanation there might be helpful.

But the lack of explanation in a tutorial doesn’t force us into maintaining behaviour that you assumed. That’s not how specifications or designs can ever work. Reality dictates how gaps are filled, not assumptions.

Only by breaking other scenarios, such as launching from the console (which is the primary scenario people use), or by reporting it to Windows and hoping they fix it (already done, but the more people who report it to them, the better).

Not sure where to go from here - it is an OS limitation, because double-clicking a file is interpreted as opening a document, whereas double-clicking an executable is launching a process, and these two operations behave differently. That’s outside of our control. If you don’t believe me, well, then I can’t help.

2 Likes
251

Could we add a command line option --infercwd (name suggestions welcome) which causes py to run Python with a modifed CWD from the first argument? Then the file association for .py files could be modified to call py --infercwd <rest of the old association command>

It’s only a workaround for the store app limitation, but it’s better than nothing and stays consistent with old behavior.

Edit: On second thought, this could be problematic if called from a shortcut to a .py file with custom shortcut working directory.

1 Like
252

Yes, it’s already in the sources, just disabled at build time (the option is --__fix-cwd).

As you note, it becomes a problem in many cases, since a lot of terminals like to read the file association and would then behave incorrectly. I kept the implementation, since looking up the right directories is a bit non-obvious, but it doesn’t actually help. The only reliable way is to put it in the script itself, which may as well be spelled os.chdir(Path(__file__).parent).

253

Once you get to packaging and installing scripts in arbitrary places, you need to start using importlib.resources to handle finding resources within the package. Presumably pyinstaller would work with that, if not then that’s a bug to report to pyinstaller.

1 Like
254

That’s rather like calling a complex math function inelegant since it could be rewritten in much less lines as return 2 and still be right some of the time.

Yes, the script’s location if a script else the executable location if frozen isn’t something that’s been given its own variable since it’s generally not how applications work[1]. But if that’s what you want then sys.executable if getattr(sys, "frozen", False) else __file__ is how you’d do it without guessing or making invalid assumptions.

The only reason pyinstaller can do this is because you’re choosing where to put the .exe application. It’s equivalent to copy/pasting the py.exe launcher out of C:\Windows\System32 and putting it next to the files you want it to operate on. Likewise with C. It’s always a fluke.

FWIW, I also wish the Python docs explained portable resource location. PyInstaller’s issue tracker is full of bogus bug reports because of misconceptions about working directories, executable locations and resource locations[2] – none of which have anything to do with pyinstaller. There’s also a stackoverflow question on it with 9 wrong answers – more than half or which are answering the wrong question.

  1. You don’t copy MS Word next to a word document if you want to open it for example ↩︎

  2. Resource location being what __file__ or importlib.resources are more usually used for. ↩︎

3 Likes
255

If you read my previous replies, you’d know that it’s only your suggested fix that doesn’t work due to file returning a temporary directory. The ironic thing is that the cwd works exactly as it always used to once the script is frozen.

Well clearly I’m not convincing anyone here, I just thought it was worth a shot. I guess good luck with development and I hope it doesn’t cause too large an influx of complaints when you fully deprecate the executable installer. It’s certainly possible to use the language and be completely ignorant of the change depending on how one interfaces with it, just really hard to know what percent of the population that applies to.

256

I would frame @BlivetWidget expectations like this: A .py file that can be executed without obviously invoking a python interpreter should behave as-if it was an executable. That’s the entire point of a script.

Using a shebang on linux achieves this. And with the old-style installer, this was also the case on windows! But the new installer breaks this expectation.

3 Likes
257

Unfortunately, this is not supported by Windows. I’ve made the same proposal to the team responsible for the feature, and they vaguely say “yeah, kinda makes sense” but can’t commit to fundamentally changing the operating system. Batch files (*.bat;*.cmd) are hard-coded into the low levels of the OS, which is why they behave “normally”,[1] but other scripts don’t get to play at that level.

As I mentioned, it was the case some of the time, where the definition of “some” is outside of our control. Powershell is also partly responsible for some changes in this behaviour, and other shells like Cygwin have their own quirks. And of course, the OS itself is responsible for most of it.

The only way on Windows to get executable-like behaviour is to have an executable, with .bat and .cmd getting special treatment. Anything else is a document. That’s just how the OS operates.

  1. Also why Rust issued a maximum severity CVE-2024-24576, so “normally” doesn’t necessarily mean “good”. ↩︎

1 Like
258

In my experience, the problem with .bat files is that while the low levels of the OS treat them as executables, not all higher levels do:

  1. There’s no equivalent of “GUI Subsystem” executables. (This is actually a low-level distinction).
  2. Calling one .bat file from another doesn’t work properly (unless you use START, the call acts as a jump, not a subroutine). This completely breaks the abstraction of a .bat file as a “native executable”.
  3. Running a .bat file from cmd.exe doesn’t isolate it from the parent process. Things like changing the working directory affect the caller. I don’t know how common it is these days to use cmd (I switched to Powershell years ago) but for people who use it, this is a non-trivial issue.

Old-style file associations allowed some level of “make it work like an executable”, but they very definitely didn’t work at the lowest levels (subprocess.run wouldn’t follow an association to know how to run a .py file, for example). And they always felt like an afterthought for anything other than associating a document editor with a document type in the GUI (allowing .docx files to be opened with Word, for example). The newer associations feel like they’ve doubled down on this world view, and seem actively hostile to anything other than the “editor associated with a document” model.

Fundamentally, my view is that the only way to write a native executable on Windows is with a compiled .exe. Everything else is limited or broken in one way or another. There are approaches which you can make work for certain situations, and if those situations are genuinely all you care about, that might be fine for you. But as soon as you go out of that limited area[1], you’re back to .exe files being the only real answer.

I wish this wasn’t the case. I love the way Linux has built shebang-scripts into the OS[2]. I wish Windows had something like this. But I see no sign that Microsoft has any interest in making life for script developers or command line users easier, so I don’t expect it to happen.

Bringing this back on topic - ideally, we’d have kept the same partial workarounds that we used in the old installer working with the new one. But that isn’t possible if we want to move to modern, supported technologies (which we do!) We have new partial workarounds, but they work in slightly different situations, because (as I said above) no workaround works everywhere. So, sorry, I guess?

  1. For example, the instant you want to share your program with others ↩︎

  2. Much as I hate the inconsistent and frustrating rules on what constitutes a valid shebang line across Unix implementations ↩︎

3 Likes
259

And what should enterprise users do?

-MS Store blocked
-users are not allowed to install software
-users are not allowed to run software from their profiles
-Python cannot handle proxies with kerberos auth, so your install manager will not be able to download anything in this case
-we need a common Python Installation on Terminal Servers, so all users have the same version

Please keep a simple system wide installer!

260

Request IT to allow you to install the software you need to do your job.
I have always found that worked, even if it took a while.

Before you try to work around your enterprises security policies beware that, atleast where I have worked, that is grounds for dismissal.

3 Likes
261

Have you tested this? We actually don’t use Python’s own networking stack at all, it just uses the native one with (relatively) default settings, so if Windows can authenticate, the install manager should be able to. (Side note, I’d love to make Python use the native stack as well, but unfortunately OpenSSL is the public API and Windows doesn’t provide an equivalent.)