Identifying gaps in Python distribution

Packaging
21

I agree that Windows is in a good spot, however I’m considering recommending in the Hatch documentation to not use the store Python because the sandboxing limitations cause issues for users.

2 Likes
22

Good point, and that’s something we need to address - I haven’t used the Windows Store build myself, and while I was vaguely aware of the restrictions, I don’t know the details. We really can’t start a trend of making the Windows Store distribution a “second class citizen” without engaging with the core team who are promoting it in the core documentation. But if the limitations are simply unresolvable because “that’s how the Windows Store works” then I think we need to consider the messaging around the store distribution - even if that means framing it as some sort of “end user” variation that can’t be used for general development.

@smm this is definitely a “gap” that I think we need to record. Can you engage with the core devs to find out how the Windows Store distribution is supported and how we engage with the appropriate people to discuss how to resolve this sort of issue? If, as I suspect, the Store distribution is basically supported by @steve.dower on his own, we should be discussing the sustainability of that distribution (and by “we”, I mostly mean the core devs, as it’s not a packaging matter except in terms of that distribution being a target where the packaging community needs to work out how or if we support it).

2 Likes
23

Most of the limitations I’m aware of boil down to:

  • %PATH% is not searched for DLLs (and Tensorflow refuses to fix this assumption in their own code)
  • files written into %AppData% (and hence %TEMP%) are not always visible to other apps[1]
  • the Scripts directory cannot be added to %PATH% on install - it has to be done manually (or use a venv, or python -m ...)
  • the global (python|pip|idle)[X[.Y]].exe shortcuts are not real executables and so anything that inspects them before running them (e.g. Cygwin) will fail

I’m already following up the last two with the Windows team to try and get things improved, but that’s a longer term plan. I’ve also been experimenting with providing a Python install manager via the Store that would be able to replace each individual app we currently publish, but I’m hesitant on that one because I’m not even sure we want to be pushing a manager that doesn’t include environment management or package management, and I don’t want to personally own one that does include those![2]

As for publishing updates, it should be documented enough in the repo (coupled with the standard docs), and the publishing accounts are linked to the PSF not to me. Hopefully Ee still has access to add alternate users, but I certainly do, and probably nobody at Microsoft has any particular access because to them it’s just a customer/partner.

I’ve been working (quietly, at least in this forum) to make the publishing process easier to hand off. At this point, 3-4 members of the release team have the permissions to deploy a signed release all the way to python.org and Nuget, and a few PSF employees have permissions to add/remove users in that group. The only thing I’m “essential” for right now is an ARM64 VM for PGO, but that can be skipped in the build with a checkbox. So hopefully that backup plan is fine, though we haven’t had cause to test it yet.

  1. The rules here are somewhat complex, and have changed a few times, but basically you can’t write stuff into %AppData% and expect other apps to search and find them. Jupyter suffers here when you register its shared kernel directory first from the Store, and then register it again from a non-Store install, because the two “shared” directories end up in different places. ↩︎

  2. And also unsure whether it makes more sense to saddle the volunteer group with such a tool, or get Microsoft to own the responsibility for their own platform. ↩︎

24

Thanks - none of those sound like they would affect hatch - @ofek what’s the problem hatch has with the store Python?

Also, “get the Windows team to fix Windows” isn’t exactly a sustainable approach. You’re in a rather uniquely privileged position there :slightly_smiling_face: And in any case, I don’t think we should get into a position where we are dependent on Microsoft’s goals aligning with those of the Python community[1].

It’s not the publishing side that worries me, it’s the support side. Who can deal with bug reports that are only reproducible in the store build? Who knows enough about store app limitations to go “oh yeah, that sounds like a sandboxing issue, let’s try X”?

This matters from a packaging perspective because we’re very reliant on being able to say “this sounds like a problem with your Python distro, go talk to your distro support team”, so that we only need to support “official Python” out of the box. It’s also very much a question of “what is officially Python” - we’ve had debates about whether Python shipped without venv is something tools must support, do we now have to question whether, if Python writes a file to the temp directory and hands the filename to another process, and that process can’t see the file, then that’s a Python bug, or something that tools like pip, virtualenv, etc, need to allow for when writing their code.

@smm This might be another case where you want to say the problem is noted and details can be deferred for later. But if so, can you create a public document maintaining a list of “the stuff we picked up on and said we’d defer details”? It doesn’t have to be complicated, just a quick summary of the issue and a link to a reasonably relevant post where it was raised/discussed in this thread would be plenty. Just enough to let people know that the item won’t be forgotten[2].

  1. I very definitely don’t want to spark a “corporate control” debate, but it’s something we should consider. ↩︎

  2. This is something I would find very useful, to help me “let go” of topics where I still have more to say. ↩︎

1 Like
25

Certainly more than just me, but I don’t think anyone on the core dev team has taken much of an interest. The limitations are public knowledge and linked from the docs.

Aside, but generally if it hands the filename out things will be fine. The resolution of %TEMP% changes, but the resolved name is accessible just fine from any other app (with access to the user’s files). It’s only a problem for apps that assume other apps will be able to locate their temp/appdata files without assistance or cooperation (and it’s “app” at the level of “pythonX.Y.exe”, so other Python scripts will all see the same files).

1 Like
26

I’d like to see better guidance on the /purposes/ of the distributions,

This is pretty important… I wonder if a “Not sure? How to choose the
right Python distribution” link could help.

and I’d prefer the nuget distribution to not require a proprietary tool
and just be a zipfile (I imagine that’s why people go for the embedded
version rather than the nuget one). But that’s just details.

Guess I’m missing something… the nuget client is open source, right?
And a .nupkg file is just a zipfile, right?

27

The code is available, and the protocol is no secret. It’s written in .NET though, which makes it not “proper” open source to some people, and I’m pretty sure all the maintainers are Microsoft employees, which makes it proprietary to many other people.[1]

Yep. Just a zip file with metadata, signing, and a bit of internal structure. Not unlike a wheel…

  1. Not entirely unreasonably. “Open source” in that context is usually meant to imply “open governance”. ↩︎

28

Re the Windows Store distribution, it happens that I once wrote a lengthy Stack Overflow answer describing how things went wrong. A lot of people have been irritated by the fact that the Store distribution comes with bootstrap/prompt apps that are on the PATH ahead of time, whereas standard installations default to not modifying PATH (because of py), resulting in prompts to install Python even after doing so.

1 Like
29

Is it (for both of those points)? I didn’t know that. I still think that having a file with the standard .zip extension, available from a URL that can be accessed with standard tools like curl, or Python itself, would be much more broadly useful (and much more commonly used!)

30

This wasn’t “us” - this was Microsoft choosing to do this. It results in 12K new downloads/day. (Technically “acquisitions”, which means exactly one per user. And doesn’t count the ~hundred people/day who end up on the page but choose not to install it.)

(Most of the really dumb looking issues people encountered were due to a bug in the .NET Core installer that interacted poorly with Windows Update in a way that I never managed to solve, but went away after .NET Core fixed their installer. If this sounds incredibly unrelated, hopefully that indicates just how obscure the issue was!)

If there’s a consensus that it should be removed, I’m sure we can make that happen, but the replacement will be the old “Command not found” error. Personally, I think that would be a big step backwards, and I know plenty of others who would agree.

31

There’s also winget which I believe ships with Windows now.

To be clear, that’s a Windows thing and not a Store thing.

32

Also they’re not on the PATH, they’re a Windows feature called “App execution aliases” which is why they don’t break applications that do look up Python on the PATH.

33

Continually nagging people who chose to install python through a different avenue is IMO much worse than “command not found”. The latter is pretty self-explanatory and can be fixed by the user, the former is surprising/annoying (“I’ve already got python installed?!”) and can’t be fixed by the user.

1 Like
34

In my book, it’s fine that Microsoft chooses to offer their distribution in that manner (I can see the antitrust argument, but I’m sure there are numerous differences between this case and the old IE lawsuits). I agree that for most users it’s better than getting “Command not found” on a system where Python hasn’t been installed.

What bothers me is that it commonly still shows up after a user (either an experienced one who goes directly to python.org due to previous experience, or a newer one who is following a tutorial that includes explicit install instructions) has installed Python following recommended practice (i.e. not adding Python to PATH, because py will take care of things) when that result should have been easily foreseen (since there is such a long history here).

I think the problem could have been solved on either side, although the Windows-side solution would have been simpler and more… responsible:

  • To solve it on Python’s side: change the default PATH recommendation. Have the installer scan PATH first for existing Python installations (for example, by shelling out python and seeing if it launches a Python process). If one is found, add a step to the installer advising the user of this, and offer to remove the corresponding PATH entries and ensure py is installed. If not found, add this first new Python to PATH by default, but install py as well by default. This approach might be awkward because of the need to communicate clearly with the user; but hopefully this situation would only be encountered by more sophisticated users anyway.

  • To solve it on Windows’ side: as I suggested on Stack Overflow - instead of just dropping a magical Store shortcut link on the path, drop a wrapper executable. If the wrapper detects py (and it only has to look in one, hard-coded place), the wrapper could then just redirect to py with no additional arguments, and otherwise bring up the Store.

Independent of all of that, it would be nice if py offered some method for explicitly registering Python installations that it doesn’t already know about.

1 Like
35

Well, if you insist

5 Likes
36

Oh and yeah, the nupkg API and file format is pretty trivial to work with, even without any special tooling. This is all you need to programmatically fetch all of the Python nupkgs and repack them into something more wheel-like: pybi-tools/do-win.py at main · njsmith/pybi-tools · GitHub

37

All the gaps identified so far have been noted. The doc will be shared Wed/Thurs next week.

38

Is there any reason it can’t be maintained in a form where the “working state” is visible, such as hackmd.io or a GitHub gist? The “we have a discussion and there will be a summary published at the end” approach doesn’t feel as collaborative, and IMO hasn’t worked that well in previous topics of this nature.

7 Likes
39

I have summarized this thread in a gist- python_distribution_recommendations.md · GitHub. Do let me know if I have the technical details wrong or if I have missed anything important.

40

I created a detailed post there some time ago:

the custom location is required for several reasons:

  1. very limited storage in the “system drive” of modern day laptops owing to SSDs
  2. cleaning up system C: drive is scary for normal user and makes system cleanups a nightmare
  3. … whereas installing to custom location maintains a separation of concerns between system & user applications and makes installed software easier to analyse and manage

make-installation-process-friendly-for-custom-locations-non-admin-on-windows