PEP 582 - Python local packages directory

[frostming]

Let me summarize some problems raised recently in this thread that PEP 582 needs to clarify or improve:

1. OS-isolated library paths

@dstufft

Now a observant person would notice that this is essentially a problem of trying to re-use the __pypackages__ across multiple operating systems and/or computers, which is obviously a nonsense thing to do,

The folder structure of the current proposal is not enough to isolate packages between different platforms, a possible solution is to name the directory with platform tags like cp310-win-amd64.

2. System site-packages ignorance

@Kwpolska

If __pypackages__ is present, Python should use packages from it and ignore system site-packages.

Partially agree, but it may cause problems for some “interpreter wrappers”. Let me explain, say we make a “beautiful python”, or bpython, with changes to the output and exception hook to make it output beautiful ANSI colors to the terminal, and distribute it as a python library uploaded to PyPI. A user installs it into the global site-packages with the system python. When he executes bpython /path/to/myscript.py, should system site-packages be ignored? If we are to ignore it, bpython will be broken. But if we don’t, and put __pypackages__ to the sys.path in front of system site-packages, chances are incompatible versions of dependencies in __pypackages__ being prioritized and, again, break bpython(refer to pdm#849). It seems this problem is non trivial to solve without tweaks to the interpreter on how and when site-packages are loaded.

3. Problem of finding __pypackages__

@cjerdonek

With projects though, it seems common to have scripts in directories other than the project root (e.g. scripts/myscript.py). For these cases, my understanding is that running the script wouldn’t find __pypackages__.

I agree that this proposal must be extended with how an interpreter looks for the __pypackages__ to load. It shouldn’t be restricted to the same directory in which the script resides. The practice of PDM is to find the current directory and its recursive ancestors at a configurable max depth.

4. Project-level __pypackages__

@methane

This is very common setup. I read PEP 582 but I can not understand these questions:

• How Python scripts in tools/ can use __pypackages__ in project root?
• How to add myapp/ in import path?
  • Do I need to make myapp installable package and use pip install -e .?

In addition to the last point, a project-level __pypackages__ must be seen by all scripts and modules inside it. A project can be defined by a pyproject.toml in the root. This is also what PDM is doing at present.

5. bin handling

@dstufft

I don’t think PEP 582 discusses how to handle bin/ ~at all

I highly suggest PEP 582 to consider this, making __pypackages__ a full install scheme in sysconfig, with bin, includes, lib, and others. It seems the current PEP only focuses on running a standalone script, and ignores usages inside a project, while in the latter case, it is common to install binaries as well as libraries into the project-level __pypackages__, and users may expect them to run with pip run <executable>

P.S. For all the problems mentioned above, PDM has solved 3,4,5, partially solved 1, and hasn’t solved 2.