PEP 648: Extensible customizations of the interpreter at startup

This does not solve the problem of virtualenv. In this case, virtualenv is never imported but is at the moment relying on some startup time initialization script to make sure the created environment is good to go… At the moment e.g. we patch some distutils oddities virtualenv/_virtualenv.py at main · pypa/virtualenv · GitHub, and while distutils is going away they are bound to be different problems. Also is used to fix issues such as conda commands as Python3 subprocesses inside a virtualenv fail under macOS · Issue #1704 · pypa/virtualenv · GitHub

1 Like

You’re creating the environment. You can put a file anywhere you like in that case, and users have already opted-in to the modified behaviour.

The better example is an uber-distro where seventeen different packages think they need to load at startup, but in most cases they’re never going to be used. In this case, we want to push it to zero per-package, while the distro maintainer is allowed/able/encouraged to add their patches at startup.

1 Like

This is a very good point, and is currently an open question. There’s nothing in the existing wheel spec that allows files to be put into a special __sitecistomize__ directory, so the PEP needs to include an update to the wheel spec, which would need to be implemented by installers.

That’s probably how it would work, but pip currently hard codes the valid schemes as ‘platlib’, ‘purelib’, ‘headers’, ‘scripts’, and ‘data’, which match to sysconfig schemes. I’d expect the following sort of change to be needed:

  1. A sysconfig scheme for sitecustomize gets added.
  2. The wheel spec gets tightened up to explicitly state that sysconfig schemes should be used in the data directory.
  3. Installers (pip, installer, wheel) get updated to implement the new spec.

At the moment, I don’t think the .data directory is used for much (mostly scripts, maybe headers) so the current implementations are probably not well tested. It may even be that only pip handles this at all, right now.

Also, something will need to be done in build backends (setuptools, flit, poetry, …) to allow projects to specify these sorts of files. That’s not something that’s been discussed at all yet.

IMO, the PEP needs a lot more discussion of how the packaging ecosystem will handle these things. Otherwise, I’d expect the result to be that the PEP specifies the mechanism, but the packaging ecosystem doesn’t change to support installing startup files like this, and the new feature will remain unused. People will continue to have to use .pth files, because they “just work” even if they are a bit ugly, and we end up not moving forward.

Conversely, all of the mechanisms needed for an entry point based solution already exist, so there’s nothing to do on the packaging side. The major exception would be the performance issue @bernatgabor pointed out, which is a definite problem with this approach. Whether it can be alleviated in some way remains to be seen, but optimising entry points would be a general benefit, not just limited to interpreter startup.

2 Likes

I will also say that there’s a lot of initial I/O going on due to imports during startup anyway, so I would be interested in seeing what the actual overhead is for using entry points before we write it off due to potential perf concerns versus actual measured perf concerns.

1 Like

On my top of the line company MacBook Pro, checked against the dev environment of tox4:

❯ .tox/4/dev/bin/python --version
Python 3.9.1
❯ .tox/4/dev/bin/python -m pip list | wc -l
89
❯ hyperfine '.tox/4/dev/bin/python  -c ""'
Benchmark #1: .tox/4/dev/bin/python  -c ""
  Time (mean ± σ):      42.9 ms ±   3.5 ms    [User: 31.3 ms, System: 9.5 ms]
  Range (min … max):    36.2 ms …  52.4 ms    58 runs
❯ hyperfine '.tox/4/dev/bin/python  -c "from importlib.metadata import entry_points; eps = entry_points()"' 
Benchmark #1: .tox/4/dev/bin/python  -c "from importlib.metadata import entry_points; eps = entry_points()"
  Time (mean ± σ):      87.3 ms ±  13.3 ms    [User: 60.8 ms, System: 15.6 ms]
  Range (min … max):    67.4 ms … 122.8 ms    23 runs

That’s an increase from 42 to 87ms, practically double-ing the startup time (for having 90 packages installed); so I’d say it’s significant and not just perceived overhead. I expect these numbers to be much-much worse on a network filesystem with 400+ packages installed (my company has many such servers running).

Here’s an example with 200 packages installed on a less powerful machine:

  # no entrypoint load
  Time (mean ± σ):     126.3 ms ±   9.1 ms    [User: 91.7 ms, System: 19.1 ms]
  Range (min … max):   108.2 ms … 145.9 ms    20 runs
   # entrypoint load
  Time (mean ± σ):     262.7 ms ±  30.5 ms    [User: 200.8 ms, System: 32.3 ms]
  Range (min … max):   212.4 ms … 326.3 ms    10 runs
2 Likes

I’d like to see a performance comparison against the __sitecustomize__ approach, too, as that will also add a bunch of directory scans.

1 Like

I must be missing something, isn’t that a single folder scan? The __sitecustomize__ one?

PS. I’m comparing here the case when there are no customization scripts available, so that we don’t regress always just because we introduce a new feature.

1 Like

I must be missing something, isn’t that a single folder scan? The __sitecustomize__ one?

I assumed it would look through all site directories for __sitecustomize__, so not quite that simple. The last time I checked, the PEP wasn’t completely clear (there’s no actual specification section, the spec is in the “Rationale” section).

I ran a quick test:

❯ type .\example.py
import os, site
for p in site.getsitepackages():
    cust = os.path.join(p, '__sitecustomize__')
    if os.path.isdir(cust):
        for custfile in os.path.listdir(cust):
            x = os.path.join(cust, custfile)

❯ hyperfine "py -c ''"
Benchmark #1: py -c ''
  Time (mean ± σ):     106.0 ms ±  20.1 ms    [User: 5.3 ms, System: 6.4 ms]
  Range (min … max):    68.5 ms … 149.7 ms    19 runs

❯ hyperfine "py example.py"
Benchmark #1: py example.py
  Time (mean ± σ):     111.4 ms ±  16.7 ms    [User: 5.6 ms, System: 10.0 ms]
  Range (min … max):    96.6 ms … 155.1 ms    19 runs

So there isn’t much overhead.

But I’m not trying to say that there is no overhead, just that the functionality available for entry points is more complete, and already available (so possibly has less risk of bugs). We need to judge the trade-off. And it’s possible that entry points could be optimised, which would benefit much more than just this feature.

1 Like

If this is true, let’s do this before we make a decision. @jaraco @pablogsal can probably answer exactly how much more entry-point discovery can be made faster. Other than trying to parallelize the process can’t see many inefficiencies in there though. What it does is fairly simple it just needs to do it for package/folder number of times.

1 Like

Also how would this work for customization added by system administrator/virtualenv? Would they need to package their script, and then install that package? If so, this would then show up under pip list, is that a feature or bug? Would allow users to uninstall those, is that something we want?

1 Like

In order to emulate all the different means of using sitecustomize.py
and usercustomize.py, such dirs must be scanned potentially in all
dirs on sys.path. See further up for this discussion.

Just having one such dir somewhere won’t allow having both site
customizations (e.g. company wide adjustments such as audit hooks)
and user level customizations (e.g. adding debugging hooks).

For package level customizations, I suppose such a dir would have
to be placed into the site-packages dir where pip installs
the package and pip would have to grow a mechanism for placing
the file there.

Now, going back to the original point about .pth files, I think
it would be useful to collect actual use cases.

The only use cases I can think of are “installing” package resources
such as codecs without having to actually import a package and
overriding packages, e.g. to replace a stdlib module with an optimized
version. I’m sure those cases can be addressed without adding
extra stat calls to Python’s startup.

1 Like

Yes, but this means potentially 3 folders, a constant number, not the number of package folders, which is dynamic. The latter can grow to hundreds fairly quickly, while the first remains up to 3.

What about the virtualenv use case? Or company-wide metric collection (e.g. nr of interpreter startup)? Also rather than leave it opaque, please do suggest alternative solutions, so we know what better ways are we missing. Thanks :+1:

1 Like

For the purposes of this discussion, could you expand on the virtualenv use case? The only description I could find in the thread was

which doesn’t give much detail. Also, given that PEP 632 to remove distutils from the stdlib has been approved, will the virtualenv use case still be applicable by the time virtualenv could switch to __sitecustomize__?

1 Like

I’m specifically just addressing cases where a package installs a .pth
file to achieve some form of customization.

The other cases you mentioned can already be had using the existing
sitecustomize.py and (to some extent) usercustomize.py scripts, so
would not need .pth files.

Solutions for the above two .pth use cases would be:

  1. “installing” package resources: my preference would be to make
    this an explicit choice by the user via an env var such as
    PYTHONRESOURCES which then lists a number of Python modules to
    unconditionally import during startup.

  2. replacing packages: I’d leave this to the user via the standard
    PYTHONPATH in order to make this an explicit choice as well, e.g. by
    adding a dir with such packages and having pip install those
    packages into that dir instead of site-packages.

venvs could simplify this by adding special options to pyvenv.cfg.

1 Like

wow, lot of activity to answer to here. Thanks all for the discussion. Let me go one by one:

In order to emulate all the different means of using sitecustomize.py
and usercustomize.py, such dirs must be scanned potentially in all
dirs on sys.path. See further up for this discussion.

This is already happening as we need to scan pth files, the pep includes an example implementation that shows how that cost would be minimal as we can just include a check for __sitecuszomize__ when we are checking the the filename ends in pth. See here.

Also how would this work for customization added by system administrator/virtualenv?

As far as I understand, the entry-points solutions won’t work for this case. Which is IMHO a strength of this pep. It offers a single way to customize startup.

IMO, the PEP needs a lot more discussion of how the packaging ecosystem will handle these things. Otherwise, I’d expect the result to be that the PEP specifies the mechanism, but the packaging ecosystem doesn’t change to support installing startup files like this, and the new feature will remain unused. People will continue to have to use .pth files, because they “just work” even if they are a bit ugly, and we end up not moving forward.

I disagree here though. There is currently no support for pth files in build backends already and the ones that need it are already using it. The PEP mentions that it wants to work with build backends to support this, but with this PEP, adding a file to sitecustomize is as hard/simple as adding a pth file IIUC.

On my top of the line company MacBook Pro, checked against the dev environment of tox4:

Thanks a lot for the benchmarks @bernatgabor ! It is true indeed that this PEP proposal adds almost no overhead (just an if check for a filename) when the feature is not used, and minimal when used (scanning a folder) which I think it is quite a strong point of it, given the recent importance that has been put into startup time.

1 Like

OK, so you’re saying that the PEP as it stands will not cover the case of packages wanting to register an “on interpreter startup” hook, and you consider that OK as the existing solution (.pth files) is also not covered by existing package tools¹. That’s a reasonable position to take, although maybe you could be a bit more explicit about that in the PEP?

¹ I’m a little surprised that .pth files can’t be put in wheels right now, but I don’t use them much and I don’t have time to check at the moment, so I’ll take your word for it.

1 Like

Indeed, in importlib.metadata, the discovery has been highly optimized already, but for the general case. The backport CI includes a benchmark test that measures the speed of discovering metadata for a single package with a number of packages (ipython’s dependencies) are installed. I’d be happy to see that benchmark expanded to include more common use cases (loading entry points is a good one).

There were some recent performance enhancements, but it’s going to be increasingly difficult to squeeze more performance out of metadata discovery.

And as Bernát points out, the extraction of entry points once the metadata is discovered is pretty straightforward (open file, read lines, parse each line). There may be some operations that can be trimmed or parallelized, but it’s unlikely you’ll find an inefficient repeated operation (for a single load of entry points).

2 Likes

Just re-stating that I think it would be better to encourage libraries to not inject things at all, leaving it up to the user to opt-in to behaviors. Making it easier for libraries to inject code doesn’t seem beneficial overall.

Since this has come up a few times, I also can’t think of reasons for libraries to inject pth files (or __sitecustomize__) except to register some things like codecs very early. I don’t think any of the uses I can think of, including that one, need pth files though. Any import side effects they produce could be produced by the user explicitly opting-in to behaviors by adding imports at the start of their code (or in sitecustomize.py or usercustomize.py, or a run.py script). This seems much better in terms of orgs being able to control what happens at startup regardless of what dependencies are installed.

I know virtualenv was brought up as a special case, but it seems to me that it being special means it’s in the position to inject / patch whatever it wants when building the env anyway, so I’m not quite sure how this PEP would matter for it.

3 Likes

Also, just looked at the docs on sitecustomize.py and usercustomize.py, and it’s possible for any package to install such files, which seems like a security issue waiting to happen. I just tried adding py_modules = sitecustomize to one of my projects, and it installed and executed just fine. It even overwrote an existing sitecustomize I had added manually.

The more I think about this, and seeing input from @mauve about wanting to control their environment, the less appealing the current or proposed behavior seems.

2 Likes

virtualenv is not in a position to control the global python, so any customization must be within the virtual environments python. As long as there exists a pth file support it can do so. I had the impression this PEP was a step to remove that. If that would happen there’d be no way to run some code on startup (unless does crazy things like generate shim executables and replace the link/copied python executable with those).

1 Like