Question about pip build isolation in special scenario

petersuter · June 22, 2023, 4:22pm

I have this (maybe quite exotic) scenario:

Download and unzip https://www.python.org/ftp/python/3.11.4/python-3.11.4-embed-amd64.zip
- (I know this is not the recommended normal Python for normal users.)
Download and run https://bootstrap.pypa.io/get-pip.py
- (I know this is maybe not considered supported.)
Uncomment import site in python311._pth
- (This makes pip work very well in that scenario.)
Create PYTHONPATH.pth containing import os, sys; sys.path.extend(os.environ.get('PYTHONPATH','').split(';'))
- My question is about this part.
Run python -m pip install jax==0.4.12 and get this error:

Collecting jax==0.4.12
  Using cached jax-0.4.12.tar.gz (1.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Python311\Lib\site-packages\pip\_internal\cli\base_command.py", line 169, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\cli\req_command.py", line 248, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\commands\install.py", line 377, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "C:\Python311\Lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "C:\Python311\Lib\site-packages\pip\_vendor\resolvelib\structs.py", line 156, in __bool__
    return bool(self._sequence)
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 155, in __bool__
    return any(self)
           ^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 47, in _iter_built
    candidate = func()
                ^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 206, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 293, in __init__
    super().__init__(
  File "C:\Python311\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 156, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 304, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\operations\prepare.py", line 516, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\operations\prepare.py", line 631, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\operations\prepare.py", line 69, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "C:\Python311\Lib\site-packages\pip\_internal\distributions\sdist.py", line 48, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "C:\Python311\Lib\site-packages\pip\_internal\distributions\sdist.py", line 118, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\distributions\sdist.py", line 95, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_internal\utils\misc.py", line 692, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_vendor\pyproject_hooks\_impl.py", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\pip\_vendor\pyproject_hooks\_impl.py", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File "C:\Python311\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "importlib\__init__.py", line 126, in import_module
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Users\WDAGUtilityAccount\AppData\Local\Temp\pip-build-env-kgrn8goc\overlay\Lib\site-packages\setuptools\__init__.py", line 15, in <module>
    import setuptools.version
  File "C:\Users\WDAGUtilityAccount\AppData\Local\Temp\pip-build-env-kgrn8goc\overlay\Lib\site-packages\setuptools\version.py", line 1, in <module>
    from ._importlib import metadata
  File "C:\Users\WDAGUtilityAccount\AppData\Local\Temp\pip-build-env-kgrn8goc\overlay\Lib\site-packages\setuptools\_importlib.py", line 44, in <module>
    import importlib.metadata as metadata  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "importlib\metadata\__init__.py", line 17, in <module>
  File "importlib\metadata\_adapters.py", line 3, in <module>
  File "email\message.py", line 15, in <module>
  File "email\utils.py", line 29, in <module>
  File "socket.py", line 51, in <module>
ModuleNotFoundError: No module named '_socket'

Run python -m pip install jax==0.4.12 --no-build-isolation and it works:

Collecting jax==0.4.12
  Using cached jax-0.4.12.tar.gz (1.3 MB)
  Preparing metadata (pyproject.toml) ... done
Collecting ml-dtypes>=0.1.0 (from jax==0.4.12)
  Downloading ml_dtypes-0.2.0-cp311-cp311-win_amd64.whl (938 kB)
     ---------------------------------------- 938.7/938.7 kB 8.5 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.21 in c:\python311\lib\site-packages (from jax==0.4.12) (1.23.5)
Collecting opt-einsum (from jax==0.4.12)
  Downloading opt_einsum-3.3.0-py3-none-any.whl (65 kB)
     ---------------------------------------- 65.5/65.5 kB ? eta 0:00:00
Requirement already satisfied: scipy>=1.7 in c:\python311\lib\site-packages (from jax==0.4.12) (1.10.1)
Building wheels for collected packages: jax
  Building wheel for jax (pyproject.toml) ... done
  Created wheel for jax: filename=jax-0.4.12-py3-none-any.whl size=1498562 sha256=7b785778d37a1f8bdd9dc58314f2cabbb7b01a7d3fe9a63ea233c25762331848
  Stored in directory: c:\users\wdagutilityaccount\appdata\local\pip\cache\wheels\67\d8\8c\05507d30cc58cab62c405012c887ef3e6f4defa9b6d912a46d
Successfully built jax
Installing collected packages: opt-einsum, ml-dtypes, jax
Successfully installed jax-0.4.12 ml-dtypes-0.2.0 opt-einsum-3.3.0

It also works without --no-build-isolation if the PYTHONPATH.pth is omitted.

My question: What happens exactly? The _socket.pyd file is there. Why is it not found (only) with build isolation + PYTHONPATH.pth? (The PYTHONPATH environment variable is not even set. Why does it still break things?)

I’m just curious and would like to understand better. Thanks.

steve.dower · June 22, 2023, 4:31pm

It’s either because virtual environments also don’t work natively with the embeddable distro, or because it relies on environment variables being passed through.

If you don’t want to use one of the other packages, then you may be able to get it to work by entirely deleting the ._pth file. This will disable all the isolation mechanisms, bringing it closer to what pip expects (though potentially confusing it if you have a regular install of Python on the same machine).

pf_moore · June 22, 2023, 5:12pm

It’s not quite that simple, I suspect. The way pip runs itself in the virtual environment is tricky - you can start from get_runnable_pip() in pip._internal.build_env. What we do there probably doesn’t support some part of your approach.

If you have a normal Python install available as well as your weird one, does normal-python -m pip --python /path/to/weird/python/python.exe install jax==0.4.12 work? It might, simply because it would be running pip from a conventional Python install. Or it might not because pip is still being injected into the weird install…

petersuter · June 23, 2023, 8:06am

Ah, it’s related to virtual environments, that makes sense I guess. Is it because it assume there’s a DLLs folder to copy, but the .pyd files here are directly next to python.exe?

No, same error.

That does seem to work. So the (nonexistent) DLLs folder does not matter after all?

pradyunsg · June 23, 2023, 8:52am

Can you do import importlib.metadata at all in the embedded Python?

petersuter · June 23, 2023, 8:57am

Yes, that works.

steve.dower · June 23, 2023, 11:49am

DLLs folder is optional - it’s just a search path, and the prefix directory is also a search path, and that’s where your pyd files are.

Most likely there’s a PYTHONPATH or PYTHONHOME variable required. The existence of a ._pth file implies isolated mode, which ignores most environment variables. Removing it allows Python to search the whole environment/registry/etc. to find files to execute.

Topic		Replies	Views
Announcement: pip 22.1 release! Packaging release	14	2300	May 31, 2022
PEP517 build isolation on closed networks Packaging	5	1527	June 29, 2020
Announcement: pip 22.3 release! Packaging	0	2435	October 15, 2022
Announcement: pip 23.1 release! Packaging	7	9214	April 29, 2023
Announcement: pip 22.2 release! Packaging	6	4419	August 3, 2022

Question about pip build isolation in special scenario

Related Topics