# Python3.11 seems to break venv created from non-system python with zipped stdlib

**URL:** <https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874>\
**Category:** Core Development\
**Created:** [November 8, 2022, 5:40pm UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874 "2022-11-08T17:40:41Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![kkpattern](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/kkpattern/32/9704_2.png) [@kkpattern](https://discuss.python.org/u/kkpattern)\
**Post date:** [November 8, 2022, 5:40pm UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/1 "2022-11-08T17:40:41Z")

</div>

We have a python built from source with a zipped stdlib in our project. The directory layout looks like this:

```auto
/path/to/our/python
    /bin
        /python
    /lib
        /python310.zip

```

Then we will create venv from this python to use.  
After we upgraded to python3.11, we found the venv created from this python is broken. The reason is that it failed to find the zipped stdlib. It’s not in the `pythonpath`.

After some investigation, we found this line in `Modules/getpath.py` causes the problem.

```python
650 elif build_prefix or venv_prefix:                                           
651 # QUIRK: POSIX uses the default prefix when in the build directory      
652 # or a venv                                                             
653 pythonpath.append(joinpath(PREFIX, ZIP_LANDMARK))

```

Here, if the python is in a venv, the default prefix is used, even if the `joinpath(prefix, ZIP_LANDMARK)` exists.

I think a possible fix is to test `isfile(joinpath(prefix, ZIP_LANDMARK))` first. If true, we add it to the `pythonpath`, otherwise we use the default prefix.

If this behavior is not intended and the above fix is OK, I’d like to create a PR.

---

<div class="post-metadata">

**Author:** ![brettcannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brettcannon/32/34895_2.png) [@brettcannon](https://discuss.python.org/u/brettcannon)\
**Post date:** [November 8, 2022, 11:59pm UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/2 "2022-11-08T23:59:47Z")

</div>

> [@kkpattern](#):
>
> After some investigation, we found this line in `Modules/getpath.py` causes the problem.

What does the git history on those lines say about the reasoning for them?

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [November 9, 2022, 12:44am UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/3 "2022-11-09T00:44:11Z")

</div>

> [@brettcannon](#):
>
> What does the git history on those lines say about the reasoning for them?

Nothing. You need to pull up the old `getpath.c` file for the reasoning that led to this comment.

Looking at [the 3.10 implementation](https://github.com/python/cpython/blob/3.10/Modules/getpath.c) again, it seems that `search_for_prefix` is always going to fail to locate the venv itself, and will only find the global install of Python if it happens to be in the same subtree. (I spent longer mapping this out the first time, so I may be missing something right now - it’s also late and I’m tired…)

But since it worked before, it’s likely that the logic was untangled incorrectly here. It’s marked as a quirk because it seems like a bad idea, but represents what the old code did - the idea is to make deliberate changes over time with proper deprecation to remove the quirks.

So I’d say, either demonstrate exactly what the old behaviour was and replicate it correctly here (i.e. a bugfix), or propose a change to the resolution behaviour (feature). Incidentally, I’ve already merged a change to the Windows behaviour for zip files to _stop_ searching, which was incorrectly translated from the old module that assumed a single location (security).

---

<div class="post-metadata">

**Author:** ![kkpattern](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/kkpattern/32/9704_2.png) [@kkpattern](https://discuss.python.org/u/kkpattern)\
**Post date:** [November 9, 2022, 2:09am UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/4 "2022-11-09T02:09:18Z")

</div>

> [@steve.dower](#):
>
> But since it worked before, it’s likely that the logic was untangled incorrectly here. It’s marked as a quirk because it seems like a bad idea, but represents what the old code did - the idea is to make deliberate changes over time with proper deprecation to remove the quirks.

I can spend some time to find out why the 3.10 implementation works for us first.

---

<div class="post-metadata">

**Author:** ![kkpattern](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/kkpattern/32/9704_2.png) [@kkpattern](https://discuss.python.org/u/kkpattern)\
**Post date:** [November 9, 2022, 4:14am UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/5 "2022-11-09T04:14:41Z")

</div>

So in python3.10, first the `argv0_path` is correctly calculated in `calculate_argv0_path` for our case.

Then, in [`search_for_prefix`](https://github.com/python/cpython/blob/3.10/Modules/getpath.c#L517) the prefix is calculated from `argv0_path`(In our case the `lib/python3.10/os.py` is deliberately kept outside from the zip file as a landmark):

```nohighlight
   516 /* Search from argv0_path, until root is found */
   517 status = copy_absolute(prefix, calculate->argv0_path, prefix_len);
   518 if (_PyStatus_EXCEPTION(status)) {
   519 return status;
   520 }
   521
   522 do {
   523 /* Path: <argv0_path or substring> / <lib_python> / LANDMARK */
   524 size_t n = wcslen(prefix);
   525 status = joinpath(prefix, calculate->lib_python, prefix_len);
   526 if (_PyStatus_EXCEPTION(status)) {
   527 return status;
   528 }
   529
   530 int module;
   531 status = ismodule(prefix, &module);
   532 if (_PyStatus_EXCEPTION(status)) {
   533 return status;
   534 }
   535 if (module) {
   536 *found = 1;
   537 return _PyStatus_OK();
   538 }
   539 prefix[n] = L'\0';
   540 reduce(prefix);
   541 } while (prefix[0]);

```

Finally in [`calculate_zip_path`](https://github.com/python/cpython/blob/v3.10.8/Modules/getpath.c#L1292), since the `prefix` is found, it’s used to calculate the zip path. Which is correct for our case.

It seems in python3.10, as long as the prefix is found, the zip path is always calculated from the prefix.

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [November 9, 2022, 8:26pm UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/6 "2022-11-09T20:26:13Z")

</div>

Yeah, provided `os.py` is found, it does seem to work out. So I guess I missed something in the logic and there are no tests for it.

It seems that just changing it to always use `prefix` instead of `PREFIX` is the closest way to get back to the old behaviour. It doesn’t search, and it doesn’t check whether the file exists. If we want to do that, it would be a 3.12-only change (unless there’s a shown security risk, which I don’t think there is because there isn’t a search).

---

<div class="post-metadata">

**Author:** ![kkpattern](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/kkpattern/32/9704_2.png) [@kkpattern](https://discuss.python.org/u/kkpattern)\
**Post date:** [November 10, 2022, 4:42am UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/7 "2022-11-10T04:42:17Z")

</div>

I think 3.12-only change is OK. Since we’re already build python from source, we can workaround this with a patch in 3.11.

For the change, I did a test. First I changed the zip file path calculation to this:

```python
    if prefix:
        pythonpath.append(joinpath(prefix, ZIP_LANDMARK))
    else:
        pythonpath.append(joinpath(PREFIX, ZIP_LANDMARK))

```

Then, two test cases will fail. test\_embed and test\_getpath.

Then I tried simply removing the venv condition like this:

```python
    # Then add the default zip file
    if os_name == 'nt':
        # QUIRK: Windows uses the library directory rather than the prefix
        if library:
            library_dir = dirname(library)
        else:
            library_dir = executable_dir
        pythonpath.append(joinpath(library_dir, ZIP_LANDMARK))
    elif build_prefix:
        # QUIRK: POSIX uses the default prefix when in the build directory
        pythonpath.append(joinpath(PREFIX, ZIP_LANDMARK))
    else:
        pythonpath.append(joinpath(prefix, ZIP_LANDMARK))

```

now all tests passed. This may be the simplest change?

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [November 10, 2022, 4:00pm UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/8 "2022-11-10T16:00:47Z")

</div>

> [@kkpattern](#):
>
> Then I tried simply removing the venv condition like this

Yeah, that may be the best option, provided it still works for a non-installed build.

`venv` doesn’t really support layouts other than our default install, so if you want it to work with an embedded runtime you probably need to be patching other things anyway. If I were structuring all this from scratch, I’d be putting the venv logic into `Programs/python.c` rather than in `libpython` itself, but we’re well past the point where we can make that design change.

---

<div class="post-metadata">

**Author:** ![kkpattern](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/kkpattern/32/9704_2.png) [@kkpattern](https://discuss.python.org/u/kkpattern)\
**Post date:** [November 10, 2022, 4:47pm UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/9 "2022-11-10T16:47:12Z")

</div>

> [@steve.dower](#):
>
> Yeah, that may be the best option, provided it still works for a non-installed build.

Cool. Do you need me to create a PR for this? I can also create a new case in test\_getpath.py for this.

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [November 10, 2022, 5:04pm UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/10 "2022-11-10T17:04:14Z")

</div>

That would be awesome if you’d like to. Put an `@zooba` in the issue and I’ll review and merge for you.

---

<div class="post-metadata">

**Author:** ![kkpattern](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/kkpattern/32/9704_2.png) [@kkpattern](https://discuss.python.org/u/kkpattern)\
**Post date:** [November 10, 2022, 5:13pm UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/11 "2022-11-10T17:13:32Z")

</div>

Great. One thing is that I’d like to verify the new test case will pass in v3.10.8, so I can make sure the test case itself is correct. But seems there is no `Lib/test/test_getpath.py` before 3.11. Do I have a way to verify this case in older python versions?

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [November 10, 2022, 5:15pm UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/12 "2022-11-10T17:15:26Z")

</div>

Maybe a test in `test_venv` makes more sense for the overall scenario? `test_getpath` is very much about simulating environments and testing the logic.

We can definitely backport a (passing) test back to 3.10.

---

<div class="post-metadata">

**Author:** ![kkpattern](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/kkpattern/32/9704_2.png) [@kkpattern](https://discuss.python.org/u/kkpattern)\
**Post date:** [November 10, 2022, 5:29pm UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/13 "2022-11-10T17:29:18Z")

</div>

Ok. I will add test cases in both `test_venv` and `test_getpath` and verify the new `test_venv` case with 3.10.

---

<div class="post-metadata">

**Author:** ![kkpattern](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/kkpattern/32/9704_2.png) [@kkpattern](https://discuss.python.org/u/kkpattern)\
**Post date:** [November 11, 2022, 6:54am UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/14 "2022-11-11T06:54:28Z")

</div>

I have pushed my change to GitHub. I have verified the new `test_venv` test case will pass in v3.10.8 and fail in v3.11.0. But it’s a little bit tricky, I have to create a fake non-installed python first in order to test the behavior.

Plus I haven’t tested it on PC, so I haven’t created the PR yet. You can review the code [here](https://github.com/python/cpython/compare/main...kkpattern:cpython:fix_zippath_venv_non_installed#diff-d1cebff01d06039d87ad2fa988261142c49a9e097f017e0c70244433bd4c4d6eR540) for now. If you think the change is OK I can create the PR and continue the discussion there.

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [November 11, 2022, 10:38am UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/15 "2022-11-11T10:38:57Z")

</div>

The principle looks fine, so let’s get the issue and PRs created and we can iterate there (with the benefit of automated builds on all platforms).

---

<div class="post-metadata">

**Author:** ![tjryan](https://avatars.discourse-cdn.com/v4/letter/t/7ba0ec/32.png) [@tjryan](https://discuss.python.org/u/tjryan)\
**Post date:** [November 14, 2022, 6:44pm UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/16 "2022-11-14T18:44:22Z")

</div>

Most of this topic just sort of flies over my head, but when I upgraded from Python 10.8 to 11.0, my PyCharm Community IDE quit stopping at breakpoints when I was in debug. Nothing I did could make it halt on a breakpoint. I finally deleted 3.11 and reloaded 3.10, and all was fine. The error messages kept saying that there were internal errors in the compilation of 3.11, over which I had no control

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [November 14, 2022, 10:31pm UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/17 "2022-11-14T22:31:50Z")

</div>

That’s far more likely to be due to other changes in Python 3.11. I’d suggest starting with JetBrains support, as they’re the ones who control their debugger.

---

<div class="post-metadata">

**Author:** ![tjryan](https://avatars.discourse-cdn.com/v4/letter/t/7ba0ec/32.png) [@tjryan](https://discuss.python.org/u/tjryan)\
**Post date:** [November 15, 2022, 1:49pm UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/18 "2022-11-15T13:49:03Z")

</div>

Steve

For now, I’m just sticking to 3.10 until 3.11 gets some runtime under its belt. I really had no reason to upgrade other than “ooh – shiny!”

Thanks.

---

<div class="post-metadata">

**Author:** ![tpudlik](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tpudlik/32/15872_2.png) [@tpudlik](https://discuss.python.org/u/tpudlik)\
**Post date:** [November 14, 2023, 7:06pm UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/19 "2023-11-14T19:06:04Z")

</div>

Was the PR for this ever created?

---

<div class="post-metadata">

**Author:** ![kkpattern](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/kkpattern/32/9704_2.png) [@kkpattern](https://discuss.python.org/u/kkpattern)\
**Post date:** [November 15, 2023, 6:35am UTC](https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874/20 "2023-11-15T06:35:38Z")

</div>

Yes, this is the PR: [https://github.com/python/cpython/pull/99371](https://github.com/python/cpython/pull/99371)
