Announcing microvenv

We had a need for an extremely small (like “can be passed on the command-line via -c” small) library that could create virtual environments when venv isn’t available in the Python extension for VS Code (e.g. Debian users who have not installed python3-venv). So I created microvenv · PyPI to provide that ability.

If size doesn’t matter then virtualenv · PyPI is probably better. But as I said, we are space-sensitive and I’m not about to ask the virtualenv folks to support our niche case, so I decided to create microvenv.

I figured some folks here might have a similar niche need as us, hence me sharing here. I also wrote a blog post explaining how virtual environments work and discuss microvenv at How virtual environments work if that interests you.

13 Likes

Windows is not supported

Just curious, what are the obstacles there?

can be passed on the command-line via -c

How does that work in practice? It doesn’t look like it fits on one line; can you have newlines in -c?

This is great, thanks! It’s a bit sad that it is needed but yeah, not sure what we can do about that.

Which environments other than Debian have you found that need this?

The embeddable package on Windows could potentially also benefit from something like this. more

I just tried out microvenv and it works nicely. Thanks for building this.

1 Like

Thanks Brett, that’s all very interesting. I have some questions, if you don’t mind explaining further.

Presumably VC Code has other OS dependencies. Why not just make venv a hard dependency and require Debian users to install it?

Why pass the setup via -c? Sure, 87 lines of Python code may be small compared to virtualenv, but its not “type this one line command at the shell prompt” small. Why not put the setup in a .py file and then run it without the -c?

You had: py -m venv --without-pip .venv. Why name the virtual environment with a leading dot?

Thank you very much for the explanation of “why virtual environments”, but that seems to contradict your stated position that beginners should use them. All the benefits of environments seem to be aimed at advanced users, people maintaining software that needs to run under multiple versions of the interpreter and dependencies. That’s not the sort of thing most beginners need to worry about. I’ve seen plenty of beginners, maybe one step up from “My First Hello World” but not that much beyond there, struggling to get their virtual environments working. Why nudge beginners towards them?

Thanks in advance.

2 Likes

Perhaps, but you can’t use the embeddable package to install packages, so you may as well just create the venv with the regular install that you also need.

There have been discussions elsewhere, and there’s generally a lot of misunderstanding about the use cases for the embeddable distro. If you want “normal” Python, use the normal installer or the Nuget package, as the docs recommend.

(Installing packages works great. This is for embedding into a larger application as the docs recommend. Happy to discuss elsewhere to not disrupt this discussion.)

If installing packages is fine, then the discussion would be specifically about using microvenv in an embedded runtime, which seems to be on topic.

I don’t have anything to add to it though, other than it won’t come pre-bundled with the embeddable distro under any circumstances.

Ah, I didn’t mean it should be bundled. Since installing packages works, I can just install it. (Unlike venv) :slight_smile:


I tested microvenv on Windows. The main obstacle seems to be the symlink to the python executable. And without venv, the venvlauncher executable is not there either of course.

So I understand why Windows support was excluded. Would you consider adding support if it was possible in <10 more lines of code?

Copying the full real python executable and all the dlls from the same directory instead of symlinking allows successfully installing e.g. numpy into the env. Is that a terrible thing to do (on Windows)?

May as well copy the stdlib as well and you don’t even have a virtual environment. Unless you’re chaining site-packages together, a venv provides no benefit (and much greater risk).

I thought you could do symlinks on modern Windows filesystems?

They have some major security risks, so aren’t enabled by default.

Essentially, if you’re an administrator and you know that you need them, you can use them. But we can’t assume that the average Python developer can use them.

1 Like

I just tested microvenv on Windows. The main problem with using the “python” symlink is that the loader on Windows doesn’t resolve symlinks to the real path of the application directory when searching for dependencies. It can be made to work with a few modifications.

  • Delete the “python3” and “python3.x” symlinks[1].
  • Rename the “python” symlink to “python.exe”.
  • Symlink the DLL dependencies (“python311.dll”, “python3.dll”, “vcruntime140.dll”, and “vcruntime140_1.dll”) in the “Scripts” directory.
  • Instead of the latter step, you could add the base installation directory to the PATH environment variable, but I prefer to avoid that[2].

For example, in the CMD shell:

C:\Temp>py -3.11 -m microvenv
C:\Temp>del .venv\Scripts\python3*.*
C:\Temp>ren .venv\Scripts\python python.exe
C:\Temp>for /f "tokens=*" %p in ('py -3.11 -c "import sys; print(sys.prefix)"') do @set "PREFIX=%p"
C:\Temp>for %f in ("%PREFIX%\*.dll") do @mklink ".venv\Scripts\%~nxf" "%f" 1>nul
C:\Temp>.venv\Scripts\python
Python 3.11.2 (tags/v3.11.2:878ead1, Feb  7 2023, 16:38:35) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'C:\\Temp\\.venv\\Scripts\\python.exe'
>>> sys._base_executable
'C:\\Program Files\\Python311\\python.exe'
>>> sys.prefix
'C:\\Temp\\.venv'
>>> sys.base_prefix
'C:\\Program Files\\Python311'
>>> print(*sys.path, sep='\n')

C:\Temp\.venv\Scripts\python311.zip
C:\Program Files\Python311\DLLs
C:\Program Files\Python311\Lib
C:\Program Files\Python311
C:\Temp\.venv
C:\Temp\.venv\Lib\site-packages

Note that the command value in “pyvenv.cfg” isn’t quoted properly to support paths with spaces in the command line. It doesn’t seem to cause a problem, but I don’t know how this value gets used.

Filesystem symlinks are implemented as reparse points. NTFS and ReFS support reparse points. However, being allowed to actually create a symlink requires either “SeCreateSymbolicLinkPrivilege” or for the system to be in developer mode.

By default “SeCreateSymbolicLinkPrivilege” is only granted to administrators. Also, for an administrator account, only an elevated logon will have the administrators group enabled, given that UAC is enabled for both the system and the account. That said, in the system management console (“secpol.msc” snap-in), an administrator can assign the right to create symbolic links to any user or group, such as the “Users” group or “Authenticated Users” group.

Note that if you need to run a virtual environment directly from Explorer, then you’ll have to use the venv launcher instead of symlinks because Explorer eagerly resolves the final path of a symlink before executing it. Also, the venv launcher is required for an environment that’s based on the store app distribution of Python.


  1. “python3.exe” and “python3.x.exe” symlinks don’t work correctly due to a bug in Python’s startup code. The base name of sys._base_executable will mistakenly be set to the same as the base name of sys.executable. For example, if the base installation is 3.11, installed for all users, and a “python3.exe” symlink is executed, then sys._base_executable will be “C:\Program Files\Python311\python3.exe”. No such file exists in the base installation directory. If __PYVENV_LAUNCHER__ isn’t set, sys._base_executable should be set to the final resolved path of GetModuleFileNameW(NULL, ...).

    The multiprocessing module is broken by this bug because it uses sys._base_executable on Windows if it’s different from sys.exectuable. It should be more defensive. It should compare the final resolved paths, and it should default to using sys.executable if sys._base_executable can’t be resolved. ↩︎

  2. Adding a directory to PATH doesn’t always work anyway. An application can delay-load its dependencies, and load them securely on demand by calling SetDefaultDllDirectories() at startup. ↩︎

2 Likes

My lack of time. :wink: I put 1.5 hours into it and kept getting it wrong, so since I didn’t have a need for such support I decided to just not worry about it.

Yes you can; microvenv/test_microvenv.py at 709388e37952ac9ea0e6c97dc52eac8c007b195e · brettcannon/microvenv · GitHub is the test which verifies this works.

I didn’t bother to go searching, but we run into weird situations all the time with VS Code. But Debian is definitely the biggest contributor to venv missing based on bug reports.

Because it’s one more step that gets in the way for beginners to get up and going. And since apt-get mostly likely will require sudo we prefer to not have to explain that detail as well (and that’s even assuming they have sudo on the machine they are working on; work and school machines can be very locked down).

I did; it’s microvenv.py and that’s what is in the wheel. The -c option is just a way to run the code, not the way (and I have potential uses someday in the Python Launcher for Unix for such usage via -c). I have tried to clarify that in the blog post.

Because .venv is about as close as we have as a convention for in-place virtual environment directory names (it’s used by VS Code, the Python Launcher for Unix, and it’s part of PEP 704). Plus most people don’t want searches to lead into their virtual environment when looking for something, so this helps some tools to ignore it.

Because I have seen tons of beginners mess up by installing their dependencies globally and breaking something (whether it’s their installation of Python, some other code that had different dependencies, etc.). Or not realizing what python or pip point at because of installation order compared to virtual environments which explicitly state what those commands represent. The multiple Python versions are definitely an advanced case, though.

It doesn’t get used by Python, so it’s really up to you and whatever software you may be running that uses it. I personally view it as an approximation of what was done to create the virtual environment, but nothing else. I don’t expect it to work in the future, for instance, since it uses sys.executable which could be a symlink or something that changes in the future.

4 Likes

IIRC values in pyvenv.cfg don’t support nor need quoting. The file format is basically lines of key = value entries, and a value is a Python string extracted with essnetially line.split("=", 1)[1].

1 Like

The closest we get to the “official” way of parsing the file is:

For a single path that’s fine. However, the command value that’s stored is generally useless without proper quoting around the executable path and script path. venv behaves the same. For example:

command = C:\Program Files\Python311\python.exe C:\Program Files\Python311\Lib\site-packages\microvenv.py C:\Temp\.venv

I didn’t know the purpose of this value. Apparently it’s just metadata that’s not intended to be used in any automated process.