How to create a Windows command line application and its installer from wheels

Packaging
1

Let’s say that we have Windows wheels for a CLI. To take a simple but realistic example, one can consider the CLI black (a Python console-script entry point) and the wheels that are installed with black[jupyter].

My real motivation would be creation of apps and installers for Mercurial (a CLI) and potentially also TortoiseHg (a GUI), but this would be more complicated so let’s first consider the simpler case of Black.

I learned that tools like pipx or uv tools have downside on Windows, in particular for security and, consequently performance.

Example for black installed with pip in a virtual env (created from Python installed from the official installer):

$ Measure-Command { black --version }
TotalMilliseconds  : 820
$ Measure-Command { C:\Users\me\venv-black\Scripts\python.exe -m black __version__ }
TotalMilliseconds  : 250

The difference (approximately half a second, i.e. going to a startup time somehow acceptable for a CLI to something really slow leading to a bad UX) seem to be due to virus scan done each time black.exe is launched. The virus scan is triggered because black.exe is not considered safe because it is a modified binary created by distlib via pipx (see About .exe wrappers created by frontends when installing wheels on Windows? - #9 by pf_moore).

I learned that:

So we should create a proper Windows application that would provide the CLI black globally and its installer.

How can it be done?

It should be quite simple to do. From black[jupyter] we can get a list of wheels with version pinned. And we have the Python embedded distribution.

But the documentation is not very clear about how this can be done in practice.

From these, it’s not clear to me how to do what @steve.dower suggested.

What I’d like to avoid would be disutils/setuptools extensions (like py2exe). If possible, I’d like to build the app from wheels, because we already have good wheels built and tested.

There was this thread https://discuss.python.org/t/installer-creation-based-on-distributions in 2023 but without a clear conclusion.

I tend to dislike PyOxidizer because (i) it strives to create a one file binary (which I don’t care and can lead to a lot of potential issue, in particular no data files and no updates/installs of Mercurial extensions), (ii) it’s no longer maintain by its creator.

I read about different tools, like briefcase (very oriented on Beeware tools i.e. GUI so I don’t know if it could be used for a CLI) and pynsist (no .msi installer).

For GUI, I also saw that Spyder has a good installer based on Miniforge and conda-forge.

I wonder if there are good news in this field in 2025.

PS: For my real use case (create an app and an msi installer for Mercurial) it would be great that pip install would be available within the Python used for the app in order to manage Mercurial extensions.

2 Likes
2

The reference implementation I’m working on for a new installer tool is a complex example of one way to do this. It uses my own build tool (pymsbuild) to script some of the build steps, but fundamentally it can be done with any build script.

The steps of the build script (to be clear, you run this yourself on your build machine, and distribute the results to users - users don’t run these steps!) will be:

  • find a normal Python install on the build machine
  • use it to pip install --target <output directory> any dependencies you need
  • download a matching embeddable distro from python.org and extract into <output directory>
  • remove python.exe and any extension modules you don’t need - leave the DLLs and ._pth file
  • if your app isn’t including the MSVC Runtime itself, you’ll need to grab vcruntime140[_1].dll from somewhere - the normal Python install will have them, but the embeddable distro does not (looks like we do? possibly a bug, or perhaps someone convinced me…)
  • create your own main.c that copies argv, modifies it to be a Python-like command line (e.g. insert -m your_module at argv[1] and 2) and calls Py_Main
  • compile your main.c and copy your main.exe into <output directory>.
  • use whatever packaging tool you like to create a MSI/MSIX/NSIS/InstallShield/SFX/etc. package containing everything in <output directory>
  • ship that package to your users

There are some tools out there that can help with some of these steps - e.g., pymsbuild has an “Entrypoint” type that can generate the main.c/main.exe for you, and briefcase/pynsist automate getting the embeddable runtime and creating the final bundle.

It’s up to you how you want to do it though, there are no opinionated tools in the core ecosystem, because usually embedders have an existing application that they’re adding Python into, and so an opinionated tool is always going to be wrong for them. It’s also a terribly complicated area to try and support, so anyone brave enough to make a “one-click” tool to do all this is going to end up with millions of bug reports from every slight variation of a system configuration on behalf of every one of their users.

But for those who are keen enough, or who need it enough, those above are the rough steps you’re going to follow.

5 Likes
3

For this, you’ll want to treat pip as one of your application dependencies. It’s trivial to pip install pip, but it’s incredibly complicated to manage it indirectly when deployed to users (some of the projects I support at work try to do this). Good luck.

(You also need to leave python.exe in your package, and when invoking pip you’ll have to lie about your sys.executable so that when it tries to relaunch itself, it succeeds. Or you can probably patch any code that does this easily enough.)

4

In general, I sympathise with the points you make, although in my experience the performance issue you are quoting is very rare (i.e., I’ve not seen it myself, nor have I heard of pip users complaining about it[1]).

In general, I think the community is fairly happy with the pipx/uv tool approach. I don’t necessarily agree with that, but I’m not aware of anyone motivated to find other solutions to the “CLI application in Python” issue (at least, not ones that haven’t been around for ages, like PyInstaller).

On this specific point, I think you’d be better treating your Python install as “private”, and providing a hg extension install command that wraps pip. That protects you from people doing weird things to the Python environment and then expecting you to support them :wink: That’s more important if you embed a private copy of Python in your app, but it’s still a reasonable thing to do if you’re working with entry points and pipx/uv tool.

  1. there are plenty of complaints about pip’s speed in general, but not about startup time ↩︎

2 Likes
5

Have you checked out GitHub - ofek/pyapp: Runtime installer for Python applications?

1 Like
6

I definitely owe everyone some documentation.

Until then, have a blog post :smiley: Building a Python App | Steve Dower

6 Likes
7

One particular point that I think could be discussed in that post is how to put the Python distribution in a subdirectory of the layout. I’ve found this important as it ensures that your app doesn’t put common DLLs like zlib1.dll onto PATH (assuming you’re shipping a command line exe where you need the application executable to be on PATH). As far as I know, in order to do this you need to add a manifest to your app executable, which tells the loader to search for DLLs in your chosen subdirectory. It’s frustratingly difficult to find out how to do this - if I recall the process, it involves side-by-side assemblies, but the docs on that subject are heavily biased towards .NET assemblies and there’s no specific example covering “how to tell your app where to find a particular DLL”.

In theory this is something that the Microsoft documentation should make clearer, but I doubt that will happen in practice, and having it clearly explained in your blog entry would help people who like to keep their app directories tidy (whether that’s because they care about exposing DLLs, or just because they like tidiness for its own sake).

1 Like
8

Yeah, I’m thinking of doing a couple of follow ups that cover stuff like this (and this particular one was high on my list).

The easy way to do it is to mark python313.dll as delay load, and then call AddDllDirectory very early in your launcher. Then when you actually go to run it, it’ll load from the right place and everything else should flow on from there.

Using manifests for assembly references is basically deprecated (as much as anything gets deprecated in Windows). It was a great idea, but everyone preferred doing tricks instead, and so the manifests aren’t encouraged (I’d have to double check timings, but I believe the security enhancements for LoadLibraryEx and the addition of AddDllDirectory were in response to everyone hating SxS assemblies so much).

And one day hopefully these posts can condense down into some kind of documentation. Doing it in blog post form is really just to force me to put the information down in writing, and then deal with making it documentation-style later.

2 Likes
9

Ah, cool. That does indeed work, although it’s fiddly to find the right parts of the Microsoft documentation to work out how to do it. That’s one of the big problems here - our audience is Python users, and while we can assume a limited amount of C knowledge, simply from the fact that they are willing to write a C driver program for their app, we’ll put off a huge proportion of the target audience if we expect them to navigate the Windows documentation for all of this. People will cargo-cult just enough to get something that works, and bad choices (like my previous assumption that SxS assemblies were the right way to go) will perpetuate themselves.

+1 on getting something into the documentation for this. I might try to add a discussion page to packaging.python.org as a starter, if only to capture this discussion somewhere more official.

FWIW, this is what I ended up with (very rough code, no real error handling yet):

#include <windows.h>
#include <pathcch.h>

#pragma comment(lib, "delayimp")
#pragma comment(lib, "pathcch")

int dll_dir(wchar_t *path) {
    wchar_t interp_dir[PATHCCH_MAX_CCH];
    if (GetModuleFileNameW(NULL, interp_dir, PATHCCH_MAX_CCH) &&
        SUCCEEDED(PathCchRemoveFileSpec(interp_dir, PATHCCH_MAX_CCH)) &&
        SUCCEEDED(PathCchCombineEx(interp_dir, PATHCCH_MAX_CCH, interp_dir, path, PATHCCH_ALLOW_LONG_PATHS))) {
        SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_DEFAULT_DIRS);
        AddDllDirectory(interp_dir);
    }
    return 1;
}

Call dll_dir(L"interp") just before invoking PyConfig_InitIsolatedConfig, link with /DELAYLOAD:python313.dll, and it works (or at least, seems to :slightly_smiling_face:).

It needs error handling, but that’s when we start getting to the point of having to cater for how the app wants to do things (print errors to stdout, what return code to use, etc, etc) and I didn’t want to go there for an example.

Suggestions for improvements would be gratefully accepted.

1 Like
10

Edit: Never mind, I found PyConfig_SetArgv and config.install_signal_handlers. Unless there are non-obvious subtleties, that’s all I need.

The other one I think needs covering (based on the fact that I did some initial digging and got scared :slightly_smiling_face:) is passing the application argv to Python. The isolated configuration is nice, as it avoids needing to worry about things like environment variables, etc, but a CLI application that doesn’t handle arguments is pretty useless.

I was also concerned by the comment in the docs:

The C standard streams (ex: stdout) and the LC_CTYPE locale are left unchanged. Signal handlers are not installed.

It’s not at all clear to me what implications that would have on the behaviour of the interpreter. One thing I see is that Ctrl-C terminates the program abruptly, rather than raising KeyboardInterrupt. That’s a bit unexpected.

The important point here (at least in my opinion) is that the audience we should be aiming at is experienced Python developers, who know how to write complex Python code (and that’s probably what they are packaging), but who have little or no experience with C, the Python C API, or the Windows API. These are the people who are currently not well served by the packaging ecosystem unless their needs fit one of our predefined models (basically entry points/pipx, or specific tools like PyInstaller or Beeware).

By the way - just to be clear, I’m not just picking your brains for my own benefit here. I’m writing up my experiences as a new section in the packaging user guide, to give people a starting point on embedding. I’ll post a link here once I have something semi-complete.

3 Likes
11

Yeah, this is what I was intending. The biggest nuance/issue with this approach is that data exports can’t be delay loaded, and so you can’t use things like Py_None that directly reference exported data. We are starting to add function call alternatives, which can be used safely as macros (e.g. #define Py_None Py_GetConstantBorrowed(PY_CONSTANT_NONE), but I need to spend a bit more time figuring out the best definitions to use when hacking your way into the existing headers.

Potentially not what you want for the simple case we’re dealing with here, but actual apps that are just adding CPython generally don’t want their settings messed about with. Some of us are already looking at pulling all of these initialization “features” out into their own functions, so that it’s easier to clone the CPython interactive interpreter when that’s what you want.

Honestly, I’d rather have my brain picked by someone who’s writing the docs than go off and write them myself :smiley:

1 Like
12

That’s fair, and I think one of the reasons people don’t immediately think of embedding is precisely because we present it as more for “actual apps” than for these “simple cases”. A Python application with many thousands of lines of code and a simple embedding app is just as much an “actual app” as something like Blender, of course, so it’s more about having appropriate docs for different audiences, which is what I’m hoping we can fix.

Having said that, what is the impact of leaving the stream handling set up the way the “isolated” config does, on a simple case like this? What differences in behaviour would I see in my Python code?

There’s a whole other article here on “how do I write Python code that will be embedded interpreter friendly?”[1] but I’ll pass on that one for now :slightly_smiling_face:

  1. The biggest gotcha being that sys.executable is no longer a Python interpreter ↩︎

1 Like
13

Very little, I think. Most of the early process-wide modifications we do are to deal with things like libc that isn’t using UTF-8, which only really matters for initialisation so we can read environment variables correctly. If you’re being isolated, then you’re likely not reading them anyway.

I think the stream settings mainly impact our interactive mode, and possibly piping scripts from other files/commands in the shell. Again, when embedding you probably don’t need these.

That said, I’m going off decade-old memories and knowledge that’s already in my head. It’s very likely there are other edge cases that I’ve forgotten about - it’ll probably take some digging into why we made those changes in the first place.

1 Like
14

I notice that when I build an app based on the approach in your post (modified to delay load python313.dll), the resulting executable depends on KERNEL32.dll and api-ms-win-core-path-l1-1-0.dll. That latter one is presumably part of the UCRT, so I assume it’s OK to expect users to have it available? For the target audience I have in mind, building something that can just run anywhere (for a reasonably broad definition of “anywhere” :wink:) is important.

15

Yes, it’s part of the operating system (broadly speaking the api-... DLLs are resolved specially and remapped to the actual DLL, so you won’t see the file there, but it can be loaded).

1 Like
16

Great news. Thanks a lot @steve.dower and @pf_moore to share your experience!

17
4 Likes
18

Thanks very much @steve.dower and @pf_moore for documenting this process!

I used your descriptions to create an executable for one of my Python apps.
I’ve built and run it, and from what I can tell it is working as I expect!

I’m really grateful to you both for documenting all the pieces to the point I could reproduce the steps!

I created a gist with the files I needed.

The most interesting file is build_embed_dist.py, it has a bunch of notes and hints at the top about things I learnt along the way.

4 Likes