Python (or third-party) DLL file versions can interact poorly with Windows Installer's "upgrade" functionality - What to do?

I maintain a Python product that is built with PyInstaller and distributed as a Windows Installer file. This week I discovered a not-quite-bug in Windows Installer that can cause missing files when performing an upgrade from one version to another. If files in the application being upgraded define the File Version attribute, and if this version goes backwards, the files can end up missing entirely after upgrade [1].

In my case, the files that define this attribute which caused a broken installer are from Python itself (i.e. pythonXY.dll and friends), and it’s pretty easy for the value to go backwards if the installer is built on different systems/CI nodes that have subtly different releases of the same major version. In my case, the old version of the product had Python files with version 3.8.10150.1013 and the new version of the product had 3.8.5150.1013 which I believe means that we build the old installer with 3.8.10 and the newer one with 3.8.5, although the application is not sensitive to the distinction between these versions.

I’m sad to say that there seems to be no official “right” way to get around this problem. The potential solutions that I’m aware of are:

  1. Watch your File Version attributes like a hawk and don’t let them go backwards. This seems more like “don’t have that problem” than it does like a fix, and isn’t an option for me.

  2. Resequence the Windows Installer actions so that RemoveExistingProducts comes before CostFinalize, the step that concludes the “which files should we install?” calculation. This is technically forbidden by the Installer system for reasons that aren’t clear to me, and it creates an error during installer validation, although I think that error can be suppressed. I haven’t tried this approach yet.

  3. Set the REINSTALLMODE property in the installed to include the flag a which indicates that all files should be installed, regardless of file version. This seems to be frowned upon because this property is not really meant to be set ahead of time in the installer itself. It does work and was the workaround I chose in my own case, but it does not seem like a long-term solution to the problem.

  4. The most extreme option: edit the offending files to change their File Version attribute. In the case of Python’s DLLs I would probably transform the version to change version A.B.C.D to just A.B.

I’m wondering if other developers have encountered this buggy upgrade behavior, and if so, what you do about it. I’m also curious about exactly how evil solution (4) is. Does anything in CPython depend on the value of this field? Am I going to bring the sky down if I kludge my way through the problem by truncating the version of these files to major.minor to prevent recurrence of the issue? Are there possibilities I’ve missed?

[1] Why Windows Installer removes files during a major upgrade if they go backwards in version numbers | Microsoft Learn

1 Like

You have a badly broken build system for your application.

You must control your build tool versions with as much care as you control your source code.

Fix that process issue that you do not need a work around.

You will also save yourself a lot of wasted time debugging off edge condition bugs that come from not having fixes in your dependences that you assume you have.

I agree that having drift in build tool versions is bad, and wish that I didn’t have to live with it. This is basically solution (1) from above and I think the best I could do is fail the build if the build environment does not exactly match hard-coded expectations.

I see where you’re coming from on not adding maintenance burden to the build chain, but I don’t think the general problem can be so easily side-stepped, especially because the same issue affects the entire dependency tree, too.

The breakage I saw happened to come from Python’s DLLs, but the majority of this project’s versioned files are actually from third-party libraries, and a maximally cautious approach to the problem should assume that any change to the versions of those dependencies (including not only rollbacks, but also upgrades) won’t cause a file version somewhere to go backwards. In hindsight, my original title for this post was too specific, because I’m also worrying about this more general problem.

I recommend asking about this on the pyinstaller’s issue tracker or community forums. You’re more likely to get a better informed and directly actionable responses there since that’s where folks most familar with pyinstaller will be.

The usual audience of this forum is more focused on “lower level” tools like pip, setuptools, PyPI etc, Python-specific packaging formats like wheels, and Python-specific interoperability standards (i.e. stuff in Packaging PEPs | peps.python.org).

I recommend asking about this on the pyinstaller’s issue tracker or community forums.

Thanks for the suggestion, I’ve created a sibling post on the discussions section of their repo.

I’m sure you will find people on the pyinstaller list that will understand the issue.
Buts it not an issue with pyinstaller, it an issue with how the install code you use to put the .exe and .dll deps on the windows system works.

Downgrading .DLLs is in 99.999% of cases the wrong thing to do.

1 Like

It’s possible that there’s something wrong with the version metadata embedded in the official windows python builds? I don’t know what that would have to do with compiler versions though, or if that’s what you’re seeing, or even how versions are embedded in dll files and what they mean, though. Most people here are not very familiar with Windows arcana, so you might need to explain in a lot more detail for us to understand what you’re asking.

It’s possible that there’s something wrong with the version metadata embedded in the official windows python builds?

I don’t think so, the version appears to be the typical major.minor.micro along with some additional information.

I didn’t realize before writing this reply what that additional information was, but poking around in the relevant part of the CPython source suggests that the format is major.minor.field3.api_version where field3 is a combination of the micro version number and the release ‘level’ and serial number.

Most people here are not very familiar with Windows arcana, so you might need to explain in a lot more detail for us to understand what you’re asking.

Sorry if I’ve been obtuse about that, I’m relatively new to this sort of arcana myself.

To restate the problem:

Windows lets files declare metadata about their version. I’ve attached screenshots showing this for python38.dll from two versions of Python so it’s clear what I mean by this.

The Windows Installer system can use this information when deciding what files may or may-not need to be installed. Unfortunately, it does not allow installers to express “it’s okay if the version of file such-and-such goes backwards.”

I think the “fix your build system/process” advice is probably the ‘real’ answer to my headache when it comes to Python itself, but I’m still worried about third-party libraries that also define this File Version field. We will likely at some point need to roll back one of these libraries, and the same broken-installer problem will happen again.

I’m hoping that other developers have already bumped into this edge case and might have some suggestions for fixes that I haven’t considered

I use the Inno setup that i think does allow control over how version info is processed. Its open source and works very well.

https://jrsoftware.org/isinfo.php

The root of the problem is an upgrade mechanism that thinks DLL’s are versioned in isolation. pythonXX.dll expects to find other DLLs and Python files that are from the same distribution, but the installer thinks the DLL can be upgraded by itself, with no regard for accompanying files. I would say that your solution 3 is correct: The upgrade mechanism is broken by design, and therefore must be disabled.

@barry-scott, I also use Inno Setup, but despite that I’ve had problems of a similar nature. I didn’t look into it too deeply, so I can’t be sure it’s exactly the same problem. I solved it by always completely uninstalling the previous version before installing a new one.

Ultimately, solution 1 (“don’t have that problem”) is the best one, and solutions 2 & 3 are going to bite you later in even more subtle ways. Solution 4 is probably your best option, though I’d strongly recommend increasing the version number, rather than truncating it (otherwise you won’t be able to patch 3.8.5 installs with security fixes that are in, say, 3.8.10 :wink: )

Nothing within Python itself depends on the field 3 version - it’s generated entirely to support installers that can’t encode prerelease markers in versions. It’s possible that other libraries may use it, but they’re probably going to read from internal fields rather than looking at the version info struct, so they shouldn’t be affected.

However, patching the binary like this (assuming you’re using our releases) will break the code signing. This may cause some machines to refuse to load the DLL, detect it as a virus, or display warnings. You’ll want to re-sign the file with your own certificate in this case.

Now that it’s shipped, you probably want to advise your existing users to run a repair (which should replace the file), and then ensure that you avoid the problem in the future by not downgrading files in upgrade releases. (Or if you need to, then you find a way to do it safely, e.g. by not allowing in-place upgrades but requiring old versions be uninstalled first.)

1 Like

Ultimately, solution 1 (“don’t have that problem”) is the best one, and solutions 2 & 3 are going to bite you later in even more subtle ways.
…
e.g. by not allowing in-place upgrades but requiring old versions be uninstalled first.

My understanding is that my option (2) is pretty much the only way to actually get this behavior, because the removal must happen before “file costing” (deciding what files to install/remove and what the disk space requirements are) occurs.

The Installer community seems to agree with the “bite you later” assessment for options (2) & (3), and since making my original post I’ve made an attempt at (2) and ended up with two simultaneously installed versions of the product as far as Windows was concerned, which is not encouraging.

Nothing within Python itself depends on the field 3 version - it’s generated entirely to support installers that can’t encode prerelease markers in versions. It’s possible that other libraries may use it, but they’re probably going to read from internal fields rather than looking at the version info struct, so they shouldn’t be affected.

This is very useful information, thanks! :grin:

I’d strongly recommend increasing the version number, rather than truncating it

I recall that while reading about this problem I saw some references to a third-party tool (I want to say it was Inno Setup?) implementing option (4) by setting the File Version field to its maximum value, although I cannot dredge up the reference now. I think you’re right that truncation is the wrong way to go about it, I’ll have to think some more about it and maybe test the approach in a few different upgrade-hazard scenarios to see how it behaves.

This discussion has left me wondering if I’ve been too hasty in dismissing option (1). Maybe it’s enough to fail the build if the Python version isn’t exactly such-and-such version (addressing the problem as I encountered it) and write a smoke test to catch a broken upgrade workflow in general. That leaves the issue of regressions of File Version in third-party libraries open, but at least we’d know that the problem has occurred and a human being can then get involved.

To be clear, I meant an entire product removal, i.e. fail the install with a message telling the user that they have to uninstall the old version first.[1] Trying to do it within a single MSI transaction is asking for trouble.

This kind of issue is why we use a Burn bundle for the Python installer, because we have a few more options when it comes to upgrading at the product level rather than the component level. (It has many other downsides…)


  1. And the best way to do this is to store the older product ID in your upgrade table and use that to fail the installer launch. ↩︎

1 Like