I maintain a Python product that is built with PyInstaller and distributed as a Windows Installer file. This week I discovered a not-quite-bug in Windows Installer that can cause missing files when performing an upgrade from one version to another. If files in the application being upgraded define the File Version attribute, and if this version goes backwards, the files can end up missing entirely after upgrade [1].
In my case, the files that define this attribute which caused a broken installer are from Python itself (i.e. pythonXY.dll and friends), and itâs pretty easy for the value to go backwards if the installer is built on different systems/CI nodes that have subtly different releases of the same major version. In my case, the old version of the product had Python files with version 3.8.10150.1013 and the new version of the product had 3.8.5150.1013 which I believe means that we build the old installer with 3.8.10 and the newer one with 3.8.5, although the application is not sensitive to the distinction between these versions.
Iâm sad to say that there seems to be no official ârightâ way to get around this problem. The potential solutions that Iâm aware of are:
Watch your File Version attributes like a hawk and donât let them go backwards. This seems more like âdonât have that problemâ than it does like a fix, and isnât an option for me.
Resequence the Windows Installer actions so that RemoveExistingProducts comes before CostFinalize, the step that concludes the âwhich files should we install?â calculation. This is technically forbidden by the Installer system for reasons that arenât clear to me, and it creates an error during installer validation, although I think that error can be suppressed. I havenât tried this approach yet.
Set the REINSTALLMODE property in the installed to include the flag a which indicates that all files should be installed, regardless of file version. This seems to be frowned upon because this property is not really meant to be set ahead of time in the installer itself. It does work and was the workaround I chose in my own case, but it does not seem like a long-term solution to the problem.
The most extreme option: edit the offending files to change their File Version attribute. In the case of Pythonâs DLLs I would probably transform the version to change version A.B.C.D to just A.B.
Iâm wondering if other developers have encountered this buggy upgrade behavior, and if so, what you do about it. Iâm also curious about exactly how evil solution (4) is. Does anything in CPython depend on the value of this field? Am I going to bring the sky down if I kludge my way through the problem by truncating the version of these files to major.minor to prevent recurrence of the issue? Are there possibilities Iâve missed?
You have a badly broken build system for your application.
You must control your build tool versions with as much care as you control your source code.
Fix that process issue that you do not need a work around.
You will also save yourself a lot of wasted time debugging off edge condition bugs that come from not having fixes in your dependences that you assume you have.
I agree that having drift in build tool versions is bad, and wish that I didnât have to live with it. This is basically solution (1) from above and I think the best I could do is fail the build if the build environment does not exactly match hard-coded expectations.
I see where youâre coming from on not adding maintenance burden to the build chain, but I donât think the general problem can be so easily side-stepped, especially because the same issue affects the entire dependency tree, too.
The breakage I saw happened to come from Pythonâs DLLs, but the majority of this projectâs versioned files are actually from third-party libraries, and a maximally cautious approach to the problem should assume that any change to the versions of those dependencies (including not only rollbacks, but also upgrades) wonât cause a file version somewhere to go backwards. In hindsight, my original title for this post was too specific, because Iâm also worrying about this more general problem.
I recommend asking about this on the pyinstallerâs issue tracker or community forums. Youâre more likely to get a better informed and directly actionable responses there since thatâs where folks most familar with pyinstaller will be.
The usual audience of this forum is more focused on âlower levelâ tools like pip, setuptools, PyPI etc, Python-specific packaging formats like wheels, and Python-specific interoperability standards (i.e. stuff in Packaging PEPs | peps.python.org).
Iâm sure you will find people on the pyinstaller list that will understand the issue.
Buts it not an issue with pyinstaller, it an issue with how the install code you use to put the .exe and .dll deps on the windows system works.
Downgrading .DLLs is in 99.999% of cases the wrong thing to do.
Itâs possible that thereâs something wrong with the version metadata embedded in the official windows python builds? I donât know what that would have to do with compiler versions though, or if thatâs what youâre seeing, or even how versions are embedded in dll files and what they mean, though. Most people here are not very familiar with Windows arcana, so you might need to explain in a lot more detail for us to understand what youâre asking.
Itâs possible that thereâs something wrong with the version metadata embedded in the official windows python builds?
I donât think so, the version appears to be the typical major.minor.micro along with some additional information.
I didnât realize before writing this reply what that additional information was, but poking around in the relevant part of the CPython source suggests that the format is major.minor.field3.api_version where field3 is a combination of the micro version number and the release âlevelâ and serial number.
Most people here are not very familiar with Windows arcana, so you might need to explain in a lot more detail for us to understand what youâre asking.
Sorry if Iâve been obtuse about that, Iâm relatively new to this sort of arcana myself.
To restate the problem:
Windows lets files declare metadata about their version. Iâve attached screenshots showing this for python38.dll from two versions of Python so itâs clear what I mean by this.
The Windows Installer system can use this information when deciding what files may or may-not need to be installed. Unfortunately, it does not allow installers to express âitâs okay if the version of file such-and-such goes backwards.â
I think the âfix your build system/processâ advice is probably the ârealâ answer to my headache when it comes to Python itself, but Iâm still worried about third-party libraries that also define this File Version field. We will likely at some point need to roll back one of these libraries, and the same broken-installer problem will happen again.
Iâm hoping that other developers have already bumped into this edge case and might have some suggestions for fixes that I havenât considered
The root of the problem is an upgrade mechanism that thinks DLLâs are versioned in isolation. pythonXX.dll expects to find other DLLs and Python files that are from the same distribution, but the installer thinks the DLL can be upgraded by itself, with no regard for accompanying files. I would say that your solution 3 is correct: The upgrade mechanism is broken by design, and therefore must be disabled.
@barry-scott, I also use Inno Setup, but despite that Iâve had problems of a similar nature. I didnât look into it too deeply, so I canât be sure itâs exactly the same problem. I solved it by always completely uninstalling the previous version before installing a new one.
Ultimately, solution 1 (âdonât have that problemâ) is the best one, and solutions 2 & 3 are going to bite you later in even more subtle ways. Solution 4 is probably your best option, though Iâd strongly recommend increasing the version number, rather than truncating it (otherwise you wonât be able to patch 3.8.5 installs with security fixes that are in, say, 3.8.10 )
Nothing within Python itself depends on the field 3 version - itâs generated entirely to support installers that canât encode prerelease markers in versions. Itâs possible that other libraries may use it, but theyâre probably going to read from internal fields rather than looking at the version info struct, so they shouldnât be affected.
However, patching the binary like this (assuming youâre using our releases) will break the code signing. This may cause some machines to refuse to load the DLL, detect it as a virus, or display warnings. Youâll want to re-sign the file with your own certificate in this case.
Now that itâs shipped, you probably want to advise your existing users to run a repair (which should replace the file), and then ensure that you avoid the problem in the future by not downgrading files in upgrade releases. (Or if you need to, then you find a way to do it safely, e.g. by not allowing in-place upgrades but requiring old versions be uninstalled first.)
Ultimately, solution 1 (âdonât have that problemâ) is the best one, and solutions 2 & 3 are going to bite you later in even more subtle ways.
âŚ
e.g. by not allowing in-place upgrades but requiring old versions be uninstalled first.
My understanding is that my option (2) is pretty much the only way to actually get this behavior, because the removal must happen before âfile costingâ (deciding what files to install/remove and what the disk space requirements are) occurs.
The Installer community seems to agree with the âbite you laterâ assessment for options (2) & (3), and since making my original post Iâve made an attempt at (2) and ended up with two simultaneously installed versions of the product as far as Windows was concerned, which is not encouraging.
Nothing within Python itself depends on the field 3 version - itâs generated entirely to support installers that canât encode prerelease markers in versions. Itâs possible that other libraries may use it, but theyâre probably going to read from internal fields rather than looking at the version info struct, so they shouldnât be affected.
This is very useful information, thanks!
Iâd strongly recommend increasing the version number, rather than truncating it
I recall that while reading about this problem I saw some references to a third-party tool (I want to say it was Inno Setup?) implementing option (4) by setting the File Version field to its maximum value, although I cannot dredge up the reference now. I think youâre right that truncation is the wrong way to go about it, Iâll have to think some more about it and maybe test the approach in a few different upgrade-hazard scenarios to see how it behaves.
This discussion has left me wondering if Iâve been too hasty in dismissing option (1). Maybe itâs enough to fail the build if the Python version isnât exactly such-and-such version (addressing the problem as I encountered it) and write a smoke test to catch a broken upgrade workflow in general. That leaves the issue of regressions of File Version in third-party libraries open, but at least weâd know that the problem has occurred and a human being can then get involved.
To be clear, I meant an entire product removal, i.e. fail the install with a message telling the user that they have to uninstall the old version first.[1] Trying to do it within a single MSI transaction is asking for trouble.
This kind of issue is why we use a Burn bundle for the Python installer, because we have a few more options when it comes to upgrading at the product level rather than the component level. (It has many other downsidesâŚ)
And the best way to do this is to store the older product ID in your upgrade table and use that to fail the installer launch. âŠď¸
)