Intermittent issues with `UNKNOWN` wheels on CircleCI Windows builders

We’ve been getting intermittent issues when doing pip install ... of our dependencies on the CircleCI Windows builders for quite some time. They always take this form:

Created wheel for xxx: filename=UNKNOWN-0.0.0-py3-none-any.whl
  WARNING: Built wheel for xxx is invalid: Wheel has unexpected file name: expected 'xxx', got 'UNKNOWN'

The important part bits are

  • We’re using latest pip (23.1.2 at the time of writing)
  • Some times the pip install works, sometimes it doesn’t. pip and package version are exactly the same, as well as the environment.
  • Only happens with packages that don’t have compatible wheels uploaded to PyPI.
  • Only happens on Windows.

I’ve seen a report of something similar in the Mozilla issue tracker https://bugzilla.mozilla.org/show_bug.cgi?id=1732228, but looking at the patches, it’s not clear to me what the root cause was.

Here’s the most recent example:

Given no hashes to check 1 links for project 'psutil': discarding no candidates
Collecting psutil==5.8.0 (from -r test_requirements.txt (line 43))
  Created temporary directory: C:\Users\circleci\AppData\Local\Temp\pip-unpack-d1x2brhn
  Looking up "https://files.pythonhosted.org/packages/e1/b0/7276de53321c12981717490516b7e612364f2cb372ee8901bd4a66a000d7/psutil-5.8.0.tar.gz" in the cache
  No cache entry available
  https://files.pythonhosted.org:443 "GET /packages/e1/b0/7276de53321c12981717490516b7e612364f2cb372ee8901bd4a66a000d7/psutil-5.8.0.tar.gz HTTP/1.1" 200 470886
  Downloading psutil-5.8.0.tar.gz (470 kB)
  Ignoring unknown cache-control directive: immutable
  Updating cache with response from "https://files.pythonhosted.org/packages/e1/b0/7276de53321c12981717490516b7e612364f2cb372ee8901bd4a66a000d7/psutil-5.8.0.tar.gz"
  etag object cached for 1209600 seconds
  Caching due to etag
     ------------------------------------- 470.9/470.9 kB 28.8 MB/s eta 0:00:00
  Added psutil==5.8.0 from https://files.pythonhosted.org/packages/e1/b0/7276de53321c12981717490516b7e612364f2cb372ee8901bd4a66a000d7/psutil-5.8.0.tar.gz (from -r test_requirements.txt (line 43)) to build tracker 'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-build-tracker-68vmmu8s'
  Running setup.py (path:C:\Users\circleci\AppData\Local\Temp\pip-install-j6i3ig7y\psutil_4c310ca6d23147f5a6c43361f6764581\setup.py) egg_info for package psutil
...
Building wheels for collected packages: psutil, ...
  Created temporary directory: C:\Users\circleci\AppData\Local\Temp\pip-wheel-45r3mc49
  Building wheel for psutil (setup.py): started
  Destination directory: C:\Users\circleci\AppData\Local\Temp\pip-wheel-45r3mc49
  Running command python setup.py bdist_wheel
  No parent package detected, impossible to derive `name`
  running bdist_wheel
  running build
  running build_py
  C:\tools\miniconda3\envs\kedro_builder\lib\site-packages\setuptools\_distutils\cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
  !!

          ********************************************************************************
          Please avoid running ``setup.py`` directly.
          Instead, use pypa/build, pypa/installer, pypa/build or
          other standards-based tools.

          See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
          ********************************************************************************

  !!
    self.initialize_options()
  installing to build\bdist.win-amd64\wheel
  running install
  running install_lib
  warning: install_lib: 'build\lib' does not exist -- no Python modules to install

  running install_egg_info
  running egg_info
  creating UNKNOWN.egg-info
  writing UNKNOWN.egg-info\PKG-INFO
  writing dependency_links to UNKNOWN.egg-info\dependency_links.txt
  writing top-level names to UNKNOWN.egg-info\top_level.txt
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  reading manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  Copying UNKNOWN.egg-info to build\bdist.win-amd64\wheel\.\UNKNOWN-0.0.0-py3.10.egg-info
  running install_scripts
  creating build\bdist.win-amd64\wheel\UNKNOWN-0.0.0.dist-info\WHEEL
  creating 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-45r3mc49\UNKNOWN-0.0.0-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
  adding 'UNKNOWN-0.0.0.dist-info/METADATA'
  adding 'UNKNOWN-0.0.0.dist-info/WHEEL'
  adding 'UNKNOWN-0.0.0.dist-info/top_level.txt'
  adding 'UNKNOWN-0.0.0.dist-info/RECORD'
  removing build\bdist.win-amd64\wheel
  Building wheel for psutil (setup.py): finished with status 'done'
  Created wheel for psutil: filename=UNKNOWN-0.0.0-py3-none-any.whl size=935 sha256=ac7c5cb9a40a898969bf47db187251d8f9279c152284b28fb20864487bf2586f
  Stored in directory: c:\users\circleci\appdata\local\pip\cache\wheels\12\a3\6d\615295409067d58a62a069d30d296d61d3ac132605e3a9555c
  WARNING: Built wheel for psutil is invalid: Wheel has unexpected file name: expected 'psutil', got 'UNKNOWN'

Full logs can be found here, including the output of pip debug and pip install -vvv: Failed installation · GitHub

Pull request: Add debugging information for Windows tests by astrojuanlu · Pull Request #2730 · kedro-org/kedro · GitHub
CircleCI job: https://app.circleci.com/pipelines/github/kedro-org/kedro/23666/workflows/f41bd673-cbc1-4a7c-8572-41bb56630494/jobs/271686 (possibly ephemeral)

Does the CI environment perhaps have an outdated version of setuptools?

I see 65.6.3 in the logs, released November 2022. Not sure if it’s recent enough.

\Hi Juan, that definetely looks a bit weird.

It does seem a relatively recent version of setuptools is being used (given the warning), however it also seems as if setuptools is being run from an empty directory…

Any chance there is a race condition that accidentally (and momentarily) changes CWD?
Another thing that I can think is permissions on the directory that the tar.gz is being expanded… Maybe there is something off with that?

Hi @abravalheri , thanks for the prompt response (and for all the amazing work you guys do with setuptools)

I guess your observation of setuptools being run from an empty directory comes from here:

running install_lib
  warning: install_lib: 'build\lib' does not exist -- no Python modules to install

That’s good insight, because initially I thought that maybe pip was somehow truncating the downloads and unzipping garbage.

I honestly have no idea what might be going on. Other wheels got built in the same command:

Building wheels for collected packages: psutil, antlr4-python3-runtime, pyspark, docopt, jax

Do you have more ideas on how can I keep debugging this issue? I ran out of -vvv I believe :smile: maybe --no-clean?

This is happening to us several times a day, so it’s not even difficult to reproduce.

What packages does it happen with? I see that psutil has a pretty extensive setup.py with lots of code and also imports distutils… Does it reproduce with packages that have a simpler setup?

It used to happen with import-linter too, until they uploaded wheels Failed installation · GitHub (see also Intermittent pip install failures make Windows tests flaky · Issue #2570 · kedro-org/kedro · GitHub)