Pyproject.toml and severe regressions in setuptools

I’ve been using python for about ten years now. Packaging has always been a bit of a sore subject; say five years ago the answer was to find a setup.py that works, copy it and question nothing. It didn’t help that prominent packages (numpy, scipy, etc) have very complicated setup.py files and didn’t make good starting points. Sometime along the way, I started using setup.cfg as a “better” way. The documentation for setup.cfg is relatively sparse or poor, but minor errors in files didn’t break “python -m pip install -e .” to install the current module. for example, I accidentally had two identical entries for packages= for years and none of the tooling complained. Fast forward to today, and to my surprise my setup.cfg that worked properly with even recent setuptools versions (at least 50 something) no longer works. Doing nothing, I got this very obscure error:

      /Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
        warnings.warn(
      Traceback (most recent call last):
...I elided the rest, use_scm_version=True not in a git repo was the problem
 
After correcting that, the .git folder not being copied over, setuptools is still not happy:
(neos) [bdube:~/src/neos_prysm]$ pip install -e .
Obtaining file:///Users/bdube/src/neos_prysm
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [16 lines of output]
      /Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
        warnings.warn(
      error: Multiple top-level packages discovered in a flat-layout: ['nb', 'neos', 'data'].
      
      To avoid accidental inclusion of unwanted files or directories,
      setuptools will not proceed with this build.
      
      If you are trying to create a single distribution with multiple packages
      on purpose, you should not rely on automatic discovery.
      Instead, consider the following options:
      
      1. set up custom discovery (`find` directive with `include` or `exclude`)
      2. use a `src-layout`
      3. explicitly set `py_modules` or `packages` with a list of names
      
      To find more information, look for "package discovery" on setuptools docs.
      [end of output]

In this setup.cfg, there was a packages = neos line

I could not convince setuptools that in fact there was a specification for what packages to use, and gave up and decided to try pyproject.toml. I interpret things as “PyPa has broken setuptools compatiblility with setup.cfg and I can use an old version of setuptools, or stop using setup.cfg”.

Syntax errors in the TOML file (missing quotes) provide massively not user friendly errors with pyproject.toml, such as this delightful wall of red:
$ pip install -e .
Obtaining file:///Users/bdube/src/neos_prysm
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 341, in run
    requirement_set = resolver.resolve(
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 75, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 506, in collect_root_requirements
    req = self._make_requirement_from_install_req(
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 468, in _make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 188, in _make_candidate_from_link
    self._editable_candidate_cache[link] = EditableCandidate(
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 316, in __init__
    super().__init__(
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 161, in __init__
    self.dist = self._prepare()
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 230, in _prepare
    dist = self._prepare_distribution()
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 326, in _prepare_distribution
    return self._factory.preparer.prepare_editable_requirement(self._ireq)
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 551, in prepare_editable_requirement
    dist = _get_prepared_distribution(
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 58, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/distributions/sdist.py", line 31, in prepare_distribution_metadata
    self.req.load_pyproject_toml()
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 464, in load_pyproject_toml
    pyproject_toml_data = load_pyproject_toml(
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_internal/pyproject.py", line 64, in load_pyproject_toml
    pp_toml = tomli.loads(f.read())
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_vendor/tomli/_parser.py", line 102, in loads
    pos = key_value_rule(src, pos, out, header, parse_float)
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_vendor/tomli/_parser.py", line 326, in key_value_rule
    pos, key, value = parse_key_value_pair(src, pos, parse_float)
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_vendor/tomli/_parser.py", line 369, in parse_key_value_pair
    pos, value = parse_value(src, pos, parse_float)
  File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_vendor/tomli/_parser.py", line 649, in parse_value
    raise suffixed_err(src, pos, "Invalid value")
pip._vendor.tomli.TOMLDecodeError: Invalid value (at line 8, column 11)

If one of the scientists that uses my package and is totally disinterested in “software itself” saw this error, they might throw the computer out the window.

Perservering, my license = “unlicensed” input for proprietary code made the parser angry:

Obtaining file:///Users/bdube/src/neos_prysm
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [80 lines of output]
      /private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-8rortaz5/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py:257: UserWarning: Unknown distribution option: 'use_scm_version'
        warnings.warn(msg)
      configuration error: `project.license` must be valid exactly by one definition (2 matches found):
      
          - keys:
              'file': {type: string}
            required: ['file']
          - keys:
              'text': {type: string}
            required: ['text']
      
      DESCRIPTION:
          `Project license <https://www.python.org/dev/peps/pep-0621/#license>;`_.
      
      GIVEN VALUE:
          "unlicensed"
      
      OFFENDING RULE: 'oneOf'
      
      DEFINITION:
          {
              "oneOf": [
                  {
                      "properties": {
                          "file": {
                              "type": "string",
                              "$$description": [
                                  "Relative path to the file (UTF-8) which contains the license for the",
                                  "project."
                              ]
                          }
                      },
                      "required": [
                          "file"
                      ]
                  },
                  {
                      "properties": {
                          "text": {
                              "type": "string",
                              "$$description": [
                                  "The license of the project whose meaning is that of the",
                                  "`License field from the core metadata",
                                  "<https://packaging.python.org/specifications/core-metadata/#license>;`_."
                              ]
                          }
                      },
                      "required": [
                          "text"
                      ]
                  }
              ]
          }
      Traceback (most recent call last):
        File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-8rortaz5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 177, in get_requires_for_build_wheel
          return self._get_build_requires(
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-8rortaz5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-8rortaz5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 174, in run_setup
          exec(code, locals())
        File "<string>", line 6, in <module>
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-8rortaz5/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-8rortaz5/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 151, in setup
          dist.parse_config_files()
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-8rortaz5/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 868, in parse_config_files
          pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-8rortaz5/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 58, in apply_configuration
          config = read_configuration(filepath, True, ignore_option_errors, dist)
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-8rortaz5/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 122, in read_configuration
          validate(subset, filepath)
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-8rortaz5/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 47, in validate
          raise error from None
      ValueError: invalid pyproject.toml config: `project.license`
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
 
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

This error message is over 100 lines long, and all of the indented content above wastes tremendous space. Opening the PEP 621 page above, I see that the docs are absolutely clear as mud. Everything uses the language “table” which is unfamiliar jargon (object would be more familiar). Some of the first results from google, e.g. this page (#1 for me) are not the correct format for listing authors. While PyPa is not responsible for third party blogs, I have to imagine this syntax was once valid and has been made invalid, which seems to be a theme here. Correcting most of those errors, I get to this, the mother of all errors clocking in at 563(!!!) lines:

Obtaining file:///Users/bdube/src/neos_prysm
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [544 lines of output]
      configuration error: `project` must not contain {'packages', 'homepage', 'python', 'documentation', 'repository'} properties
      DESCRIPTION:
          Data structure for the **project** table inside ``pyproject.toml`` (as
          initially defined in :pep:`621`)
      
      GIVEN VALUE:
          {
              "name": "neosprysm",
              "version": "0.1",
              "description": "NEOS integrated diffraction model",
              "authors": [
                  {
                      "name": "Brandon Dube",
                      "email": "brandon.dube@jpl.nasa.gov"
                  }
              ],
              "license": {
                  "file": "LICENSE.md"
              },
              "python": "^3.6",
              "homepage": "https://github.jpl.nasa.gov/bdube/neos_prysm",
              "repository": "https://github.jpl.nasa.gov/bdube/neos_prysm",
              "documentation": "https://github.jpl.nasa.gov/bdube/neos_prysm",
              "keywords": [
                  "neos"
              ],
              "packages": [
                  "neos"
              ],
              "classifiers": [
                  "Intended Audience :: Science/Research",
                  "Programming Language :: Python :: 3.6",
                  "Programming Language :: Python :: 3.7",
                  "Programming Language :: Python :: 3.8"
              ]
          }
      
      OFFENDING RULE: 'additionalProperties'
      
      DEFINITION:
          {
              "$schema": "http://json-schema.org/draft-07/schema",
              "$id": "https://packaging.python.org/en/latest/specifications/declaring-project-metadata/",
              "title": "Package metadata stored in the ``project`` table",
              "type": "object",
              "properties": {
                  "name": {
                      "type": "string",
                      "description": "The name (primary identifier) of the project. MUST be statically defined.",
                      "format": "pep508-identifier"
                  },
                  "version": {
                      "type": "string",
                      "description": "The version of the project as supported by :pep:`440`.",
                      "format": "pep440"
                  },
                  "description": {
                      "type": "string",
                      "$$description": [
                          "The `summary description of the project",
                          "<https://packaging.python.org/specifications/core-metadata/#summary>;`_"
                      ]
                  },
                  "readme": {
                      "$$description": [
                          "`Full/detailed description of the project in the form of a README",
                          "<https://www.python.org/dev/peps/pep-0621/#readme>;`_",
                          "with meaning similar to the one defined in `core metadata's Description",
                          "<https://packaging.python.org/specifications/core-metadata/#description>;`_"
                      ],
                      "oneOf": [
                          {
                              "type": "string",
                              "$$description": [
                                  "Relative path to a text file (UTF-8) containing the full description",
                                  "of the project. If the file path ends in case-insensitive ``.md`` or",
                                  "``.rst`` suffixes, then the content-type is respectively",
                                  "``text/markdown`` or ``text/x-rst``"
                              ]
                          },
                          {
                              "type": "object",
                              "allOf": [
                                  {
                                      "anyOf": [
                                          {
                                              "properties": {
                                                  "file": {
                                                      "type": "string",
                                                      "$$description": [
                                                          "Relative path to a text file containing the full description",
                                                          "of the project."
                                                      ]
                                                  }
                                              },
                                              "required": [
                                                  "file"
                                              ]
                                          },
                                          {
                                              "properties": {
                                                  "text": {
                                                      "type": "string",
                                                      "description": "Full text describing the project."
                                                  }
                                              },
                                              "required": [
                                                  "text"
                                              ]
                                          }
                                      ]
                                  },
                                  {
                                      "properties": {
                                          "content-type": {
                                              "type": "string",
                                              "$$description": [
                                                  "Content-type (:rfc:`1341`) of the full description",
                                                  "(e.g. ``text/markdown``). The ``charset`` parameter is assumed",
                                                  "UTF-8 when not present."
                                              ],
                                              "$comment": "TODO: add regex pattern or format?"
                                          }
                                      },
                                      "required": [
                                          "content-type"
                                      ]
                                  }
                              ]
                          }
                      ]
                  },
                  "requires-python": {
                      "type": "string",
                      "format": "pep508-versionspec",
                      "$$description": [
                          "`The Python version requirements of the project",
                          "<https://packaging.python.org/specifications/core-metadata/#requires-python>;`_."
                      ]
                  },
                  "license": {
                      "description": "`Project license <https://www.python.org/dev/peps/pep-0621/#license>;`_.",
                      "oneOf": [
                          {
                              "properties": {
                                  "file": {
                                      "type": "string",
                                      "$$description": [
                                          "Relative path to the file (UTF-8) which contains the license for the",
                                          "project."
                                      ]
                                  }
                              },
                              "required": [
                                  "file"
                              ]
                          },
                          {
                              "properties": {
                                  "text": {
                                      "type": "string",
                                      "$$description": [
                                          "The license of the project whose meaning is that of the",
                                          "`License field from the core metadata",
                                          "<https://packaging.python.org/specifications/core-metadata/#license>;`_."
                                      ]
                                  }
                              },
                              "required": [
                                  "text"
                              ]
                          }
                      ]
                  },
                  "authors": {
                      "type": "array",
                      "items": {
                          "$id": "#/definitions/author",
                          "title": "Author or Maintainer",
                          "$comment": "https://www.python.org/dev/peps/pep-0621/#authors-maintainers",
                          "type": "object",
                          "properties": {
                              "name": {
                                  "type": "string",
                                  "$$description": [
                                      "MUST be a valid email name, i.e. whatever can be put as a name, before an",
                                      "email, in :rfc:`822`."
                                  ]
                              },
                              "email": {
                                  "type": "string",
                                  "format": "idn-email",
                                  "description": "MUST be a valid email address"
                              }
                          }
                      },
                      "$$description": [
                          "The people or organizations considered to be the 'authors' of the project.",
                          "The exact meaning is open to interpretation (e.g. original or primary authors,",
                          "current maintainers, or owners of the package)."
                      ]
                  },
                  "maintainers": {
                      "type": "array",
                      "items": {
                          "$id": "#/definitions/author",
                          "title": "Author or Maintainer",
                          "$comment": "https://www.python.org/dev/peps/pep-0621/#authors-maintainers",
                          "type": "object",
                          "properties": {
                              "name": {
                                  "type": "string",
                                  "$$description": [
                                      "MUST be a valid email name, i.e. whatever can be put as a name, before an",
                                      "email, in :rfc:`822`."
                                  ]
                              },
                              "email": {
                                  "type": "string",
                                  "format": "idn-email",
                                  "description": "MUST be a valid email address"
                              }
                          }
                      },
                      "$$description": [
                          "The people or organizations considered to be the 'maintainers' of the project.",
                          "Similarly to ``authors``, the exact meaning is open to interpretation."
                      ]
                  },
                  "keywords": {
                      "type": "array",
                      "items": {
                          "type": "string"
                      },
                      "description": "List of keywords to assist searching for the distribution in a larger catalog."
                  },
                  "classifiers": {
                      "type": "array",
                      "items": {
                          "type": "string",
                          "format": "trove-classifier",
                          "description": "`PyPI classifier <https://pypi.org/classifiers/>;`_."
                      },
                      "$$description": [
                          "`Trove classifiers <https://pypi.org/classifiers/>;`_",
                          "which apply to the project."
                      ]
                  },
                  "urls": {
                      "type": "object",
                      "description": "URLs associated with the project in the form ``label => value``.",
                      "additionalProperties": false,
                      "patternProperties": {
                          "^.+$": {
                              "type": "string",
                              "format": "url"
                          }
                      }
                  },
                  "scripts": {
                      "$id": "#/definitions/entry-point-group",
                      "title": "Entry-points",
                      "type": "object",
                      "$$description": [
                          "Entry-points are grouped together to indicate what sort of capabilities they",
                          "provide.",
                          "See the `packaging guides",
                          "<https://packaging.python.org/specifications/entry-points/>;`_",
                          "and `setuptools docs",
                          "<https://setuptools.pypa.io/en/latest/userguide/entry_point.html>;`_",
                          "for more information."
                      ],
                      "propertyNames": {
                          "format": "python-entrypoint-name"
                      },
                      "additionalProperties": false,
                      "patternProperties": {
                          "^.+$": {
                              "type": "string",
                              "$$description": [
                                  "Reference to a Python object. It is either in the form",
                                  "``importable.module``, or ``importable.module:object.attr``."
                              ],
                              "format": "python-entrypoint-reference",
                              "$comment": "https://packaging.python.org/specifications/entry-points/"
                          }
                      }
                  },
                  "gui-scripts": {
                      "$id": "#/definitions/entry-point-group",
                      "title": "Entry-points",
                      "type": "object",
                      "$$description": [
                          "Entry-points are grouped together to indicate what sort of capabilities they",
                          "provide.",
                          "See the `packaging guides",
                          "<https://packaging.python.org/specifications/entry-points/>;`_",
                          "and `setuptools docs",
                          "<https://setuptools.pypa.io/en/latest/userguide/entry_point.html>;`_",
                          "for more information."
                      ],
                      "propertyNames": {
                          "format": "python-entrypoint-name"
                      },
                      "additionalProperties": false,
                      "patternProperties": {
                          "^.+$": {
                              "type": "string",
                              "$$description": [
                                  "Reference to a Python object. It is either in the form",
                                  "``importable.module``, or ``importable.module:object.attr``."
                              ],
                              "format": "python-entrypoint-reference",
                              "$comment": "https://packaging.python.org/specifications/entry-points/"
                          }
                      }
                  },
                  "entry-points": {
                      "$$description": [
                          "Instruct the installer to expose the given modules/functions via",
                          "``entry-point`` discovery mechanism (useful for plugins).",
                          "More information available in the `Python packaging guide",
                          "<https://packaging.python.org/specifications/entry-points/>;`_."
                      ],
                      "propertyNames": {
                          "format": "python-entrypoint-group"
                      },
                      "additionalProperties": false,
                      "patternProperties": {
                          "^.+$": {
                              "$id": "#/definitions/entry-point-group",
                              "title": "Entry-points",
                              "type": "object",
                              "$$description": [
                                  "Entry-points are grouped together to indicate what sort of capabilities they",
                                  "provide.",
                                  "See the `packaging guides",
                                  "<https://packaging.python.org/specifications/entry-points/>;`_",
                                  "and `setuptools docs",
                                  "<https://setuptools.pypa.io/en/latest/userguide/entry_point.html>;`_",
                                  "for more information."
                              ],
                              "propertyNames": {
                                  "format": "python-entrypoint-name"
                              },
                              "additionalProperties": false,
                              "patternProperties": {
                                  "^.+$": {
                                      "type": "string",
                                      "$$description": [
                                          "Reference to a Python object. It is either in the form",
                                          "``importable.module``, or ``importable.module:object.attr``."
                                      ],
                                      "format": "python-entrypoint-reference",
                                      "$comment": "https://packaging.python.org/specifications/entry-points/"
                                  }
                              }
                          }
                      }
                  },
                  "dependencies": {
                      "type": "array",
                      "description": "Project (mandatory) dependencies.",
                      "items": {
                          "$id": "#/definitions/dependency",
                          "title": "Dependency",
                          "type": "string",
                          "description": "Project dependency specification according to PEP 508",
                          "format": "pep508"
                      }
                  },
                  "optional-dependencies": {
                      "type": "object",
                      "description": "Optional dependency for the project",
                      "propertyNames": {
                          "format": "pep508-identifier"
                      },
                      "additionalProperties": false,
                      "patternProperties": {
                          "^.+$": {
                              "type": "array",
                              "items": {
                                  "$id": "#/definitions/dependency",
                                  "title": "Dependency",
                                  "type": "string",
                                  "description": "Project dependency specification according to PEP 508",
                                  "format": "pep508"
                              }
                          }
                      }
                  },
                  "dynamic": {
                      "type": "array",
                      "$$description": [
                          "Specifies which fields are intentionally unspecified and expected to be",
                          "dynamically provided by build tools"
                      ],
                      "items": {
                          "enum": [
                              "version",
                              "description",
                              "readme",
                              "requires-python",
                              "license",
                              "authors",
                              "maintainers",
                              "keywords",
                              "classifiers",
                              "urls",
                              "scripts",
                              "gui-scripts",
                              "entry-points",
                              "dependencies",
                              "optional-dependencies"
                          ]
                      }
                  }
              },
              "required": [
                  "name"
              ],
              "additionalProperties": false,
              "if": {
                  "not": {
                      "required": [
                          "dynamic"
                      ],
                      "properties": {
                          "dynamic": {
                              "contains": {
                                  "const": "version"
                              },
                              "$$description": [
                                  "version is listed in ``dynamic``"
                              ]
                          }
                      }
                  },
                  "$$comment": [
                      "According to :pep:`621`:",
                      "    If the core metadata specification lists a field as \"Required\", then",
                      "    the metadata MUST specify the field statically or list it in dynamic",
                      "In turn, `core metadata`_ defines:",
                      "    The required fields are: Metadata-Version, Name, Version.",
                      "    All the other fields are optional.",
                      "Since ``Metadata-Version`` is defined by the build back-end, ``name`` and",
                      "``version`` are the only mandatory information in ``pyproject.toml``.",
                      ".. _core metadata: https://packaging.python.org/specifications/core-metadata/"
                  ]
              },
              "then": {
                  "required": [
                      "version"
                  ],
                  "$$description": [
                      "version should be statically defined in the ``version`` field"
                  ]
              },
              "definitions": {
                  "author": {
                      "$id": "#/definitions/author",
                      "title": "Author or Maintainer",
                      "$comment": "https://www.python.org/dev/peps/pep-0621/#authors-maintainers",
                      "type": "object",
                      "properties": {
                          "name": {
                              "type": "string",
                              "$$description": [
                                  "MUST be a valid email name, i.e. whatever can be put as a name, before an",
                                  "email, in :rfc:`822`."
                              ]
                          },
                          "email": {
                              "type": "string",
                              "format": "idn-email",
                              "description": "MUST be a valid email address"
                          }
                      }
                  },
                  "entry-point-group": {
                      "$id": "#/definitions/entry-point-group",
                      "title": "Entry-points",
                      "type": "object",
                      "$$description": [
                          "Entry-points are grouped together to indicate what sort of capabilities they",
                          "provide.",
                          "See the `packaging guides",
                          "<https://packaging.python.org/specifications/entry-points/>;`_",
                          "and `setuptools docs",
                          "<https://setuptools.pypa.io/en/latest/userguide/entry_point.html>;`_",
                          "for more information."
                      ],
                      "propertyNames": {
                          "format": "python-entrypoint-name"
                      },
                      "additionalProperties": false,
                      "patternProperties": {
                          "^.+$": {
                              "type": "string",
                              "$$description": [
                                  "Reference to a Python object. It is either in the form",
                                  "``importable.module``, or ``importable.module:object.attr``."
                              ],
                              "format": "python-entrypoint-reference",
                              "$comment": "https://packaging.python.org/specifications/entry-points/"
                          }
                      }
                  },
                  "dependency": {
                      "$id": "#/definitions/dependency",
                      "title": "Dependency",
                      "type": "string",
                      "description": "Project dependency specification according to PEP 508",
                      "format": "pep508"
                  }
              }
          }
      Traceback (most recent call last):
        File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/bdube/miniconda3/envs/neos/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-pgypnloy/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 177, in get_requires_for_build_wheel
          return self._get_build_requires(
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-pgypnloy/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-pgypnloy/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 174, in run_setup
          exec(code, locals())
        File "<string>", line 4, in <module>
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-pgypnloy/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-pgypnloy/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 151, in setup
          dist.parse_config_files()
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-pgypnloy/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 868, in parse_config_files
          pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-pgypnloy/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 58, in apply_configuration
          config = read_configuration(filepath, True, ignore_option_errors, dist)
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-pgypnloy/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 122, in read_configuration
          validate(subset, filepath)
        File "/private/var/folders/82/rq6mxvbx2jsdhmzdmk90c5jh0000gq/T/pip-build-env-pgypnloy/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 47, in validate
          raise error from None
      ValueError: invalid pyproject.toml config: `project`
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
 
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
 
note: This error originates from a subprocess, and is likely not a problem with pip.

Scrolling through this wall of, frankly, non-information, I eventually find that the same blog post that lead me astray on the format of authors lead me astray on homepage, etc, as well. I paste all of these error messages in here inline so you can get the same “wow that’s a load of useless text” that is the user experience getting these errors. Checking the clock, we’re now about three hours into me trying to convert a working setup.cfg into a pyproject.toml, just so that I can locally install my own code. This is an absolute usability disaster. The “no documentation” problem is no different to the way it was back in the days you had to read setuptools’ (or distutils) cryptic documentation. If pyproject.toml is not read, and as best I can tell it is a moving target for users, please do not force it on people by breaking setuptools. My time is my employer’s money, and mucking about in thousands of lines of cryptic error messages is a good use of neither.

Please, I implore you, either retract pyproject.toml until it is actually working, un break setuptools for alternatives like setup.cfg, or improve the documentation. Maybe all three.

The relevant line seems to be right at the very top of the error:

configuration error: `project` must not contain:
     {'packages', 'homepage', 'python', 'documentation', 'repository'} properties

However, some of those properties are clearly present in your project config (which is conveniently reproduced in full in the error message as GIVEN VALUE).

EDIT: the latest PEP regarding pyproject.toml has what appears to be a fairly comprehensive example. There you can see that URLs (e.g. “homepage”, “documentation”, etc.) go under a [project.urls] section.

BTW I am just a bystander, not a packaging expert or packaging tools maintainer. But I’ve managed a large OSS project for more than a decade and gotten terrific help here on multiple occasions (and been in the position to dispense support help for my project elsewhere on thousands of occasions). Purely by way of friendly advice: you might catch more flies with honey.

5 Likes

@anon82781497 according to the first error message you reported, it seems that your original configuration is missing the packages = declaration (e.g. packages = neos) inside the [options] section. Maybe you would like to share with us your original setup.cfg? You can always open an issue in the setuptools repository with a reproducer and we will have a look on why things are going wrong.

In terms of pyproject.toml, it is like @bryevdv says, the error messages are giving you hints of what is going wrong. Yes, there is a traceback that you might not find important (but other people may find it relevant…)

I’m sorry you had such a frustrating experience! Hopefully we can make it better for others in the future.

Instead of manually converting your setup.cfg by hand, did you try just running ini2toml to do so automatically? This would have likely avoided at least the great majority of these issues, provided more helpful feedback the remaining, and taken much less time overall.

Also, while the messages are rather detailed, at least the pyproject.toml validation-level ones have the actual error message featured right near the top, and the remaining details can be highly useful to those looking to diagnose the problem—I’d much rather all the key details than be left guessing. Maybe the actual error message could be made more prominent in the output?

The message here is not specific to setup.cfg (AFAIK) and rather appears to be due to the new check @abravalheri added for multiple top-level import packages when using an implicit (“flat”) project layout, which is usually a (common) mistake. If you have explicitly set packages with just the name of your package and the error message is still occurring, then its possible that either the error message should be revised, or something is not quite right with the check. I could not reproduce the problem locally with a simple test package following the same layout, but if you can (or at least can boil it down to what’s causing it), you might want to report a Setuptools issue if this hasn’t been reported already, as @abravalheri mentioned.

It might be a good idea for pip (in this case) to except and wrap the low-level error and re-raise it without a long traceback and a more helpful preamble to the error message explaining the pyproject.toml has a TOML syntax error.

To be fair, they shouldn’t ever see this, as any attempt to actually build a distribution package with a syntax-errored pyproject.toml would fail early and loudly on your end (as it did).

The spec is perhaps not as clear as it could be, but as you figured out, there is currently no top-level string value for license; that is reserved for a SPDX identifier/expression (in your case, it would be LicenseRef-Proprietary) as defined by the (currently still draft) PEP 639.

“Table” is the standard, correct term for that construct in TOML, similar to a Python dict, and is used consistently for such in the spec. “Object” is a term specific to JavaScript, and has a very different meaning in Python (and most other languages/formats), so would not only incorrect but also potentially quite confusing. To note, I do plan to add the examples in the PEP to the spec to help aid understanding, as well as propose some other clarifications to the terminology.

In fact, AFAIK there haven’t been any substantive changes to the valid values for any of the standardized tables in pyproject.toml since the specs were finalized, approved and implemented. What you may have found is older syntax that only applied to the legacy, tool-specific setup.cfg or setup.py formats, or simply posts that were flat-out incorrect.

It may certainly not be perfect, but I’m not sure what you mean by “no documentation”. The official packaging Python projects tutorial was recently updated to focus specifically on explaining how to use pyproject.toml; the Setuptools docs has a guide on using them with Setuptools, and the spec itself provides relatively concise but complete details on the [project] table.

The standard [build-system] and [project] tables have been stable and fixed since the beginning of when tools first implemented them, as they are standardized in formal specifications that all tools follow. Backend (e.g. Setuptools)-specific options, by contrast, may still until finalized, but they are clearly labelled experimental with a rather prominent warning message in the docs:

Perhaps, then, your employer might consider contributing either time or money toward helping improve things for everyone?

6 Likes

(trying to go in approximate order, and I apologize but I had to remove several links due to a 2 link limit for new users)

I think the relevant line being at the top of an error is an ergonomics issue. Especially when the errors are long (here, up to more than 500 lines) - the user sees the bottom of the error first in the console, and has to scroll up. If you are inside a screen or tmux session on a remote server with limited scrollback, it may not be visible at all.

The original setup.cfg was,

[metadata]
name = neos
author = Brandon Dube
author-email = brandon.dube@jpl.nasa.gov
description = NEOS integrated diffraction model
packages = neos
license = unlicensed
platform = any
classifiers =
    Intended Audience :: Science/Research
    Programming Language :: Python :: 3.6
    Programming Language :: Python :: 3.7
    Programming Language :: Python :: 3.8

[options]
zip_safe = true
include_package_data = true
python_requires = >= 3.6
tests_require = pytest
test_suite = tests
setup_requires =
    setuptools >= 38.3.0
    setuptools_scm
install_requires =
    numpy
    scipy
    astropy

[options.packages.find]
exclude = tests/, docs

[bdist_wheel]
universal = true

[sdist]
formats = gztar

[flake8]
max-line-length = 120
exclude = .git, .eggs, __pycache__, tests/, docs/, build/, dist/

I have been copy-pasting the same setup.cfg and tweaking it for new projects for a while. The lower bound on setuptools version is probably a good indicator for the last time I had to change it.

I did not try running an automated tool to convert the setup.cfg to a pyproject.toml; I looked at a few examples online and saw the main ‘heading’ had changed and figured that would be a futile exercise, since things would have to move anyway and the syntax is generally similar, mostly a matter of putting strings in quotes.

I think very long error messages in a console output are not useful to an end user. I may sit somewhere between engineer or scientist and “software developer” and be willing to look through errors (indeed, that is how I made any forward progress at all in this process for so many hours) but for those for which the code is a necessary evil, not even a means to an end, an error that does not say in a few sentences what is wrong and how to fix it is probably not going to be read. I’ve even gotten flak for not catching array shape mismatches in my code and effectively just changing Numpy’s error messages as “unhelpful” errors! I’m not sure how the numpy shape mismatch error could be improved, but people find even those, which are super terse, imperfect. By comparison, hundreds of lines is, well, less terse.

Perhaps the very long errors could be dumped to a file, say if len(str(err).splitlines()) > 20: dump_err_to(pwd/'setuptools_error.txt', err) or something, with only the “important” lines printed to the console. Important could be first 3 and last 3 or something, not quite sure. That way those who want the full error can still have it, but users who are only going to be frustrated by it don’t see it. This could be bad UX, too, because it is different to other error handling and so unfamiliar.

To be fair, they shouldn’t ever see this, as any attempt to actually build a distribution package with a syntax-errored pyproject.toml would fail early and loudly on your end (as it did).

I’m less sure of this. My standard workflow is to work in a jupyter-based sandbox until the functions or what-not are “done” and then move them into a module. So most of my packages, at least on my machine, have <the_pkg> nb data as folders, and it’s expected that a pip install https://... would get the data directory, too (which a MANIFEST file does do, in “old world” packaging). However, if pwd ever points to the enclosing folder or it is otherwise in sys.path, then python’s path searching semantics can find neos and import it. I feel that I follow relatively due diligence to always pip install -e . packages for local development, but at my workplace there is a prevalant culture of sys.path hacking. I reported a bug in someone’s code that python setup.py install (in their manual) does not work, and found that it hasn’t been a correct setup.py file for over ten years! This story to say, unfortunately, I think invalid package specifying files are not so uncommon. There is also the case of open sourcing something, which we can do but often means copying all of the code into a new git repo and manually syncing and pushing every few months. Fully in the open development is difficult due to organization policy to review materials prior to dissemination. In this world, the developer may not even use the open source copy at all. I have certainly tried to run the open source copy of colleague’s code and had it not work at all, and I’m sure been guilty of the same myself. In every case, asking them about it resulted in finding the closed copy works just fine, and it’s just a copy-paste type problem.

The preferred answer to these problem may just be “do a better job,” which is fair, but a lot of the scientists’ code is undocumented, difficult to read, does not follow PEP8, black, or any other standard formatting. Those probably should be taken care of before packaging concerns. The python story is so much more difficult than go.mod or similar to get working (not that gomod doesn’t have its own sharp edges). Maybe it is out of scope for pip, but a pip init CLI that produces a valid, minimal setup with whatever the packaging setup du jour would go a long way. I guess if it went in setuptools used as python -m setuptools init or something, then the difficulty can be made the computer’s problem.

(in your case, it would be LicenseRef-Proprietary) as defined by the (currently still draft) PEP 639.

In my opinion, if pyproject.toml represents current best/desired practice (and setuptools dying on this setup.cfg now leaves few other options!) then there should be no draft content required for use. Packages can have a long lifetime and these files are rarely updated. If a user writes a pyproject.toml today and have to fix it in two years just because the spec changed, I do not think they will think kindly of pyproject.toml.

“Table” is the standard, correct term for that construct in TOML, similar to a Python dict, and is used consistently for such in the spec. “Object” is a term specific to JavaScript, and has a very different meaning in Python (and most other languages/formats), so would not only incorrect but also potentially quite confusing.

Given that even the current release of python (3.10) does not have TOML included with the batteries, I think using terms from the TOML specification is misguided. I don’t think I have read the standard for any file format I use (JSON, YAML, FITS, TIFF, JPG, PNG, h5, .np, .mat, …). I could not tell you what words any of them use to describe their constructs.

I disagree that Object is specific to Javascript – the term “object oriented” programming is nearly 60 years old, and it is common vernacular. For example, many people refer to structs with methods in Go as “objects” and folks seem none-too-bothered by the imprecision in the language; what is meant is understood, even if it is a technically imperfect label.

If the desire is not to use the word “object” then it may be palatable to describe things in terms of the python types they map to. Where list vs tuple would be ambiguous, just saying list is probably good enough. When I saw the word “Table” I thought “yeah, no idea what that means” and had to search for examples to see, “oh, it’s basically a dictionary.”

In fact, AFAIK there haven’t been any substantive changes to the valid values for any of the standardized tables in pyproject.toml since the specs were finalized, approved and implemented. What you may have found is older syntax that only applied to the legacy, tool-specific setup.cfg or setup.py formats, or simply posts that were flat-out incorrect.

I don’t know either, but the post in question is result #1 on Google for me:

PyPa is not responsible for third party posts and SEO is dark art not science, but if the first results on google do not work, it leads only to frustration.

It may certainly not be perfect, but I’m not sure what you mean by “no documentation”.

The words are not meant so literally. In the days of yore, it’s certainly true that distutils has/had docs and if you wanted to know how to craft a seutp.py (and later setup.cfg) the answer was to reference these. Widespread sentiment was or is that these docs are not so useful for users; in the documentation system they fill the role of reference, but not tutorial or how-to; there is no “getting started” so to speak.

The situation today is more or less comparable. You can find examples on Google, and result #2 is the setuptools docs which specifies… how to put setuptools stuff in pyproject.toml, not “how to make a pyproject.toml.” Eventually you find PEP 621, which is ~= the old distutils setupscripts docs.

The official packaging Python projects tutorial

In probably too harsh of a criticism, I have opened and immediately closed this tutorial as “wrong” many times. In the descriptive vs prescriptive sense (search “prescriptive vs descriptive grammar” if unfamiliar… link limit reached), it is prescriptive. It opens by telling the reader to place their code in a src directory, but the majority of commonly used python packages do not do thi, e.g. numpy, scipy, requests.

In rare examples of packages which do use src, you can find e.g. matplotlib or flask. But at least of the packages I use, src is the overwhelming minority. Many (myself included) find numpy and scipy to be the gold standard of how to do python and so where the official tutorials prescribe that you do differently, “surely the tutorials are wrong”. I think there is large sentiment, too, that python packaging is a fractured landscape and “official” or not, this is just one opinion. I don’t actually know what official means, either. Years ago, after the language “official” had started being used, there was no relationship between PSF and PyPa, but maybe that has changed.

the Setuptools docs has a guide on using them with Setuptools

Unfortunately, though, the fragment shared there is not a working pyproject.toml in its entirety.

the spec itself provides relatively concise but complete details on the [project] table.

(Echoing my previous comment) - I have never read the spec on any file format I use, and I think asking users to read the spec to find an example is a bar the overwhelming majority will fail to meet.

Backend (e.g. Setuptools)-specific options, by contrast, may still until finalized, but they are clearly labelled experimental with a rather prominent warning message in the docs:

My 2c on the whole packaging fiasco is that the only things that are reliable are pip and setuptools, because they come with python and have some air of authority that results from that. Conda, poetry, whatever, regardless of whether you use them they aren’t “python” they are just tools for python. If pyproject.toml is erasing the past (setup.cfg, etc) then it is essential that that erasure not take its first step before there is rock solid integration between pyproject.toml and setuptools is complete. If it is nascent today, and “beta” is the very definition of not not rock solid in software.

Perhaps, then, your employer might consider contributing either time or money toward helping improve things for everyone?

I could ask; we are not forbidden from contributing to open source. Actually, the policy there is rather permissive (if there is no business conflict, just proceed). But funding time is probably forbidden through government accounting rules. On the money side, the same; as an FFRDC money comes in, but for it to leave without “buying goods or services” is a non-starter.

The problem with your original setup.cfg file is that you have the packages line under the [metadata] section when it actually belongs under [options]. Once you fix that, you should find that modern setuptools builds your project successfully.

If your package has been working properly with that setup.cfg previously, then it’s really been by chance, and I suspect that your wheels have not contained what you wanted them to contain.

1 Like

But who is the end-user in this case? If you mean your users, then they should never be exposed to any of this in the first place. It’s not relevant to them, because they aren’t building your package, you are are. So really, the “end-users” in this case are the small subset of people who are creating Python packages, who reasonably can be expected to need to know more about the details of packaging.

In any case, speaking as one of those users who builds packages, I think it is vastly preferable to have more information than less. To wit: you never actually included your failing config in your original post, and the only reason why anyone was able to speculate in any detail at all is because the error message did. If nothing else, long error messages make it easier for you to help other people help you. [1]


  1. I guess I am venting years of frustration here. I have had to say “I can’t help you without more information” literally hundreds of times of the last decade. Anything that helps users arrive with more details in hand when they seek help is an unquestionable good thing to me. ↩︎

3 Likes

that seems like an error something should have been able to detect and
complain about, no? “invalid setup.cfg”

With regards to invalid setup.cfg - wheels on public PyPI borne of the setup.cfg that was copy-pasted to make this one do indeed include all source and data files, so things did work properly.

It’s not relevant to them, because they aren’t building your package, you are are

I do not agree with this statement. Within my org, installation instructions of “git clone and pip install -e” are the norm. All of these error messages were generated by that process, which all users will undertake.

@mwichmann didn’t it?

error: Multiple top-level packages discovered in a flat-layout: ['nb', 'neos', 'data'].

The message could be more actionable or point to other resources, for sure. But “detect and complain about” is exactly what it did.

Unfortunately, moving the packages = into [options] doesn’t appease setuptools

Surely that’s a process issue, the process is forcing users who don’t have the knowledge or interest in dealing with packaging issues, to be exposed to them.

I’m not saying that organisations must never have process issues, or that we should somehow punish organisations that do. But it’s not the responsibility of the packaging ecosystem to work perfectly under every possible workflow, no matter how dysfunctional it is. We do our best, and we’ll happily advise you on how you could improve your workflow to be more in line with the tools’ expectations, but at some point we have to say “well, don’t do that”. Also, please remember that pretty much everyone maintaining Python’s packaging tools is a volunteer, so complaining that it’s all the fault of the tools is likely to have the opposite effect than what you intend - it will simply demotivate the people you need to help you, who will focus on interactions that are less hostile.

2 Likes

Surely that’s a process issue, the process is forcing users who don’t have the knowledge or interest in dealing with packaging issues, to be exposed to them.

Setting aside the, “your cell reception problem is because you’re holding the phone wrong” nature of this response, perhaps a little background.

I am an optical engineer, who happens to write a lot of software for simulation and analysis (actually, the group I work in is literally called Optical Simulation and Analysis). The users in this case are other engineers and scientists who release their own code and are familiar with python, matlab IDL, whatever language is being used. We could quibble over whether “writes python” and “is knowledgeable about packaging issues” are synonyms or not, but it is certainly true that none of these people (myself included) are interested in dealing with packaging issues. For example, a real-time control system I implemented in Go has the following installation instructions:

  1. Clone the repo
  2. Type go build
  3. run the program

The Go tooling makes getting all of the dependencies, even those on our github enterprise that are not visible to the public completely plainless. Or, the story with matlab where it’s just “download this folder of m files.” Whether that distribution method is good or bad is separate to whether it is “easy.”

With python the analagous directions are,

  1. Clone the repo
  2. Run conda env create -f conda.yml and activate the environment
  3. Type python -m pip install -e .
  4. import the library into your script, jupyter notebook, etc

software of the type that is in question is used by a wide range of engineers and scientists, with a wide range of operating systems, processor architectures, or even access to accelerators like GPUs. Could wheels or sdists be made and distributed? Sure, anything is possible. Is it convenient, when this workflow requires only git pull to get updates? No, it is not. Is management generally supportive of “meta work” that is new and intangible in its benefit? No. And this git and pip based workflow has been working for more than ten years. And the very notion that things are a package and users aren’t sys path hacking to get the code importable is a step forward.


To be honest, I find the notion that the tools have expectations to be rather user-hostile. It is true that when you carve out your own space (a library, say) you have no “duty” to users; the no warranty clause in most open licenses is about this very topic. But tools such as pip do have a duty to users, and to python as a cultural phenomenon. I have been using python for a very long time and have tried most of the alternatives (Julia, etc) and found them to be inferior for my use case. But if packaging gets to be so bad that I can’t even write the necessary configuration file to define a package in half a working day, due to breaking changes in version 10 30 60+ of the packaging tools, I will eventually be pushed away from python entirely. It does not need to be this hard. Bad packaging for python has been the subject of xkcd and widespread criticism for a very, very, very long time. The only “new” problem is that what is old and used to work, now no longer works and we are left with rather the same great level of difficulty in using the new solution, which will itself one day be erased in favor of the next solution du jour.

I am also a volunteer and have invested many thousands of hours into releasing, writing docs for, and writing tests for open source python packages in my spare time. Are they as widely used as software like pip, or setuptools - no. But please do not insinuate that I am not also a “python volunteer.”

Getting back to the original problem of setup.cfg not working, I’d like more information as to why my suggested fix of moving packages to [options] didn’t work. Did you get the same error as in the OP, or a different error? Can you describe your project layout (possibly in the form of tree output) or, even better, link to a publicly-accessible code repository?

Setuptools has not removed support for setup.cfg in any way, so your old configs should still keep working, but only if they were valid to begin with.

Also, FYI, I’ve e-mailed the author of that pyproject.toml blog post and informed them of the errors therein; they appear to have mixed up the PEP 621 [project] table with Poetry’s [tool.poetry] table.

Normal package creators do not know the pip internals either. To me it looks like some applications including pip have a “debug-mode” traceback permanently enabled.

IMHO a finished application should not show a (full) traceback but a descriptive error message. …and I understand it requires additional work to equip an application with reasonable error messages and to maintain them.

If the user wants more information about the error they should reproduce it with traceback enabled or if we are expecting hard-to-reproduce errors the traceback can be dumped to a debug log file by default.

1 Like

@anon82781497 sending a warning as a mod here. I understand you’re frustrated, but the way you’re communicating in this thread is not acceptable. The people responding have been incredibly gracious so far. Stop responding with novel length posts and tone down the angry finger pointing.

For future readers confused by Python package installation who might stumble on this, a few tips

  • Installing in editable mode (using the -e flag) is not needed to install from a local directory, in the stated use case pip install . should have been enough
  • In fact, there was no need to even clone the repositories: pip can install directly from a git repository url (and of course, one does not need to clone a git repository to download its content either).
  • If you end up installing packages with pip, conda might not be the tools you want: pyenv and virtualenv are worth checking out.