Help testing experimental features in setuptools

Thank you very much for the feedback Lewis. Most of the points you specified are definitely corrections to be made. I am collecting all these issues in a backlog!

If you add trove-classifiers to your build-system: requires it should do the validation. Let me know if it does not work.

I will have a look on minesweeper to understand what is happening. I was doing an experiment with pybind11, and it seemed to work at that point it time.

Ah in which case it could well be something to do with me switching to namespace packages at the same time or something - I’m happy to diagnose a bit further and let you know what I find :slight_smile:

2 Likes

Was this the case before to validate Trove tags? Can this be made a default dependency? Its pretty tiny and requiring every user add boilerplate to build-system.requires to get this validation, which will cause PyPI upload to fail, seems rather non-ideal.

The intention (with PEP 639, and in the original PEP 621) is to deprecate both and simply have a flat string value for project.license that is a SPDX identifier, e.g. MIT, or expression.

I might be wrong here, but I think currently no validation is done at build time. I will try to improve the situation.

Currently, it is not easy to add a default dependency because that would add a dependency to setuptools itself and create a cycle (that might be improved in the future).

2 Likes

I believe the idea is to add it to twine check so that built
packages can be tested for PyPI compatibility prior to attempting to
upload anything. See `twine check` should guard against things not accepted by PyPI like version format · Issue #430 · pypa/twine · GitHub for
discussion.

3 Likes

I’ve now realised this is standard with setuptools entrypoints, although I still wonder why it has to be this way.

These two points are no longer a problem and can be ignored - sorry for the noise :slight_smile:

2 Likes

Is there currently not support for the build_editable hook?

I am testing out some features with the experimental branch, but I am finding it really difficult to find much information on the build_editable hook. If there is support, it would allow further testing. I understand this is all up in there air and there currently is no “best practice”.

Thank you.

I have to double check that. I have not explicitly removed any entry-point and I imagine that the callback is still there. Maybe this problem is related to the auto-discovery just happening after the plugin was already processed? (I will have a look on that, thank you very much for the pointer!)

Looks like that was my mistake; the setuptools.finalize_distribution_options hook does still work, I must have just left out the plugin from the build-system.requires after running ini2toml!

2 Likes

I am afraid that is not included in this batch of changes.

I also had a problem entry point validation. The error contradicts the description:

Schema: data.anyio must be python-entrypoint-reference

Given value:
"anyio.pytest_plugin"

Offending rule: "format"

Definition:
{
  "type": "string",
  "$$description": [
    "Reference to a Python object. It is either in the form",
    "``importable.module``, or ``importable.module:object.attr``."
  ],
  "format": "python-entrypoint-reference",
  "$comment": "https://packaging.python.org/specifications/entry-points/"
}
2 Likes

Seems like this is what @LewisGaul ran into above, expecting that __main__.py and/or the if name == "__main__" block would be executed if just a module/package was specified (i.e. like with python -m), but evidently not. This has been my experience as well with the current setuptools setup.cfg config, so it seems like the limitation may pre-date and not be specific to this new feature (though it still should be fixed).

2 Likes

The problem faced by @agronholm is an error in the validation, indeed the standard seems to allow entry points without the :obj.attr part. I am correcting this in the supporting validate-pyproject library.

I am not sure about this behaviour being standard though. I just did a quick test (with stable setuptools, not my experimental branch) with a dummy project using a console script with entry-point in the form of executable = package.module. Running python -m build works perfectly, however installing the package with pip install . fails:

ERROR: For req: myproj==0.0.post1.dev1+g93a69b6.d20220222. Invalid script entry point: <ExportEntry skeleton = myproj.skeleton:None []> - A callable suffix is required. Cf https://packaging.python.org/specifications/entry-points/#use-for-scripts for more information.

I think this is related to the fact that pip is used to build the CLI wrappers. I recon that without the :obj.attr part, pip does not know which function to call inside the wrapper.

Please note that I am not 100% sure about this, It might be just a pre-existing limitation of setuptools as @CAM-Gerlach mentioned. However I do have the impression that the standard requires the entry-point for console_scripts to point to an specific function:

Two groups of entry points have special significance in packaging: console_scripts and gui_scripts . In both groups, the name of the entry point should be usable as a command in a system shell after the package is installed. The object reference points to a function which will be called with no arguments when this command is run.

(This “requirement” is not explicitly mandatory, but the text does suggest it).

I see that you have fixed the problem in validate-pyproject. When should we expect an update to the experimental branch?

Hi @agronholm I have been working lately on the different feedbacks I received here, creating PRs in the different repositories involved (distutils, validate-pyproject, etc…)

There is one major feedback that I still haven’t had the chance to address:

I plan to update the experimental branch as soon as I manage to solve this problem.

2 Likes

Hello all,

I just updated the GitHub - pypa/setuptools at experimental/support-pyproject branch of setuptools with the latest improvements based on the feedback received so far. Thank you very much.

There are still some things pending, but I think the major issues were solved.

  1. Improved package discovery to prevent errors with version = {attr = ...} because of missing package_dir.
  2. The correct content-type (md/rst/txt) will be set in the PKG-INFO when project.readme is set to a string instead of a table (previously it was default-ing to rst).
  3. Trove classifiers will be downloaded if the package is not installed, or ignored if the download fails. An entry will be logged in the DEBUG level about the download though.
  4. URLs not starting with a schema will not fail (but I am still leaving a warning, since the W3C says URLs should have a schema, unless they are a local relative URL)
  5. Several improvements in validate-pyproject and error messages a bit more clear.
    • To decrease verbosity I am relying on logging: some things are being logged using the ERROR level and the details use the DEBUG level.
    • However the existing logging configuration in setuptools is a bit complex, and therefore the default behaviour ends up being verbose.
    • It seems to be possible to use 2>/tmp/null or 1>/tmp/null to make the output quieter.
  6. Prevent warn on false positive for author/maintainer's email by abravalheri · Pull Request #116 · pypa/distutils · GitHub

Not directly related but I also implemented several improvements/fixes on ini2toml, e.g. fixing the missing terminating line, adding an heuristic to detect the most suitable format for TOML strings.

4 Likes

I’ve tested with

[build-system]
   requires = [
       "wheel",
       "setuptools @ git+https://github.com/pypa/setuptools@experimental/support-pyproject"
   ]
   build-backend = "setuptools.build_meta"

[project]
    name = "pkg_dir_name"
    # and all other required
    dynamic = ['version']

[tool.setuptools]
    package-dir = {"" = "pkg_dir_name"}
    include-package-data = true

[tool.setuptools.packages]
    find.where = ["pkg_dir_name"]
    find.exclude = ["tests", "scripts", "utils"]

[tool.setuptools.dynamic]
    version = {attr = "__version__"}

It completed first step OK

* Building sdist...
...
Creating tar archive

but failed on next

* Building wheel from sdist
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (setuptools @ git+https://github.com/pypa/setuptools@experimental/support-pyproject, wheel)
* Getting dependencies for wheel...
...
Traceback (most recent call last):
  File "/tmp/build-env-r5a4apod/lib/python3.9/site-packages/setuptools/config/expand.py", line 79, in __getattr__
    return next(ast.literal_eval(value) for value in matching_values)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/build-env-r5a4apod/lib/python3.9/site-packages/setuptools/config/expand.py", line 178, in read_attr
    return getattr(StaticModule(module_name, spec), attr_name)
  File "/tmp/build-env-r5a4apod/lib/python3.9/site-packages/setuptools/config/expand.py", line 81, in __getattr__
    raise AttributeError(f"{self.name} has no attribute {attr}") from e
AttributeError: __init__ has no attribute __version__

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
...
...
  File "/tmp/build-env-zde5d6vt/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 253, in _expand_dynamic
    return _expand.read_attr(directive["attr"], package_dir, root_dir)
  File "/tmp/build-env-zde5d6vt/lib/python3.9/site-packages/setuptools/config/expand.py", line 182, in read_attr
    return getattr(module, attr_name)
AttributeError: module '__init__' has no attribute '__version__'

While tree looks more or less:

$ tree -L 1

pkg_dir_name_repository
├── pkg_dir_name
│   ├── submodule
│   ├── __init__.py     # contains __version__ = '1.0.0'
│   └── entrypoint.py
├── LICENSE
├── pyproject.toml
├── README.md
├── requirements-dev.txt
├── requirements.txt
├── scripts
├── tests
├── utils
└── venv

If I omit find.where

[tool.setuptools]
    package-dir = {"" = "pkg_dir_name"}                                                                                           
    include-package-data = true

[tool.setuptools.packages]
    # find.where = ["pkg_dir_name"]
    find.exclude = ["tests", "scripts", "utils"]

It completes both steps successfully, but also venv and other items are packed, including those “excluded”.

Can you say what this line is for?

If i read [`pyproject.toml`: part 3] Add means to apply configuration from `pyproject.toml` by abravalheri · Pull Request #3067 · pypa/setuptools · GitHub correctly, it should be set, in order to discover dynamic version properly. Setup.py, which i was converting to pyproject.toml had also dynamic entry points, but I wanted do it step by step. However, even if i don’t read above correctly, commenting this out fails with the same exception as above.

Hi @kkrolczyk thank you very much for the feedback, do you have a repository somewhere that I could have a look?

There are definitely better error messages that could be implemented here.

As pointed out by Eric, the package-dir configuration looks a bit off… When you do package-dir = {"" = "pkg_dir_name"} you are telling setuptools to look inside pkg_dir_name, which excludes pkg_dir_name itself… Maybe that is the origin of the error?

Also setuptools should complain about version = {attr = "__version__"}, instead of just leaving it pass. The expected syntax is {attr = "modname.__version__"}.

This sounds like a bug. I have to look into this one in more detail (the things documented in https://setuptools--2894.org.readthedocs.build/en/2894/userguide/package_discovery.html#automatic-discovery should work). If you have a reproduction that would really help!

Could you try to simply remove the package-dir and the find.where configs, and use something like?

[tool.setuptools.packages]
    find.exclude = ["tests", "scripts", "utils"]

[tool.setuptools.dynamic]
    version = {attr = "pkg_dir_name.__version__"}

(Both find.where and package-dir will look inside a directory, but not the directory itself)