Continuing the discussion from Meta: organizing several proposals related to the future of pyproject.toml:
I have just begun writing this up as a PEP locally; but I’m laying the idea out here - in more detail, but still informally - before I get too deep into it.
I pitched the concept in the previous thread like so:
I misspoke there. In my original conception, the
build-wheel target would have been for dependencies at build time; as I was writing this post, I dropped it entirely.
pyproject.toml spec is expanded to include optional
[required-for] tables. There are a lot of use-cases to handle; this proposal is designed to push the complexity onto tool implementers - one writes things in
pyproject.toml that are intuitive, and tools are expected to do the right thing.
The keys of
required-to can have any name, like
required-to.<task-name>, where is some string describing a “task” that can be done with the code such as running it, testing it, building a wheel etc. Certain names have explicit meaning defined by the spec; names with a leading underscore are for “private” use by individual tool suites, or even given an idiosyncratic meaning by the developer. Other names not starting with an underscore are left up to the community to standardize; it is hoped that if multiple tools exist to perform a task that isn’t covered here, the authors can collaborate to name and define the task.
This scheme gives a consistent, regular, readable description of the purpose of each list of dependencies, while allowing generalization to any conceivable such purpose. The phrase “required to” is followed by some action that causes the requirement, while “required for” is followed by a noun describing the thing that imposes the requirement (and then by the corresponding action - it seems redundant to try to insert an extra “to” here).
The values for each such key are a list of requirements, in identical format to the current
<task-name> keys under separate
required-for.<extra-name> tables (i.e., sub-tables of
required-for) are used to define additional dependencies needed for the task-name under the condition that the extra-name is available.
I plan to define the following task names and their semantics explicitly:
install- dependencies that must be installed when the wheel is installed (as well as when testing); i.e., dependencies required at runtime when the code from the wheel is used.
run- dependencies needed simply for running the code, without producing a wheel.
- This is meant for simplicity for people who aren’t intending to build a wheel; however, it’s conceivable that the same code could have use as both a library and a Pipx-installed standalone application, and have different dependencies in those two contexts.
test- dependencies that must also be installed when testing the code, including the test harness itself. This allows for pinning a version of Pytest, for example.
- A test runner would put dependencies from
testinto the test environment, as well as dependencies from
use-wheelif that key is present, and
- A test runner would put dependencies from
use-wheel- dependencies that must be installed when the wheel is installed, that are not used in testing. For example, an application distributed as a wheel, that uses Requests, might mock out all the networking calls for testing, and not want to include it in the test environment.
- Conceivably, this could be used to share “third-party package data” - for example, if multiple image processing libraries want to provide the same sample data for tutorials. I don’t know how useful this is, but it seems pointless and impractical to forbid it.
build-doc- dependencies needed for preparing documentation, including the documentation writer itself. This allows for pinning a version of Sphinx, for example.
- Conceivably, this could be [ab]used for building other distributables that are ancillary to the actual code. However, it would be better to figure out what those might be, and let others define separate names for such tasks.
develop- dependencies used in a development environment, such as Black, MyPy,
precommit, a linter etc. This would allow multiple devs on the same project to clone a consistent development environment.
The information currently in
[project.dependencies]can now be split across
[required-for.<extra-name>.use-wheel]. It would be an error for
pyproject.tomlto include both
[project.dependencies]along with either of the corresponding
required-toentries, and similarly for the optional dependencies. (“In the face of ambiguity, refuse the temptation to guess.”)
- However, now it’s possible to make this split, and there’s a clear use case for it.
Other targets are approximated by third-party tools already. For example, Poetry’s
[tool.poetry.dev-dependencies]would subsume many of the new entries. The new scheme allows for much more specificity.
[required-for.<extra-name>.run]entry is present (even if empty), but there is no
[build-system]table nor any of
[required-for.<extra-name>.use-wheel]entries, this signifies that the project is not intended to build a wheel, and tools shall not attempt to do so.
This allows for protection against Pip downloading an sdist of something intended to work only as an “in-place” application and trying to install it as a library.
We could perhaps relax the requirement to specify
[project.version]in these cases. That seems worth separate discussion.
Aside from that, in general the
installlists are meant to be fallbacks for each other. A script runner would set up dependencies from
runis absent (as well as
test, of course); a test harness would set up dependencies from
installis absent. (Either would ignore
For symmetry, it would seem to make sense to include a
build-wheeltask. However, this the one thing where it doesn’t make sense to offer extras-specific variants, and it would be completely redundant with
[build-system].requiresas things stand.
I know I originally was opposed to a
[run]table; the new
[required-to.run]is different in that it’s expressly only for describing dependencies (and not other environment setup). It’s also part of a more general system, that tries to put the various use cases for Python code on relatively even footing (rather than holding wheel-building above all else).
The version of Python required for any task that ultimately involves executing the
.pyfiles in the project, ought to be the same regardless of that task. Therefore, there is no attempt to specify a Python version in the new tables; that’s what
[project.requires-python]is for. While that corresponds to a core metadata field in wheels, that’s not to say that this is the sole purpose of the information; the documentation describes this value as simply “The Python version requirements of the project.”, and I’m not changing that. It’s fine if e.g. script runners don’t have this information; they are free to complain, or else try using whatever version is available.