It would be nice to define something about the relationship, just to clarify confusing interactions. The thing that springs to mind is if someone defined the same name in two places: once as a key in optional-dependencies and a second time in the requirements table.
Just forbidding that is simple enough–if it ever makes sense in the future the prohibition could be lifted with a new proposal.
At the risk of being a cookie licker, may I take a crack at it next week? It’s very very close to what I was thinking of proposing (if anything, I was going to complicate it with unnecessary bits!).
I haven’t written a PEP before but I think I’m familiar with the problem space, as I maintain libraries, distributed applications, and webapps. I was looking forward to gaining the experience.
Yep, that’s fine by me. You can list me as the sponsor if the proposal adheres closely to what I outlined.
You can also list me as a co-author if you would rather go that route.
This reminds me a lot of how Hatch handles environments (which I love and use extensively now)
Especially the ability to easily specify whether to include/exclude the “project” itself
A couple additions on top of this (pretty much all inspired from existing Hatch features) would go a very long way to covering every use case I particularly want
pre-/post-install commands
A command or list of commands (i.e. string or array of strings) that should be run before and after installing the requirements
scripts/aliases
A table of shortcut commands for each of the environments, also allowing references to other aliases
Not 100% sure on the semantics of this, but also
As a hopefully-helpful cross-post, I’ve just put up a draft PEP for a proposal based on @brettcannon’s “strawman” above, with very slight modifications based on my concerns around future extensibility:
It probably needs a good amount of further refinement, but it is a concrete artifact which I’m hoping will help push the discussion, and the tools we provide to users, forward.
Thanks. I agree that this is a useful, actionable step. However, I think there’s still a very fundamental problem that we haven’t made much progress on - if a project (in the broadest possible sense of the term) is not intended to be built into a wheel, is pyproject.toml an appropriate place to store project-level configuration? Assuming it is, how do we reconcile the issue that as currently designed, the existing standardised data in pyproject.toml is very strongly tied to assumptions that are rooted in the “build a wheel” use case?
Like it or not,
[build-system], defined in PEP 518, is about how to specify what’s needed to execute a build system (i.e., build a wheel).[project], defined in PEP 621, is about storing core metadata in pyproject.toml. And core metadata is defined as being the data that gets stored in a wheel and an installed project.I think it’s very risky to make incremental additions to pyproject.toml in support of “non-wheel” cases without at a minimum having a better understanding of whether we try to re-interpret the existing standardised data for non-wheel cases (which would likely need one or more PEPs to change the existing specification), or treat what’s already standardised as “for wheels and installed projects only” and create new data for non-wheel cases.
In the spirit of making standards from existing practice, I think “re-interpret standardized data” makes the most sense. To me pyproject.toml is a big success story for Python standardization, to the point that it was adopted beyond its intended usage. I don’t think that can be undone at this point, and I don’t think it should be. It’s a stroke of luck that something intended for wheels worked so well for other users.
So expanding/tweaking the [project] spec to accommodate these workflows feels like the right move, with the caveat that disrupting the wheel-building workflow is not an acceptable trade-off.
I think we came to the conclusion earlier in the thread that any tweaking we did would disrupt the wheel-building workflow to an unacceptable level. At least, no-one came up with an idea that lasted long enough to get made into a complete proposal…
I’d be happy to be proved wrong, but we’re not at that point yet.
Personally, I’m not actually a fan of the “one huge config file” approach, and I prefer (for example) separate, focused requirements files. To give an example, it’s much easier to have a template “add sphinx documentation to a project” template if I can just unzip a bunch of files and be ready to go. If I also need to add a new section to pyproject.toml, that’s added complexity and makes it harder to transition from a smaller project to something more complex and “professional”.
I understand I’m an outlier here, but it’s important to remember that “what works for me” might not work for others. There are lots of people who haven’t adopted pyproject.toml enthusiastically, but have felt pushed into doing so and have found it a frustrating and inconvenient experience.
Yeah, it’s possible there isn’t an incremental change that can accomplish this. Certainly not to everyone’s satisfaction, if that could even be achieved.
I actually tend to agree, with the distinction that I like having a top-level file that references all those other files. e.g. using pyproject.toml to refer to requirements.txt and dev-requirements.txt using dynamic metadata. So I’d want to add a doc-requirements.txt file and a line to my pyproject.toml that can read it and use it appropriately, but not more than that.
I guess that idea (and a lot of similar ones) is along the lines “turning pyproject.toml into something more like a Makefile”. Which is certainly not what it was built for, but the ability to add [tool] sections got people[1] moving in that direction.
like myself ↩︎
Personally, I’m not actually a fan of the “one huge config file”
approach, and I prefer (for example) separate, focused
requirements files. To give an example, it’s much easier to have
a template “add sphinx documentation to a project” template if I
can just unzip a bunch of files and be ready to go. If I also need
to add a new section topyproject.toml, that’s added complexity
and makes it harder to transition from a smaller project to
something more complex and “professional”.
Another use case, which covers hundreds of projects I help maintain,
is that you can for example configure your CI system to only run
specific jobs when relevant files change. Being able to do docs
builds when doc/requirements.txt (or anything inside the doc/
directory) changes and not on changes to
tests/functional/requirements.txt is quite useful, whereas if all
requirements lists were packed into pyproject.toml your job
scheduler would need to be able to parse that in order to decide
which jobs should and shouldn’t run for a given commit.
I will note that merely having a pyproject.toml file is sufficient for a project to be something that can be built into a wheel, since setuptools can and will happily proceed without any build configuration.
So, at least mechanically, most directories that contain a pyproject.toml is a pip-installable project.
I think so. With the level of adoption of [tool] I think this has already happened.
By adding other standardized data that isn’t “strongly tied to assumptions” around wheels? 
Seriously, we just use the file for other stuff. It’s to the point now that tools like Black only support pyproject.toml for configuration, regardless of whether you build a wheel or not.
I will say that when pyproject.toml gained popularity among tool providers, I was personally thanked multiple times for helping to tame the number of configuration files necessary for a project.
We have talked about revisiting that bridging default and dropping the implicit support.
FYI for wheel building Hatchling sort of copied what setuptools does as a fallback when not selecting explicit inclusion/exclusion options and tries to guess and doesn’t fail. Multiple issues and my lurking on social media has indicated to me that this is wrong behavior and the next release an error will be raised.
In case it hasn’t been mentioned already (sorry, huge thread), for prior art Poetry has an open PR to add a non-package-mode given how often this feature has been requested: