PEP 751: one last time

[mitsuhiko (Armin Ronacher)]

I think if the syntax is extended for markers then it should be a goal to reserve this in the marker PEP too so that a singular parser can be used for both of them. It would be disappointing if this ends up with two actual, incompatible grammars.

[pf_moore (Paul Moore)]

Agreed. And furthermore, as a pip maintainer, I’d expect to handle the marker expressions using packaging, so someone’s going to have to work out an ergonomic way of supporting the two different syntaxes before we get support for lockfiles in pip…

To be honest, I hadn’t realised the proposal was to have a subtly different marker expression syntax in lockfiles than we have in other places (I’d assumed the proposal was to support the new syntax everywhere). I’m not sure I’m comfortable with that.

[brettcannon]

@mitsuhiko

I think if the syntax is extended for markers then it should be a goal to reserve this in the marker PEP too so that a singular parser can be used for both of them.

We actually don’t update PEPs once they are marked as final; they are considered a historical document at that point. But I would update Dependency specifiers - Python Packaging User Guide accordingly as it holds the official grammar for the marker expression syntax.

@pf_moore

I’d expect to handle the marker expressions using packaging, so someone’s going to have to work out an ergonomic way of supporting the two different syntaxes before we get support for lockfiles in pip…

There is only a single syntax as proposed by the PEP.

@pf_moore

I hadn’t realised the proposal was to have a subtly different marker expression syntax in lockfiles than we have in other places (I’d assumed the proposal was to support the new syntax everywhere).

There’s a misunderstanding here as there isn’t different syntax based on where it’s used, any more than what the spec already does and supports.

First, the grammar change proposed by the PEP is small and does not split the grammar into two:

diff --git a/source/specifications/dependency-specifiers.rst b/source/specifications/dependency-specifiers.rst
index 06897da2..c9ab247f 100644
--- a/source/specifications/dependency-specifiers.rst
+++ b/source/specifications/dependency-specifiers.rst
@@ -87,7 +87,7 @@ environments::
                      'platform_system' | 'platform_version' |
                      'platform_machine' | 'platform_python_implementation' |
                      'implementation_name' | 'implementation_version' |
-                     'extra' # ONLY when defined by a containing layer
+                     'extra' | 'extras' | 'dependency_groups' # ONLY when defined by a containing layer
                      )
     marker_var    = wsp* (env_var | python_str)
     marker_expr   = marker_var marker_op marker_var

So there’s still a single syntax just like there is now. I will add that diff to the PEP when I update it again to make this very clear (assuming this all even makes it to Monday when I said I would stop taking feedback).

My guess is the misunderstanding stems from me saying that extras and dependency_groups should only be used in lock files and that they shouldn’t support all operations. But both of those constraints have precedence in the current spec.

Consider extra itself when thinking about the restriction of where to use the new markers. extra is part of the official grammar and yet it isn’t really meaningful in a requirements.txt file since it makes no sense to say, e.g. numpy; extra == 'extra-a' as requirements files don’t have extras. I’m saying the same thing with extras and dependency_groups when it comes to only using them in lock files, so I don’t think I’m being unreasonable nor am I introducing some new concept about where the markers work (I can water down the language in the PEP to make this explicit).

Now let’s consider operations on extras and dependency_groups. Let’s look at what the spec says about operations:

Comparisons in marker expressions are typed by the comparison operator. The <marker_op> operators that are not in <version_cmp> perform the same as they do for strings in Python. The <version_cmp> operators use the version comparison rules of the Version specifier specification when those are defined (that is when both sides have a valid version specifier). If there is no defined behaviour of this specification and the operator exists in Python, then the operator falls back to the Python behaviour. Otherwise an error should be raised. e.g. the following will result in errors:

“dog” ~= “fred”
python_version ~= “surprise”

Let’s use python_version and sys_platform as two example markers. The former falls under the <version_cmp> part of the spec while sys_platform falls under the string part. So, how does the grammar separate these things? The operations are in the grammar as:

version_cmp   = wsp* <'<=' | '<' | '!=' | '==' | '>=' | '>' | '~=' | '==='>
version       = wsp* <( letterOrDigit | '-' | '_' | '.' | '*' | '+' | '!' )+>
version_one   = version_cmp:op version:v wsp* -> (op, v)
version_many  = version_one:v1 (',' version_one)*:v2 (',' wsp*)? -> [v1] + v2
versionspec   = ('(' version_many:v ')' ->v) | version_many
marker_op     = version_cmp | (wsp* 'in') | (wsp* 'not' wsp+ 'in')

The markers are covered by:

env_var       = ('python_version' | 'python_full_version' |
                 'os_name' | 'sys_platform' | 'platform_release' |
                 'platform_system' | 'platform_version' |
                 'platform_machine' | 'platform_python_implementation' |
                 'implementation_name' | 'implementation_version' |
                 'extra' # ONLY when defined by a containing layer
                 ):varname -> lookup(varname)

So, both operations and markers are in no way separate in the grammar even when the spec says the valid operators for e.g. python_version and sys_platform are different (for instance, as the spec says, strings do not work with ~=, and yet the grammar doesn’t prevent it). So saying certain operations only work with extras and dependency_groups does not require a separate grammar from an operator perspective (although I’m sure you could write a parser that was that strict if you didn’t follow the official grammar exactly).

Probably my mistake in the PEP was trying to make the changes to the marker expression syntax as small as possible without being more concrete. So, what I will do is change the PEP to say that extras and dependency_groups are sets and not be generic about them being containers. I will also change:

Comparisons in marker expressions are typed by the comparison operator. The <marker_op> operators that are not in <version_cmp> perform the same as they do for strings in Python.

to say:

Comparisons in marker expressions are typed by the comparison operator and the type for the marker. The <marker_op> operators that are not in <version_cmp> perform the same as they do for strings or sets in Python based on the whether the marker is a string or set itself.

The spec can then state which of the markers can hold a version, string, or set which then determines which operators are expected to work with each marker and how those operators are expected to work. But do note I am not updating the spec to allow for specifying set literals like you can for versions and strings, so that limits which operators will actually work when evaluating markers simply based on the arguments to the operators.

Does that alleviate your concerns over the syntax, @pf_moore ?

[pf_moore (Paul Moore)]

@brettcannon

Does that alleviate your concerns over the syntax, @pf_moore ?

Yes, thanks.

[ncoghlan (Alyssa Coghlan)]

@brettcannon

My guess is the misunderstanding stems from me saying that extras and dependency_groups should only be used in lock files and that they shouldn’t support all operations. But both of those constraints have precedence in the current spec.

I think the other thing to clarify is that it’s fine for metadata parsers to accept the updated grammar anywhere (so folks don’t need separate parsers, they can just improve the existing ones), but it’s not fine to use the nicer "name" in extras syntax in the current versions of the pyproject.toml and package metadata formats.

Those files still have to rely on the legacy extra == "name" format for now, until there are version bumps for those formats updating them to the improved marker grammar for extras (and there’s no ETA on when those version bumps will happen).

[radoering (Randy Döring)]

@kapinga

The only “trick” is that “main” was created by the locker (poetry) rather than be specified in the pyproject.toml file.

I really haven’t even considered that it is allowed to add a dependency group that is not defined in the pyproject.toml - even though that is exactly what we have already done in our custom lock file format. Thanks for the clarification.

@ncoghlan

I think this is the key point as far as the “everything is optional” tool specific use case goes: it is supported in the format (by the tool defining a “main” dependency group in pylock.toml rather than marking the default dependencies from pyproject.toml as unconditional), but actually using that capability means that exporting a separate lock file will still be necessary when defining specific group selections for basic installers to consume (no major process change from the status quo, just more consistent file formats)

This is also an interesting insight for me. Previously, I thought we either export a pylock.toml or replace our custom lock file format with pylock.toml - not both. Now, I realize that we still need the export even if we replace our internal lock file.

@brettcannon

The other option is a Boolean flag on [[packages]].

If I do not miss anything, a Boolean flag will not be sufficient if a package is not unconditionally required.

@brettcannon

I realized this morning a third option is introducing a default-group key that takes a string and represents a synthetic dependency group

That might be nice for interoperability between tools. Otherwise, other tools cannot know the name of this synthetic group.

@brettcannon

(I would probably leave the synthetic name out of dependency-groups, but that’s not a strong opinion).

No strong feelings from my side.

@brettcannon

packages.directory.editable isn’t a “SHOULD” in terms of following it, but it could be. I honestly put the flag in there because I think @charliermarsh and @radoering asked for it; I have no great attachment to it as I think whether something is editable can be a decision at install time.

I would also consider it as “by default do an editable install” but the installer is allowed to ignore it if the user requested “no editable installs”. If the general opinion is against this setting, I assume it would be possible to put it in the tools section and still be compliant with the PEP. (However, I am still in favor of keeping it.)

[kapinga (Jon Harding)]

@radoering

I would also consider it as “by default do an editable install” but the installer is allowed to ignore it if the user requested “no editable installs”. If the general opinion is against this setting, I assume it would be possible to put it in the tools section and still be compliant with the PEP. (However, I am still in favor of keeping it.)

As a user, I agree that I’d want some ability for the locker to signal that a particular package should be an editable install (even if the end user should be able to override that). This is currently supported in requirements.txt, and if it’s not included in pylock.toml, then the alternative is to exclude those packages entirely (as Paul and I were contemplating above) or instruct users to pass a (possibly long and/or unintuitive) set of (not-yet-developed) flags to pip to signal which packages within the provided lock file should be editable installs.

That said, unless theres a security reason why you’d want to require an editable install (which seems unlikely), installers should be allowed to ignore the “editable” signal if the end user requests that everything be installed normally.

[groodt (Greg Roodt)]

No strong feelings on the topic of “editable”.

I know it’s a useful feature and there is already widespread usage.

From a design POV though, it feels like pylock.toml primarily describes precisely “what” to install. Not “how” to install it.

So from that point of view having the information in the “tool” section sounds best to me. Or even leaving it to installers runtime arguments.

As mentioned, just sharing my perspective in case it’s useful. I’d prefer the PEP progress rather than debating this topic.

[brettcannon]

PEP 751: address feedback by brettcannon · Pull Request #4306 · python/peps · GitHub has changes based on the latest feedback:

• Make packages.*.name optional (@charliermarsh )
• packages.directory.editable can be ignored at install-time (@konstin)
• Clarify the marker syntax changes
• Introduce default-group (@radoering )
• Clarify how this PEP does not fully replace requirements files (@konstin )
• Clarify how different installers can be used to install from a lock file, but different lockers could lead to different outcomes (@konstin )

I’ll give people until Wednesday to make any comments about the changes.

[charliermarsh (Charlie Marsh)]

Hopefully not a large ask: could we use default-groups and accept a list? We have a setting for that in uv (and the default is ["dev"]). Otherwise, no further comments.

[charliermarsh (Charlie Marsh)]

Per the changes to the marker specification: peps/peps/pep-0751.rst at f663eaacafafc8348732e8093bf78988c8d57bf4 · python/peps · GitHub

What should tools do if they encounter extras or dependency-groups outside of that context? E.g., what if we see that in a Requires-Dist marker?

[brettcannon]

@charliermarsh

Hopefully not a large ask: could we use default-groups and accept a list?

I’m fine with that. I also removed the requirement that dependency-groups must have values for default-groups to be allowed.

@charliermarsh

What should tools do if they encounter extras or dependency-groups outside of that context? E.g., what if we see that in a Requires-Dist marker?

I’ve updated the PEP to say it will be just like extra; an error like there’s no variable defined when used outside of lock files.

Both changes are covered by PEP 751: Make `default-group` be `default-groups` by brettcannon · Pull Request #4308 · python/peps · GitHub .

[brettcannon]

@brettcannon

@charliermarsh

Hopefully not a large ask: could we use default-groups and accept a list?

I’m fine with that. I also removed the requirement that dependency-groups must have values for default-groups to be allowed.

This isn’t enough to change things, but making it a list means installers that are not the locker won’t know what to do with any individual default group. Since uv also makes a “dev” group that I assume will also end up in dependemcy-groups, it does at least give tools a chance to offer to exclude the “dev” group as it’s name is “public” and thus known.

[brettcannon]

@brettcannon

I’ll give people until Wednesday to make any comments about the changes.

It’s Wednesday and I addressed the comments that came in.

As such, with some excitement , trepidation , and relief , I’m officially asking for pronouncement, @pf_moore !

[pf_moore (Paul Moore)]

Eek! That means I need to review the discussion here against the PEP

Thanks for all the work you’ve put into this. And to everyone who has contributed to the discussions. It’s going to take me a while to review, so bear with me…