PEP 621: how to specify dependencies?

[ssbarnea (Sorin Sbarnea)]

I subscribe to @ofek remarks. AFAIK, putting dependencies, including extras inside setup.cfg seems to be convenient and effective. I would support making this the recommended way instead of creating yet another place to stick dependencies.

Once adoption is 90% we will have more date to make an informed decision one that would also allow tool creators to prepare for.

[dstufft (Donald Stufft)]

I would say too that this PEP is not really mandatory. Due to the fact we’re not in greenfield development here, we’re likely going to be effectively supporting installing/building projects without that table ~forever. So if a build tool feels strongly enough against one of the decisions in the PEP, there is nothing stopping them from choosing not to support it. Of course that weakens the value of the PEP to some degree (but I think it’s a natural consequence of our inability to come to consensus), and that tool might get other pressures to conform, but as far as the PEP is concerned, it fundamentally can’t be mandatory.

[brettcannon]

@pradyunsg

I’m thinking of 2 separate PEPs (one w/ TOML tables, one w/ PEP 508) with PEP 621 deferring to whichever gets accepted.

That’s how I would suggest doing it, but I am definitely not writing either of those other PEPs.

@pf_moore

If you want a strong guide as to what I would find most persuasive, I’ll say this. Stop worrying about process, and about how to present the choice, and concentrate for now on getting an “exploded format” proposal that’s specific, documented, and agreeable to everyone who’s arguing for “exploded format”. Present a unified alternative. Stop worrying about persuading people who prefer PEP 508, and focus on clarifying what alternative you are suggesting.

Based on that, I think the idea of resolving this discussion with a final answer instead of PEP 621 becoming a recommendation/guide is very much going to come down to who is willing to volunteer to write a PEP 508-based PEP and an exploded table PEP just on the topic of how to write them down in a TOML file.

How about this: if people can get themselves organized enough to have a draft proposal being actively discussed as a topic here by Tuesday, September 8th then we can consider this idea of keeping PEP 621 a standard alive (that’s in 3 weeks). So this would be mostly a forcing function for people to organize themselves (on a separate topic!) and show dedication by showing forward progress on these other PEPs. But if people can’t get themselves organized that quickly then I would consider this dead due to lack of motivation and PEP 621 would no longer on the standards track and instead will either be withdrawn or touched up to be an informational PEP.

Does that work for people (and specifically @pf_moore)?

@pf_moore

This is why I just say the PEP is “weak” - it doesn’t need to act as a sales push for the new format, but it could do better at explaining the reasons to people in that sort of situation.

Fair enough, although the motivation section has already been rewritten I think twice now based on feedback from people so I don’t know how to improve it without simply culling less motivating reasons and going all-in on the “you only need to document it once” argument as @pganssle pointed out (which I’m fine with, but I don’t want to do the work if that won’t be beneficial).

@pganssle

would be to recast PEP 621 as an informational PEP describing the design decisions that went into it and how you can apply them for your backend.

True, it could just change its type instead of being withdrawn if it was reworked to phrase its motivation that way. I’m fine with this if the idea of separate PEPs on the topics are not enough to reach a conclusion. And even if dueling PEPs are written and this is still the way we go then at least we have written out suggestions on how people handle either format for dependencies.

[ofek (Ofek Lev)]

I’d be willing to do the PEP 508 version. What should be included exactly?

[dstufft (Donald Stufft)]

@brettcannon

How about this: if people can get themselves organized enough to have a draft proposal being actively discussed as a topic here by Tuesday, September 8th then we can consider this idea of keeping PEP 621 a standard alive (that’s in 3 weeks). So this would be mostly a forcing function for people to organize themselves (on a separate topic!) and show dedication by showing forward progress on these other PEPs. But if people can’t get themselves organized that quickly then I would consider this dead due to lack of motivation and PEP 621 would no longer on the standards track and instead will either be withdrawn or touched up to be an informational PEP.

I’d say that if nobody steps up to write one of the PEPs, but someone does step up to write the other one, then that shouldn’t stall this effort, but I think it’s fine to set some dead line (with flexibility, if someone says they’ll do it, but need until a different date I think that’s fine).

[pradyunsg (Pradyun Gedam)]

@ofek

What should be included exactly?

What the “RHS” values in dependencies = ... (and hence, optional-dependencies = ...) should be.

[ofek (Ofek Lev)]

I made an initial draft PEP 631 - Dependency specification in pyproject.toml (I’ve never written a PEP btw)

[brettcannon]

Discussion of the PEP 508 PEP can be found at PEP 631 - Dependency specification in pyproject.toml.

[brettcannon]

@brettcannon

if people can get themselves organized enough to have a draft proposal being actively discussed as a topic here by Tuesday, September 8th then we can consider this idea of keeping PEP 621 a standard alive (that’s in 3 weeks). So this would be mostly a forcing function for people to organize themselves (on a separate topic!) and show dedication by showing forward progress on these other PEPs.

As a reminder, there is less than two weeks left to start discussions around some draft PEP. The PEP 508 camp have their draft and started discussions; I have not seen anything from the exploded table camp up as of today.

[brettcannon]

We are now one week out from needing a discussion started for an exploded table format in TOML if we are going to consider it.

/cc @sdispater, @finswimmer, @EpicWink as they seemed like proponents for the exploded table approach (sorry if I’m wrong about that assessment).

[EpicWink (Laurie O)]

I’m ambivalent but not too invested. I’ll write up a basic PEP now

Edit: done. Discuss here

[abn (Arun Babu Neelicattu)]

Thought I would share this here as well. A few weeks ago, mostly to cater to my own curiosity, I created a (totally unscientific) survey to collect some feedback regarding the dependency specification options and share it via various means hoping it will reach a wider audience than this thread. Ended up getting some responses. Hopefully it has some useful insights for someone (most likely nothing new to the folks on this thread).

• Survey Form
• Survey Responses

[steve.dower]

Thanks for doing this!

Can you share the examples from the second last question? They don’t appear in the results, and I don’t really want (all of us) to take the survey just to look at what the questions were.

My takeaway, unsurprisingly: the strongest responses were “we don’t need yet another way to do it”, nobody in the verbatims considered compatibility with existing formats (and a few people seemed to think that PEP 508 was “new”), and complaints about having to parse PEP 508 style didn’t acknowledge that they’d still need to parse the version constraint in all the likely TOML formats.

So despite the percentages seeming to suggest both are equally acceptable to end-users (2/3 of which maintain a package on PyPI), I’m not feeling swayed by the data.

[abn (Arun Babu Neelicattu)]

@steve.dower

My takeaway, unsurprisingly: the strongest responses were “we don’t need yet another way to do it”,

I’d be careful generalising it like that. Especially since only 22 descriptive responses exist for 113 responses (at the time of writing this). And out of that 22, only less than 1/4 even indicate anything to that effect. And it should not be discounted that around 80% chose the TOML version over the PEP 508 version. And as you mentioned 2/3 of these are people who maintain packages on PyPI. Unsurprisingly when considering more that “simple” specification of dependency, this percentage bumps to over 90%.

Of course all this needs to be taken with a grain of salt because 113 responses does not cover the ecosystem. This, if anything highlights that the current way needs to change.

Personal Opinion: While, I definitely agree that having multiple ways can be troublesome, this needs to be considered as a transition that helps forge a better future for the language’s packaging ecosystem. Moving into a structured format has its advantages.

As for the examples I used. Here are the direct links.

• PEP 508
• TOML

[ssbarnea (Sorin Sbarnea)]

This survey is welcomed but I have to say it has biased options and forces users to make incorrect picks. I find “I am familar with it, but never used it in any meaningful way.” more of a leading question because is the only thing you can pick if you fit into category “I used it because there was not other option (or used it and hated it)”.

Never heard of it / Never had to use it.
I am familar with it, but never used it in any meaningful way.
I make use of it regulary.

The way is phrased it it forces user hand and also produces misleading results. Even if the user figured it out that the middle is most appropriate it produces misleading results, mainly because there is no way to distinguish between users that are somehow familiar with that option but that do not like/support it. That is because “never used it in any meaningful way” is leads towards “that user didn’t had any clue about the subject”.

Sadly filling the form reminded me a little bit of Piers Morgan. Wouldn’t a simple “I am relatively familiar with” be less leading?

[pf_moore (Paul Moore)]

Survey design is hard. I really appreciate the fact that @abn did this at all. We need to be careful how we use the results, certainly, but given how little hard data we have in any of this discussion, I think that even with any flaws it has, this is really useful information to add to the discussion.

I’d be perfectly happy to see other surveys, too. With extra data we can smooth over any biases that individual surveys might have.

[steve.dower]

@abn

@steve.dower

My takeaway, unsurprisingly: the strongest responses were “we don’t need yet another way to do it”,

I’d be careful generalising it like that. Especially since only 22 descriptive responses exist for 113 responses (at the time of writing this). And out of that 22, only less than 1/4 even indicate anything to that effect.

Yep, fair response. Reading verbatims is more of an art form than a science though - numbers don’t really mean anything because they’re essentially anecdotes.

@abn

As for the examples I used. Here are the direct links.

Thanks! These are all good examples to have used. My only concern (which I’ll withdraw if these aren’t how they were actually displayed in the survey) is that the colour contrast is terrible and anyone who chose the better looking option is almost certainly being influenced by the PEP 508 examples being almost unreadable. The TOML examples I feel are also unfair, because while the colours make the structure more obvious, it makes it harder to read the actual information.

It might be interesting to convert these into a timed comprehension test, but then again, dependency specifications are generally write once, update rarely, read occasionally, so if anyone has the time and motivation to run this kind of study I’ll happily suggest some other interfaces that would be more beneficial to do instead besides this one

[ofek (Ofek Lev)]

This is a real-world example port of what docker-compose defines, which both PEPs show.

PEP 508

[project]
dependencies = [
  'cached-property >= 1.2.0, < 2',
  'distro >= 1.5.0, < 2',
  'docker[ssh] >= 4.2.2, < 5',
  'dockerpty >= 0.4.1, < 1',
  'docopt >= 0.6.1, < 1',
  'jsonschema >= 2.5.1, < 4',
  'PyYAML >= 3.10, < 6',
  'python-dotenv >= 0.13.0, < 1',
  'requests >= 2.20.0, < 3',
  'texttable >= 0.9.0, < 2',
  'websocket-client >= 0.32.0, < 1',

  # Conditional
  'backports.shutil_get_terminal_size == 1.0.0; python_version < "3.3"',
  'backports.ssl_match_hostname >= 3.5, < 4; python_version < "3.5"',
  'colorama >= 0.4, < 1; sys_platform == "win32"',
  'enum34 >= 1.0.4, < 2; python_version < "3.4"',
  'ipaddress >= 1.0.16, < 2; python_version < "3.3"',
  'subprocess32 >= 3.5.4, < 4; python_version < "3.2"',
]

[project.optional-dependencies]
socks = [ 'PySocks >= 1.5.6, != 1.5.7, < 2' ]
tests = [
  'ddt >= 1.2.2, < 2',
  'pytest < 6',
  'mock >= 1.0.1, < 4; python_version < "3.4"',
]

Exploded table

[project.dependencies]
cached-property = { version = '>= 1.2.0, < 2' }
distro = { version = '>= 1.2.0, < 2' }
docker = { extras = [ 'ssh' ], version = '>= 4.2.2, < 5' }
docopt.version = '>= 0.6.1, < 1'
jsonschema.version = '>= 2.5.1, < 4'
PyYAML.version = '>= 3.10, < 6'
python-dotenv = { version = '>= 0.13.0, < 1' }
requests = { version = '>= 2.20.0, < 3' }

# Conditional
'backports.shutil_get_terminal_size' = { version = '== 1.0.0', markers = 'python_version < "3.3"' }
colorama.version = '>= 0.4, < 1'
colorama.markers = 'sys_platform == "win32"'

[project.dependencies.texttable]
version = '>= 0.9.0, < 2'

[project.dependencies.websocket-client]
version = '>= 0.32.0, < 1'

[project.dependencies.'backports.ssl_match_hostname']
version = '>= 3.5, < 4'
markers = 'python_version < "3.5"'

[project.dependencies.colorama]
version = '>= 0.4, < 1'
markers = 'sys_platform == "win32"'

[project.dependencies.enum34]
version = '>= 1.0.4, < 2'
markers = 'python_version < "3.4"'

[project.dependencies.ipaddress]
version = '>= 1.0.16, < 2'
markers = 'python_version < "3.3"'

[project.dependencies.subprocess32]
version = '>= 3.5.4, < 4'
markers = 'python_version < "3.2"'

[project.optional-dependencies]
socks = { PySocks = { version = '>= 1.5.6, != 1.5.7, < 2' ] } }

[project.optional-dependencies.tests]
ddt = { version = '>= 1.2.2, < 2' }
pytest = { version = '< 6' ] }
mock = { version = '>= 1.0.1, < 4' }

[project.optional-dependencies.tests.mock]
markers = 'python_version < "3.4"'

[pf_moore (Paul Moore)]

Thanks for posting a real-life comparison. It’s a useful checkpoint on what we’re debating about.

I presume it won’t come as much surprise to anyone here, but personally I find the PEP 508 version very easy to read, and the TOML one really difficult - there’s too much clutter from the extra syntax.

Of course, readability is only one aspect. How hard is it to write something like this is also a question. Personally, I don’t find PEP 508 syntax up to the point of 'docker[ssh] >= 4.2.2, < 5' at all difficult to write - we see it all the time in pip command lines, in requirements.txt files, etc, none of which are going away. For more complex stuff like markers or extras, I’d have to look up the syntax. But for the TOML version, I would too (and it’s quite possible I’d have to look up the TOML version of simpler cases as well, where I know PEP 508).

To put it another way, simple cases in PEP 508 syntax have the significant advantage of “looks like a requirements.txt line”. And unless we’re going to deprecate PEP 508 totally, that’s a pretty significant difference between the two options.

[ofek (Ofek Lev)]

Definitely anecdotal and not as well-formed as @abn’s survey (great job again) but here’s a minor survey of Datadog’s internal Python slack channel (I’ll try to keep updating the image):

Here’s a particularly poignant feedback:

'backports.shutil_get_terminal_size' = { version = '== 1.0.0', markers = 'python_version < "3.3"' } should be enough to discard the 2nd solution IMO