PEP 665, take 2 -- A file format to list Python dependencies for reproducibility of an application

They are formulations that IMHO are sub-optimal. It’s more than a typo but less than questioning the whole existence. I could enlist them here, but that would be harder to follow and discusses…

2 Likes

The PR is PEP 665: take 2 by brettcannon · Pull Request #2131 · python/peps · GitHub

Is the std-lib’s inability to natively parse Z-suffixed UTC iso-format date-time strings the only reason to use the full time-zone offset in the created-at field? I feel if UTC is the only allowed time-zone for that format, then the PEP should follow the conventions recommended by database guides and use Z instead.

It’s pretty easy to write code which produces the same behaviour to support this

# parse
dt_str = created_at.replace("Z", "+00:00")
dt = datetime.fromisoformat(dt_str)

# produce
dt_utc = dt.astimezone(timezone.utc)
dt_str = dt_utc.isoformat("T")
created_at = dt_str.replace("+00:00", "Z")

Edit: as noted below, TOML handles the date-time format

1 Like

That and I think it’s more unambiguous than Z if you don’t know about it ahead of time. For instance, if some beginner opened up a .pylock.toml file and saw a datetime ending in Z would they know what it meant? And how easy is it to find out what Z means?

Why do you think we should specifically follow what databases suggest in this instance? Does ISO 8601 have something specific to say on the matter? RFC 3339 is silent on the matter.

Basically my question comes down to is using Z worth forcing potentially every consumer of this file to have to have that extra replace() call to parse the string?

4 Likes

For details as to what it might take to get Z into datetime, see Issue 35829: datetime: parse "Z" timezone suffix in fromisoformat() - Python tracker .

I have reached out to various projects to ask them if they would support this PEP:

If anyone knows of a project that you feel I missed and could/should ask for support of this PEP, please let me know and I will reach out.

I don’t know if tox/nox would be relevant in this context…

Depends on the level of optimizations they want to provide. I cannot speak for nox, but as far as tox goes we already implemented a pip requirements.txt parser, so we likely would be involved in this to.

1 Like

To me the “decision” to not use Z is more like “meh, who cares”. If anything, since the timestamp can only be in UTC, we don’t need the timezone part in the first place, and the addition is simply for explicitness, so the goal should be to have something as simple as possible, and +00:00 is that.

2 Likes

There should be generators and consumers, among consumers I think pipx is related?

It’s the business of the TOML parser, and TOML v1.0.0 spec allows Z-suffixed format. So IMHO it doesn’t make sense to restrict the datetime in created-at field, both +00:00 and Z are parsed as the same datetime object in Python. The strings are to be consumed by the TOML parser not datetime.isoformat and it is not necessary to do validation against the raw TOML string.

1 Like

17 posts were split to a new topic: Supporting sdists and source trees in PEP 665

Since we’re talking about reproducibility, what’s the use case for the non-reproducible created_at timestamp? The PEP doesn’t address it.

While it won’t affect the reproducibility of an application that a .pylock.toml file describes, it will affect the bit-for-bit reproducibility of anything that may contain a .pylock.toml file.

2 Likes

Thank you for working on this!

As a distro packager, the PEP doesn’t affect my use case directly, but I’d be happy if it briefly mentioned that it doesn’t:

Development is one use case of many. Please consider expanding it into something like this (at the end of Motivation?):

Lock files are not meant as a replacement for specifying flexible dependencies: ranges and constraints as described in PEP 440.
Flexible dependencies are useful for development or as input for generating lock files. Also, they can be useful for integrating applications into larger environments which aren’t necessarily Python-centric, such as Linux distros. Such environments must of course handle integration, integration testing, and reproducibility on their own.


The PEP’s title is a mouthful, but I’ll still suggest making it just a bit longer. Could it be “PEP 665 – Lock file: A file format to list Python dependencies for reproducibility of an application”?

I’m terrible at remembering numbers, so I’d like to encourage people to write “PEP 665: Lock file” or “the lock file PEP” rather than “PEP 665”.


fromisoformat accepts other separators than T. Please either allow a space as well (for readability), or mention that a fixed format also benefits other (possibly non-Python) tools.
(This point may be moot if TOML handles the datetime.)


Typo: unambiguously


I first read this as “project” being the thing managed by Poetry. Consider rewording this as: “even if Poetry chooses not to adopt this PEP as its primary lock file format.”

It is a conscious effort (well, my conscious insistence) to avoid Lock File in both the title and abstract because I observed some people have a preconceived perception to this term (that’s different from the definition used by PEP 665), and tend to want to force what they think the term means onto the PEP. I want to make the title and abstract as unambiguous as possible by only describing the idea with better defined terms instead.

Would “PEP 665 – pylock.toml: A file format …” work?

1 Like

Allowing for sdists and source trees is an open issue (and there are other open issues to discuss as well).

But the trick is how do you pin build dependencies? All you can actually pin is the builder, but not what it may need to do the build (i.e. you can pin what is specified using PEP 517, but that isn’t the complete story). It also then makes it so the installer has to have complete resolution logic as you can’t know ahead of time what the source tree will require to be installed once built, and this is very likely true for an sdist as well (unless they fill out their PKG-INFO, use core metadata 2.2, and don’t have Requires-Dist marked as dynamic). As of right now the installer’s logic is much simpler on purpose so as to not require a complete SAT solver.

That was an explicit ask in the last round to make it easy to know when the lock file may be out of date based on someone’s personal experience with requirements files.

Good point. Unfortunately there’s no way to resolve that conflict, so we will have to choose which use case is more important.

Sure, happy to figure out a way to pull it in.

That’s the point that @frostming made and it seems like a good point, so I will change the PEP to say the date time must be in UTC but otherwise leave the format requirements to the TOML spec.

1 Like

If you’re doing reproducible, you generally override all the timestamps on everything. So you’d override this one too - tools (that want to be reproducible) should use SOURCE_DATE_EPOCH or whatever their input is.

Having the field is important. When everyone involved consents to lying about it, then it’s okay to lie about it.

2 Likes

I just pushed PEP 665: address feedback (#2134) · python/peps@4b865b9 · GitHub to address the following:

  • Wording/clarifications from @bernatgabor , @encukou .
  • Drop formatting restrictions for created-at (still must be UTC); thanks to @EpicWink , @frostming , @encukou .
  • Listed pip and PDM as supportive of the PEP (and implicitly pip-tools since the current ideas floating on the pip issue tracker would somewhat obsolete pip-tools :sweat_smile:).

Should already be rendered and available at PEP 665 – A file format to list Python dependencies for reproducibility of an application | peps.python.org and updated above.

2 Likes

In my experience, support for relative paths in file URLs is inconsistent. Some tools seem to support them, other tools don’t. For example, is file://c:/some/path relative or absolute? On Unix, it’s “obviously” relative, and on Windows it’s “obviously” absolute. I wouldn’t trust how a tool would interpret this without experimenting, so IMO it’s risky to assume that a locket and an installer will interpret the URL the same way.