PEP 694 -- PyPI upload API 2.0 (Round 2)

Packaging Standards
1

I’m very happy to share round 2 of PEP 694 – PyPI upload API 2.0. This latest round incorporates all the changes discussed in the first round of DPO discussion, plus changes from @EWDurbin based on our discussion at PyCon 2025. Please use this thread for any further feedback.

Given that @dstufft would normally be the PEP Delegate for PEPs affecting PyPI, and that he along with Ee are co-authors, we will be putting this PEP on the Steering Council’s agenda for pronouncement. Should they wish to delegate, @dustin has agreed to serve in that role.

10 Likes
2

Thanks for sharing this @barry! I’m really excited to see the upload 2.0 designs move along.

Some scatterbrained thoughts from reading the current proposal:

  • The expires-at key is currently described as an ISO 8601 formatted timestamp. What do you think about constraining this a bit further, and requiring that it (1) be a UTC timestamp with the Zulu marker, and (2) forbidding fractional seconds (i.e. whole seconds only)? IME (in other contexts) these are unnecessary sources of malleability, especially since these are server-originated times that should only ever be UTC anyways.
  • This is pedantic, but perhaps the filename and version fields should specify that they correspond to their relevant PEPs/living standards? For filename I think that would be the Source distribution file name and Binary distribution file name convention and for version I think it’s Version specifiers. I think this is arguably obvious from context, but maybe being explicit will help future implementers.
  • Two thoughts about the nonce/session design:

    • The current gentoken implementation lacks domain separation between the fields. I think this could result in some niche cases where two users end up with the same session token. Specifically, if two users both don’t include a nonce and have a (name, version) pairs that compose into the same “domain,” then they’ll end up with the same session key.

      This is pretty easy to contrive: for example, (foo, 11) (foo version v11) and (foo1, 1) (foo1, version v1) both compose into foo11 for the purpose of the session key computation. The “classic” way to solve this is to encode each field such that it’s implicitly domain-separated, e.g. length-prefix each component.

      Another option (which is probably sound but harder to formally assert) is adding delimiters between the fields that can’t occur in each field, e.g. \t or ASCII SOH.

    • More generally, I’m curious if anybody would object to making the nonce fully required – I think the above domain separation should still be addressed, but having the nonce would have effectively prevented it as well.

      I completely understand the value of being able to disclose the staging URL (I want that feature!), but I think maybe we can preserve that property while reducing the flexibility of the session creation itself: uploading clients are in full control of the session URL regardless of the nonce state, so the UX of a --publish-staging (or similar) flag isn’t contingent on the upload API itself allowing the nonce to be unset.

      I think this has two virtuous effects: the first is that it’s a defense in depth against the scenario above, and the second is that it eliminates a source of user error/nonfamiliarity (i.e. users needing to understand what a nonce is to get their intended behavior, versus the self-describing behavior of “you’ve disclosed the staging URL, regardless of how the machinery did that internally.”

3

Thanks for thorough review and comments @woodruffw ! I really apprecate your keen eye, especially on security.

IIRC @EWDurbin and I talk about the UTC/Zulu detail, and agree with it, but I wasn’t explicit in the last PR[1]. I’d put together a PR to update this.

Yes, I think that also makes sense. I should also add that the project name in the session creation request must conform to the standard name format and will be normalized.

Great observation, thanks for the callout!

So something like:

    def gentoken(name: bytes, version: bytes, nonce: bytes = b''):
        h = sha256()
        h.update(f'{len(name)}'.encode('ascii') + name)
        h.update(f'{len(version)}'.encode('ascii') + version)
        h.update(nonce)
        return h.hexdigest()

?

If we required it, would we have to disallow explicitly setting nonce to b''?

We may still have to describe the nonce and its purpose though, although I could imagine upload tools doing the hefty lifting behind something like a --secret flag. If given, the upload tool could generate its own nonce internally without the user really having to understand the gory details.

  1. the given example is compliant ↩︎

1 Like
4

Maybe I’m missing something, but I don’t see much of a benefit in constraining this? Most of the time, this expires-at key will be read by some client software which (hopefully![1]) uses an ISO 8601-compliant datetime parser; so these additional constraints would actually be more work to implement. And in cases where this key will be read by a human directly (or if it’s just passed through by a simple client), then fractional seconds are a bit of noise (though fairly easy to ignore); but I could imagine scenarios where a non-UTC timestamp would be desirable. (E.g., for a company-internal package index, where the local time zone of the server and all users is the same.)

On the other hand, the additional complexity of specifying a custom subset of ISO 8601 seems like a noticeable downside to me. It makes it ever so slightly harder to read the PEP, a bit more work to implement, and I expect at least some client implementations will ignore these constraints anyway. (Which is probably fine, but I’d rather not rely on Postel’s law more than I absolutely need to.)

  1. I was taught that cryptography and datetimes are the two areas where you must always use a library written by domain experts and never write your own solution. ↩︎

1 Like
5

I think it’s okay to be constraining here, since the expires-at keys are send by the index servers, so it’s the “be conservative in what you send” side of the equation.

Even in this case, it’s always better to use timezone-aware timestamps for clarity, and if you do that, you might as well just send UTC timestamps. They’re always going to be unambiguous, easy to parse in any language, and convertible to local timezones.

Aside: I’m now remembering the bit that @EWDurbin and I talked about: using a timestamp and an extend-until key in lieu of the extend-for key in request bodies. We decided against that, opting for extend-for and an integer number of seconds.

I was taught that cryptography and datetimes are the two areas where you must always use a library written by domain experts and never write your own solution.

Oh for sure. I recall some very entertaining talks by @pganssle on the topic at previous Pycons.

4 Likes
6

I can see the practical value in that constraint, yes. (Sorry, the language in my earlier post was a bit too broad there.)

… however, this I disagree with. Timestamps using non-UTC time zones are just as unambiguous, easy to parse and convertible; so for a software client, it makes absolutely no difference.

However, when this key is read by a human directly, having it in the local time zone can be much nicer in some scenarios. Using the example with the internal packaging server again: When that server sends 2025-08-08T01:23:45+02:00I can just skip over the +02:00 at the end (because after the first 2–3 times I know the server uses my local time zone) and read the time off at a glance. If the server were to send 2025-08-07T23:23:45ZI would have to do the math in my head every time and there’s a higher risk of making mistakes.[1]

I know this is a fairly niche scenario (and UTC certainly is a good default); but I don’t see a downside to allowing other time zones, where it makes sense to those users.

  1. Probably even more so for people whose time zones are offset by a non-integer number of hours. Or right after DST ends, when I need to retrain myself to add only one hour, instead of two. ↩︎

7

Using fixed-precision, standardised-timezone ISO-8601 strings means you can compare and sort these date-times without needing to do any parsing.

Regarding the nonce and gentoken, what’s the rationale of making the token generation algorithm standard over simply returning the token in the get/create response and saying the token’s value is opaque?

8

ISO 8601 has cruft like week dates and fractional minutes, as in 2024-W33-4T09:59,5+0200. You probably want RFC 3339 instead of that.

But of course, integer seconds are even simpler.

3 Likes
9

Thanks @barry!

Yeah, that’s what I was thinking – IMO it’d be nice for the spec to say that the nonce MUST be present and nonempty, and perhaps stipulate a minimum length.

(Length being a very weak proxy for entropy, since there’s no easy way for the server to enforce entropy. But this is similar to TLS and other protocols, where the assumption is that the client is responsible for sufficiently strong nonce generation. The PEP could also guide that in the right direction by providing a gennonce example like with gentoken :slightly_smiling_face:)

Yep! I need to convince myself that that’s sufficient, but that’s along the lines of what I was thinking.

(There might still be some collision potential there, since there’s no explicit field separation.)

Oh, perhaps this was my misunderstanding: I thinking that upload tools would handle the nonce entirely internally, with no user interaction with nonces whatsoever. My rationale for this was that getting users to properly enforce the “once” property of a nonce is pretty hard (users love reusing nonces, especially if described generically as “secrets”), so the upload tool should be internally responsible for unique nonce generation.

To ground things, this is kind of the UX I was expecting:

$ twine upload --staged dist/*
<... normal upload output ...>
Staged index available at: hxxps://pypi.org/blahblah/staging/<token>/...

i.e. the tool entirely handles nonce generation, and it’s up to the user to decide whether or not to share that staging URL.

(This becomes interesting in public CI environments where logs are publicly viewable. But workflows like gh-action-pypi-publish already deal with this by registering secret masks :slightly_smiling_face:)

I put a longer comment (sorry…) below about why I think transacting in UTC is generally good with services, but I think there’s a straightforward reason why this PEP specifically should use it: the timestamps in question originate entirely from the server, and the client has no a priori knowledge of where the server is (much less has any reason to find that information useful).

Or in other words: IMO it’s very surprising for a server on the Internet to respond to client interactions with its own local times, especially in a world where “server local time” can change between HTTP requests in a session (e.g. if an upload session is rebalanced to another host in a different zone).

I probably could have said this more explicitly: I was really just thinking of a subset like RFC 3339 (thanks @encukou!) with the stipulation that the server transacts in UTC :sweat_smile:. I agree it doesn’t make sense to really innovate on time formats; I think RFC 3339 with UTC time is on the happy path.

I think there’s really two things here: there’s what the server transacts in (which IMO should always be UTC), and what users are presented with (which should typically be a localized time). It’s possible I’ve misunderstood the PEP’s intended scope, but my interpretation of it was that it defines how an upload client and server interact, and leaves the client presentation layer open.

(With my maintainer hat on: insofar as twine would present these timestamps, I think it would make sense to localize them to the user.)

Yes, completely agreed! I think in effect what we really want is a RFC 3339 subset of ISO 8601 :slightly_smiling_face:

1 Like
10

In the context of a public server like PyPI, I agree with everything you say. However, this PEP is more broadly applicable to Python package indices; and in the context of an internal package index, I think many of your points no longer apply.

Let me give a bit more context: I’m working in the e-Research team at a university. Among other things, we’re building Trusted Research Environments (VMs used for working with e.g. sensitive medical data), which have restrictions on network access and package installation. If this PEP is accepted, we will likely use it in a local package index and write various custom scripts to interact with the API.

In this context, I do know exactly which time zone (and which data centre and rack) the server is located in, I know this time zone is not going to change and I do actually want to see the server local time.

Most of those custom scripts are going to be fairly simple; in many cases, they’ll probably just echo the server response verbatim[1]. Enforcing UTC in the protocol and requiring all these scripts to explicitly parse the datetime and perform time zone conversion would add a bit of unnecessary friction.

(And yes, this isn’t too much work in the grand scheme of things; so if I haven’t convinced you after this post, I’m happy to drop the topic.)

  1. The proposed API is nicely designed and mostly human readable—at least if those humans are sysadmins :wink: ↩︎

11

This is understandable, but I still think it’s principally a presentation question. To use your example: if local time is critical to present, then you’ll presumably want to present it correctly when the local time doesn’t match the server’s local time as well. In other words you’ll still need to do some kind of local time adjustment if (for example) you happen to be in the next time zone over on a given day.

(Overall, I think my opinion here is that this is a low-level implementation detail within the PEP, and that it’s up to client tools like twine and uv to decide if and how to take machine-to-machine time information and present it sensibly to users. In other words, I think this should be UTC for roughly the same reason that the nonce would ideally be required – it reduces degrees of freedom/malleability at the protocol level while preserving them at the client layer.)

2 Likes
12

Let me give a bit more context: I’m working in the e-Research team at a university. Among other things, we’re building Trusted Research Environments (VMs used for working with e.g. sensitive medical data), which have restrictions on network access and package installation. If this PEP is accepted, we will likely use it in a local package index and write various custom scripts to interact with the API.

In this context, I do know exactly which time zone (and which data centre and rack) the server is located in, I know this time zone is not going to change and I do actually want to see the server local time.

Does server local time change in this context? That is to say, does the server exist in a locale that observes twice-yearly time changes? Taking my time zone as a reference, if I upload a package in February then the local time is UTC-0500, but if I’m downloading it in July my local time is UTC-0400. Which one is the relevant time to display? Does the server or client need to perform some conversion to “correct” it to now-local vs then-local time? Do you simply swap the offset indicator or do you add/subtract an hour on the timestamp too? Do you care about uploads that occur during the overlap of “fall back” hours where a later upload can have an earlier “local” timestamp?

Using UTC for the storage and transmission protocols simplifies this into a purely front-end implementation detail that doesn’t require standardizing.

5 Likes
13

I thought about this a bit more and I think it’s sound, but I find it a little scary (especially since Python package names can start with/be all numbers) :sweat_smile:

Another option that occurred to me: what if the server was entirely responsible for generating the session token? That would allow it to be entirely random without requiring us to perform a safe hash construction here, and would still allow the same “disclose the staging URL” flow for users.

Is there any drawback/downside to having the server be responsible for this token?

14

Following up a bit of of order, there’s use case I had in mind for the client to provide the nonce component of the session token: it (possibly?) makes out of band sharing of the stage URL easier. If the upload client and the stage-testing client were disconnected, they’d only have to agree on the nonce out of band, and then both could calculate the stage URL without having to communicate between them or with the server, assuming of course they also both know the package name and version.

If the server generates the token, then it can use any means necessary to do so; i.e. it doesn’t necessarily have to use the package name and version, or it could generate its own random nonce. But then of course the preview link used for --extra-index-url would have to be shared between the client creating the session and any client accessing the stage.

15

I thought about this too, but couldn’t come up with a way to exploit it. If we keep nonce generation in the client, then I’m not against using a field separator as you suggested earlier.

16

Thanks! RFC 3339 seems ideal.

I agree, and would prefer to keep the UTC/Z specification. The other factor for me is that the protocol’s expires-at field is not intended as an attestation of the server’s timezone. While that might be useful to know, I don’t think this PEP is the place to accommodate that information.

17

An alternative is to encode the field lengths as 64-bit integers using struct.pack rather than including their textual representation.