Packaging Standards and PEPs

Packaging
1

Overview

tl;dr; I’m planning on being stricter in the future requiring packaging PEPs to follow the existing processes. My goal here is to reduce the number of standards we have in an “agreed, but not yet implemented” state. I’d appreciate feedback.

Recent packaging PEPs have prompted me to think more about the current standardisation process. In particular, I’m concerned that by focusing on standardisation as an end in itself, we’ve ended up in a situation where we spend more time discussing principles than actually implementing functionality, and that hurts us when our specifications end up impractical to implement, or unimplemented simply because packaging projects[1] don’t have the resources to implement them.

To that end, I think we should tighten up how we develop packaging standards.

The PEP process

We currently develop standards via the core Python PEP process. That has some rough edges, because the process wasn’t designed for developing standards the way we use it, but it does an adequate job. It’s heavily dependent on the PEP delegate, and I’m conscious that as a result, people who are not happy with the way @dstufft or I have performed the role may have differing opinions, but I’m not proposing that we stop using the PEP process. If someone does want to propose that we stop using the PEP process, PEP 609 outlines how to do that - basically, it would need an alternative proposal, which would have to be put to a PyPA vote.

I’m going to assume here, though, that we retain the PEP process, and focus on how we can improve the way that process serves our needs. To an extent, this is going to be me saying “this is what I will be doing in future when I’m responsible for approving a PEP, if people think it’s unreasonable then let’s discuss the options”. But I hope it will also serve as a starting point for people (and in particular people likely to be writing PEPs) to think about what they want from the standards process, and feed back how they think we could improve things.

The role of the SC

In the PEP process, the SC acts as the final arbiter of any proposal. They step in when there’s no PEP delegate, or if there is a need to override the delegate’s decision. Neither of these has ever happened in practice for a packaging PEP, and in reality, it’s not clear that the SC would want to be involved in packaging PEPs, as they don’t have much background in the area (maybe @brettcannon could comment here). I’ve been sort of hoping we never need this option, to be honest. But it is there, should we need it.

Documentation

The packaging user guide includes a section which contains (in theory) all of the packaging specifications. In practice, we’re still migrating a lot of the older specs to that area, and frankly it’s a lot of work that doesn’t add a huge amount of value (until we need to update the spec), so it isn’t really progressing particularly fast.

The full details of the process for maintaining the standards documentation is here. In particular, it requires the following for all PEPs defining new standards:

Such proposals must be accompanied by a pull request to the Python Packaging User Guide repository, against the PyPA Specifications section, that adds a new subsection defining the purpose of the new specification and the role it plays in the wider Python packaging ecosystem.

I plan on strictly enforcing this requirement for any new PEPs that I approve, as we’ve not been doing that with recent PEPs, which means that the specifications section of the PUG is gradually becoming less useful as a reference, rather than more.

I’m not sure what “defining the purpose and role of the new specification” is meant to mean in this context. Historically, we’ve documented the actual specification in the PUG, and the process for making changes seems to assume that’s what happens, so I’m taking that view. If anyone disagrees, I encourage them to get the process changed to clarify (I assume such a process change would be a proposal plus PyPA vote, as usual, but maybe it could just be a PR to the process document? I’m not sure who would make that call). If no-one disagrees with my interpretation, I will at some point propose a clarification to that effect myself.

Reference Implementation

The documentation for the PEP process itself, PEP 1, explicitly states that a successful PEP should include a reference implementation:

Reference Implementation – The reference implementation must be completed before any PEP is given status “Final”, but it need not be completed before the PEP is accepted. While there is merit to the approach of reaching consensus on the specification and rationale before writing code, the principle of “rough consensus and running code” is still useful when it comes to resolving many discussions of API details.

That requirement is less approprate for packaging PEPs, where standards typically impact many tools and there is no one place where a “reference implementation” makes sense to add. However, IMO it is still an extremely useful requirement, as it ensures that the proposal is usable in real-world code, and that there are no major problems with implementing the spec.

Again, I plan on being stricter in the future on expecting this requirement to be followed. While it may not always be possible to implement a major specification in a key project like pip or setuptools as part of the process of developing a PEP, I will expect to see at least a library or a standalone tool that implements the spec[2].

If the authors of a particular standard want to omit the reference implementation, they should explicitly note that in the PEP, and include their reasons for not providing an implementation. I’ll happily confirm in advance of the final approval whether I have a problem with their reasons, and I assume any other PEP delegate would be willing to do so as well. In general, though, an implementation should be included - not providing one should be the exception, not the norm.

Acceptance, Rejection and Provisional Status

In general, a PEP will get accepted or rejected. I’m not a great fan of “provisional” status, in particular I don’t think it works well for interoperability specifications (where “might change in incompatible ways” essentially translates as “doesn’t provide interoperability”…) So while provisional status, as defined in PEP 1, is a valid outcome for the PEP process, personally I tend to avoid using it.

There’s not much else to say here, although it’s worth noting that some projects, like pip, work on the principle that new functionality should be standardised before it gets included. There’s a nuance here, in that technically, “standardised” means “marked as Final”, and acceptance doesn’t immediately imply the PEP is Final. In reality, the acceptance may include follow-up tasks that must be completed before final status can be recorded (including the standard in the PUG, completing the reference implementation, etc). Ultimately, though, it’s up to individual projects what policies they set.

Transition

One thing the PEP process doesn’t cover is how we transition to the new standard. This can involve a lot of complexity, deprecating older legacy mechanisms, interpreting legacy data in terms of the new standard, timescales for dropping non-standard approaches, etc. As a result, the packaging ecosystem is held back by the “long tail” of legacy data, often in unmaintained or obsolete packages.

I propose that we add a requirement for a “Transition Plan” section in new PEPs, that explicitly discusses this sort of issue. It would include such items as:

  • Plans for how we let projects know that they are no longer standards-compliant, and explaining what they need to do to update.
  • Conditions under which it is considered reasonable for tools to drop support for legacy approaches.
  • Discussion of how projects can support the new standard alongside legacy approaches.
  • Deprecation timescales, setting expectations for how long it should be before tools can reasonably assume that they will only have to deal with projects compliant with the PEP.

Covering these questions as part of the PEP avoids situations where the standard works in an “ideal world” where everything is compliant, but is unimplementable in the real world of legacy data and unmaintained but still useful projects.

Summary

OK, that’s it. Sorry for the huge wall of text - I didn’t mean to go on this long, but I wanted to cover the whole process in one place. Thanks to anyone who’s made it all the way through.

I welcome comments or suggestions on what else we might do to make our standards process more effective, more streamlined, and ultimately more effective, both for PEP authors and for end users[3].

  1. Which are mostly under-resourced and volunteer-only. ↩︎

  2. Such a reference implementation can ignore complexities like handling legacy data. ↩︎

  3. And for PEP delegates :slight_smile: ↩︎

10 Likes
2

Does it make sense to recommend the PEP teaches end-users how to upgrade/migrate to a new standard (rather than just teaching the new standard in isolation), or should that be left to tools?

3

Personally, I’d consider that to be under “Transition Plan”. The PEP could say that the transition plan is to expect users to read the PEP and follow a set of steps documented in the PEP. Whether that is a good transition plan would depend on how likely it is that affected users would read the PEP.

As a general point, one of the things that those of us heavily involved in packaging forget is that the average user has little or no interest in following packaging standards, probably doesn’t read PEPs, and likely gets their understanding of “best practices” by copying snippets of code from Stack Overflow or the results of a web search. I’d like the “Transition Plan” section to encourage PEP authors to think a bit more about this fact, and consider the implications.

1 Like
4

I would mention here that in the future I’d prefer the PEP delegate can only pass judgment if they are not participated in any shape or form in creating the PEPs. There’s no clear rule against this, but the SC members did set such precedence in past. In PEP 660 vs PEP 662 e.g., you wrote and maintain the backend powering PEP-660. So IMHO this (at least from the outside) gave the impression that you might not be totally independent when deciding on competing PEPs. There’s been a precedent that SC members abstained from voting on PEPs (e.g. Pablo the syntax/parser changes) when they participated in creating the PEPs. In this spirit, I’d have preferred someone else to make the judgment, and in future, we should follow that principle. (I’m not personally debating that PEP-660 shouldn’t have been accepted, just that IMHO we should avoid such potential personal conflict situations). While we’re reviewing standards and practices we should address this too.

5

The existing process covers this:

Whenever a new PEP is put forward on distutils-sig, any PyPA core reviewer that believes they are suitably experienced to make the final decision on that PEP may offer to serve as the Steering Council’s delegate (or “PEP czar”) for that PEP. If their self-nomination is accepted by the other PyPA core reviewers, the lead PyPI maintainer and the default PEP-Delegate for package distribution metadata PEPs, then they will have the authority to approve (or reject) that PEP.

Otherwise, the default (“standing”) PEP-Delegate depends on the area the PEP affects.

If anyone feels that they would be better placed to make a decision than the default PEP delegate, all they have to do is volunteer.

I’ll also point out that having competing PEPs is (or should be) an extremely rare situation, and already represents a failure of the community to reach a consensus, so it’s likely that there’s no-one with a truly impartial view[1].

But ultimately I’ll abstain from any discussion on the rules for who can be PEP delegate, as I’m bound to get accused of bias :slightly_frowning_face:

  1. I’m tempted to suggest that if we have 2 competing PEPs, then the default should be to reject both, as there’s clearly no consensus. ↩︎

6

IMHO competing PEPs should always go to the SC, and not delegates :man_shrugging:

7

I have discussed PyPA governance and PEPs with a couple people in the past. I think have PEPs tied to the SC is not great because the SC is elected by core devs, which often aren’t involved in packaging and naturally won’t necessarily take the best interest of the packaging ecosystem into account when electing the SC.
That said, I do share (I am assuming) the same concern as @bernatgabor that the current PEP process is not optimal for how much the ecosystem grew.

I have thought a fair bit about this already, and I think I would like to propose to the SC the creation of a Packaging SC. The Packaging SC would be elected by PyPA members, it would work under the authority of the SC and act as PEP delegates, it would also be the point of contact for other non PEP worthy matters (similarly to Issues · python/steering-council · GitHub). This could also help bridging the gap between the Python core team and the Packaging ecosystem, the SC could meet semi-frequently with Packaging SC to get and update on packaging work and issues/difficulties in ecosystem. When we have issues that involve Python core, the Packaging SC could request to meet with the SC to request a resolution and/or coordinate.

I think the main concern for the SC is that they would be offloading the packaging PEPs to a PyPA controlled body, instead of an SC picked one, but the SC could revoke this delegation of their powers if it becomes an issue.

Any thoughts?

2 Likes
8

And how does the new Python Packaging Manager role play into this? From the role announcement, it seems that this is something they would be overseeing:

From the announcement (emphasis mine):

This role will also serve as a community manager to solicit feedback and facilitate discussions amongst stakeholders in the Python packaging community to generate consensus and establish new specifications.

9

To be honest, there’s been little or no public communication from them, so I don’t know what’s going on there :slightly_frowning_face:

10

+1 on from my side.

11

The Python Packaging Manager could coordinate with and help the Packaging SC where needed. The Python Packaging Manager role goes beyond than the Packaging SC’s scope, the former’s purpose is to help managing the work and help projects like PyPI become financially viable, while the SC is meant to be the technical body that reviews and gives the final answer on issues and proposals.

12

The SC does not want to be involved in packaging PEPs because, as a whole, we don’t have much background in the area. :wink:

+1 to this. I personally think it’s okay to withhold final acceptance until such a PR is ready, but I also wouldn’t want to burden someone who ends up with a rejected PEP with writing the docs for PUG upfront.

+1 as well. Again, I personally think withholding final acceptance until an implementation is available is fair, although the difficulty in an implementation may make conditional acceptance prior to implementation okay (says the person who has asked for this in their latest PEP :grin:).

+1 from me.

FYI the SC has no issue asking a delegate to choose between competing PEPs (and has happened under Guido).

I don’t think that would be a concern if that’s what people wanted to do.

I would personally hope they would do what Ewa did for the SC: act as a conduit to the PSF, take notes, some form of continuity between SCs (if the whole council was completely replaced).

13

I’m just a lowly community member, but for what its worth as a recent practical example, the rewrite of PEP 639 heavily focused on a lot of the issues here, in particular documentation and managing the transition experience for users, project authors and packaging tools.

This included:

  • Carefully thinking through provisions for such in the spec itself, and adding/modifying the spec accordingly
  • Adding a detailed rationale for the backward compatibility aspects in the rejected ideas section
  • Adding a top-level “User Scenarios” section (perhaps it should be renamed?) aimed at users (which would be transcluded in or readily adapted into a new user-focused guide on PyPUG or elsewhere), giving a concise but specific and plain language step by step explaining and giving an example of how different use cases should handle the transition.
  • Including, as an appendix, both a simple but fully worked user-focused example, and an advanced example with the specific contents it would entail source project metadata formats, distribution package formats and installed projects, plus other additional examples.
  • In lieu of adding a new bespoke “Transition Plan” section, describing the changes needed from a packaging tool ecosystem perspective and their implications in detail in the “Backward Compatibility” section, which is already supposed to be focused on essentially the same overall issue
  • Describing, broken down by spec, the specific changes it would include in each of the PyPI specifications (which I would, of course, make as a PR to the packaging guide repo once the if and when the PEP was ready to be accepted)

I’m not sure how generally applicable all of this is to every packaging PEP, particularly at that level of detail, but this seems to reflect a lot of the same thinking as the above, and was motivated by my own confusion I’d run into with previous PEPs and standards lacking many of the elements in these areas, in examining and prototyping implementations in existing tools (beyond just the original author’s fully complete and well-maintained reference implementation of license expression parsing), questions and concerns others had about the PEP, and the negative experiences of myself and others dealing with Python packaging.

Also, on a bikeshedding note, do we really need a new bespoke “Transition Plan” section, or can it be a required piece of the `Backward Compatibility" section for packaging PEPs? At least in PEP 639, it seemed to fit pretty naturally there.

14

I’m happy to see it in “Backward Compatibility”. I wanted to explicitly call out the need for a discussion of transition, but I’m not too concerned about where precisely it goes. What I do want to avoid, though, is the approach I’ve seen in a number of PEPs, of stating that “this is a new feature, so there are no backward compatibility concerns” - which ignores the fact that as it’s a new feature, we need to discuss how to encourage people to move off the old method.

3 Likes
15

Excellent point, of course. I’ve seen the same perfunctory statement numerous times reading PEPs, and can certainly see an argument for including a new section with a name that more clearly encompasses that aspect. This could also be useful beyond packaging PEPs, as well. Though, on the other hand, it would leave the Backward Compat section perhaps not so useful on most PEPs, since if narrowly construed most PEPs these days add new features rather than directly breaking existing working code.

1 Like
16

To everyone on this thread, this is the Python Packaging Project Manager. How can I help?

17

I’m not sure. What do you see your role as involving in relation to packaging standards? The announcement of the role here mentions “This role will also serve as a community manager to solicit feedback and facilitate discussions amongst stakeholders in the Python packaging community to generate consensus and establish new specifications” but I’m not sure what that means in relation to the existing standards process.

Edit: On further reflection, one thing that would definitely be useful would be working with PEP authors to help them get clear consensus on their PEPs before submission. This could involve finding communities who need the PEP and getting them actively involved in the process, engaging with people who raised objections to get confirmation that they feel their issues have been addressed, etc. Helping reconcile differing viewpoints so we have a single agreed approach rather than competing PEPs would also be useful, but that’s a special case (and thankfully rare).

1 Like