Structure of the Packaging Strategy Discussions

[CAM-Gerlach]

I want to preface this by highlighting @pf_moore 's post on the Packaging Strategy Part 2 thread that summarizes many of the concerns people have about this process, and directly led to the creation of this “Structure” thread:

Python Packaging Strategy Discussion - Part 2

Agreed, but as @pradyunsg noted, this isn’t something that our contributor base is likely to pick up on (anyone interested in doing this probably already is, the rest of us aren’t likely to do so just because someone says it’s a good idea). Much like with the previous discussion, we can talk endlessly about the problem, but not much will happen unless someone steps up and says they can organise something and deliver people able and willing to work on it. And in that case, they can say what the…

As the person currently responsible for facilitating this process, it would be particularly helpful to hear your viewpoint and responses to the points described there.

@smm

I feel I need to clarify what I see as the expected deliverable from these 5 strategy threads (2 completed, 3 planned).

Err, well, is the second thread really “completed”? It seems that the discussion primarily petered out on that specific thread not because it was in any way “complete”, nor because the nominal “deadline” elapsed, nor due to lack of interest in the specific topic (as active discussion on the same topic has continued on this thread and others).

Rather, as the other folks here commented, many participants were likely just simply exhausted from the first thread, and perhaps discouraged by the lack of tangible results of the “strategy thread” format, and concluded their limited time and energy, so essential to the continued health of the Python packaging ecosystem, would be better spent elsewhere on more productive avenues.

I bring this up because if the whole goal of the strategy discussions was:

@smm

The strategy discussions are a way to validate whether each strand should be taken further. This is the reason why each prompt has been a low ball question to gauge how strong the signal is for each prompt. They are not meant to be deep-dives that would end in a white paper.

…Then the primarily “signal” is likely merely going to be a strong decay in responses (and overall enthusiasm for the effort) from discussion to discussion regardless of sub-topic.

Furthermore, there are a lot of other confounding variables at play—for example, particularly given the lack of active moderation and guidance (as others have mentioned), the discussion naturally bounced between the different primary topics as well as other unrelated ones within one topic’s thread.

And conversely, a lot of simultaneous discussion about or directly related to those same topics happened in other threads; for example:

• On the “Part 1” topic of unification, the prior Wanting a singular packaging tool/vision thread and the Pip/Conda compatibility thread it spawned has still been going strong.
• On the “Part 2” topic of user support and documentation, the discussion here and the new PyPA mechanism for making official recommendations thread has been quite active and constructive.
• The overall packaging strategy, processes and priorities as well as both subtopics has featured prominently in the PEP 704 thread, and to some extent the PEP 582 thread.

Given all of that, it’s hard to see how much useful, reliable “signal” that actually measures the relative community interest and support for each of the topics can actually be gained from this approach, at least as I’ve interpreted the above (perhaps incorrectly).

If that was the primarily “deliverable” here rather than actually having the discussions themselves actually mean something and directly motivate funding and action in the directions participants agreed upon, is there a reason the present approach was taken over a single thread with a Discourse poll, which would seem like a far quicker, less effort-intensive and unreliable (though of course still far from perfect) way to collect such “signal”?

@smm

The expected deliverable from these 5 threads is a high-level strategy document that will indicate which threads (workstream) we will be pursuing in depth and who will be working on each workstream. I expect this deep-dive to produce the policy doc/white paper/technical roadmap that is expected by many in this thread. Generally, I expect the strategy to look like this:

Is the following a correct translating of the envisioned process?

Overall schematic

       User Survey
          / | \
         V  V  V
  Thread 1 ... Thread 5   <--- WE ARE HERE
          \ | /
            V     
High level strategy document 
       /    |    \
      V     V     V
  Stream 1 ... Stream 5
      |     |     |  
      V     V     V
    WG 1   ...   WG 5
      |     |     |  
      V     V     V
 Objectives & deliverables
      |     |     |  
      V     V     V
  Targeted user surveys
      |     |     |  
      V     V     V
     Options analysis
      |     |     |  
      V     V     V
     Choose solution
      |     |     |  
      V     V     V
     Build consensus
      |     |     |  
      V     V     V
 Define technical roadmap    
      |     |     |  
      V     V     V
    Develop solutions
      |     |     |
      V     V     V
    Deliver solutions

To be frank, this whole process seems quite drawn out, bureaucratic and complicated, not to mention volunteer-intensive, a resource which is in critically short supply. It also doesn’t sound close to any of the approaches actually proposed on the strategy thread (competing PEPs with packaging council who decides, evolve pip to meet these requirements, consider new governance structures, get everyone in a room for an in-person summit, etc). Furthermore, it would seem to be at odds with the fundamental established principles of how decisions are made and implemented in the Python and packaging community:

• Discussions are open to all interested, not just members of defined “workgroups”
• Ideas are freely proposed, discussed and iterated until consensus is reached, rather than proceeding through a rigid “waterfall” process
• Solutions are chosen as the result of consensus, not chosen first and then consensus built around them
• Anyone can propose an idea, but it is the proposer’s responsibility to help provide a reference implementation
• Proposals are typically accompanied by a working prototype, rather than actual implementation only coming at the very end

Furthermore, who’s the “we” here? Who’s going to be deciding which to pursue in depth? Who’s going to be part of the workgroups? Who’s going to write the roadmaps? Who’s going to develop the solutions?

Assuming the proposed solutions involve creating something other than a brand new tool/website (which would likely end up in a xkcd 927 situation), if the maintainers and community of the affected tool(s) aren’t involved and supportive throughout this process, any proposed solution will (after going through all those steps) be dead on arrival and you’ll have to go back to square one.

Certainly the current process is far from perfect, and it’s well within scope to propose major changes (as we explored in the first strategy thread). However, a full departure from the underlying fundamental principles is likely to not provide a lot of buy-in from the community, without which any resulting strategy unfortunately has little hope of succeeding.