A Python packaging newsletter?

[pradyunsg (Pradyun Gedam)]

This is an idea that came up in a chat I had in early 2021, according to my notes. Not taking names since I have not checked if they want me to call out to them in public.

The core idea is that PyPA projects don’t have a consistent unified communication channel to end users for telling them about what’s happening and, by virtue of that, we also collectively don’t have a shared understanding of what’s happening across various projects.

A regularly scheduled “here’s what’s happening” update would (done well) give us:

• a place to tell users that we need their feedback
• a place to announce things that we expect would require significant user-side work
• a place to “see” where the major discussions are happening within the community

An irregular newsletter done poorly would serve as busywork on our end with not a lot of value to ourselves or the subscribers/readers.

There’s a few open questions about this:

• Who will coordinate and manage something like this?
• Do others think something like this would be valuable?
• Are there any pros/cons to this that I’ve not mentioned here?

[pradyunsg (Pradyun Gedam)]

FWIW, I do think this should not be a “single person manages this” thing — at least 3 people being interested in helping run this is likely necessary for this to actually happen and make this sustainable within a volunteer group.

Also, I’m well aware that this isn’t software development but it is still vital and valuable community management work that we have collectively not done. I’m hoping that we (at least) end up with a consensus that we should be doing more of that, even if we don’t end up going with this idea and/or specific individuals to run this specific sort of thing.

[CAM-Gerlach]

I initially was a little skeptical when I first read this, but I could see a monthly newsletter with a curated and concise roundup of the most important updates, changes and discussions for PyPA projects and the packaging ecosystem being a valuable resource, at least for PyPA project maintainers, packaging community members and other devs interested in keeping up to date on packaging. It’s also something I’d personally be pretty interested in both subscribing to and contributing to the editorial team, since it falls pretty squarely within my wheelhouse.

One caution, though—its very hard to see any kind of newsletter reaching or appealing much to the ≈95% of Python users who don’t already care about packaging—even with heavy promotion and tailoring it accordingly, it’s hard to see it reaching more than a tiny fraction of “regular” end users beyond those who already have some interest in packaging, or are in a particular position where they feel they need to stay updated on it.

However, IMO it isn’t the end of the world, as we can still have the news trickle down to those users indirectly so long as we can attract enough prominent community members, “go-to packaging people” for their org, and folks with enough time/interest in packaging to care somewhat even if not enough to follow all the discussions regularly. With that in mind, publishing should of course be on an existing (relatively) high-profile blog/website in addition to email, Discourse, etc., which gives those folks something to share with others, helps people come across it later, reduces the investment to skim it and allows third party content aggregators (Reddit, Hacker News, etc) to pick it up if there’s a particularly interesting item that captures wider attention.

NumFOCUS, perhaps the closest rough equivalent for scientific Python as the PyPA is for packaging, has a similar newsletter they send out giving the latest updates for NumFOCUS sponsored and affiliated projects (basically all the core packages in the scientific Python ecosystem, and lot more), where each project sends in a few bullet points plus NumFOCUS mentions any major happenings on their end as an organization.

It’s nice in theory, and we (Spyder) regularly contribute to it, but I’m not sure it’s really that useful or that many people actually read it regularly since there’s just way too many projects (usually several dozen each newsletter), only a handful of which any given user actually care about, and for which the updates vary in significance with little central editorial control.

To avoid this issue and be have a high value to time ratio for readers, such a newsletter should be reasonably selective about which projects and updates it includes, focusing on those which have a meaningful impact on a substantial number of users, and organizing them in a way that’s quick for the user to skim to just what’s relevant to them.

[konstin]

yes please!

Something like This Week in Rust but as “This Month in Python Packaging” would be invaluable. What’s also neat about TWiR is that it’s just a markdown page you can put in your rss reader or even make pull requests against.

Personally, i’d be interested in seeing the following sections included:

• PEP status changes or expected status changes
• Calls for testing/feedback
• Updates from major packaging projects (cpython/pypy, pip, pypi, virtualenv, tox, poetry, hatch, pdm, …)
• A edited selection of new guides/tutorials

[CAM-Gerlach]

Sounds like a good starting point. I’d also suggest including major PyPI/warehouse updates and PyPUG additions/overhauls (which might be what you’re referring to with “a edited selection of new guides/tutorials”).

[hugovk (Hugo van Kemenade)]

Something for issue 1:

GitHub now detects dependencies and usage from pyproject.toml, no need for the empty setup.py:

Dependency graph supports the Python PEP 621 standard | GitHub Changelog

Standard Python support

[pradyunsg (Pradyun Gedam)]

As a casual nudge: this still needs one/two more people to volunteer to help with this, before I’d feel comfortable saying that we can do this in a sustainable manner with volunteers.

[konstin]

I’d also be interested in contributing/editing!

[takluyver (Thomas Kluyver)]

I think it would be relatively easy to put together lists of small concrete things that are mostly uninteresting for people not already in the packaging world - ‘Project X released 1.3’, ‘Project Y merged support for auto-Z-definition’, etc. I periodically leaf through the NumFOCUS newsletter and ‘This week in GNOME’, and they both fall into this pattern. These little isolated events don’t make a great overview, and if there’s one project I’m really interested in, I’m probably better off following it directly.

What I think would be more valuable - but harder - is to trawl through discussions on here and various crucial repos (like pip & warehouse) and try to give people a ‘state of play’ on where packaging more broadly is going. E.g. ‘Status of [project] metadata (PEP 621) support’, or ‘Debate reignites over local packages folder proposal (PEP 582)’.

LWN does this kind of summary at times - in fact they have one or two (2nd paywalled) on Python packaging at the moment. Of course, there’s a reason they charge readers - it takes quite a lot of time and effort to read what we’re all writing and produce a coherent summary, and the writer has to make judgment calls on how much weight to give different opinions.

[CAM-Gerlach]

@takluyver

I think it would be relatively easy to put together lists of small concrete things that are mostly uninteresting for people not already in the packaging world - ‘Project X released 1.3’, ‘Project Y merged support for auto-Z-definition’, etc.

Yup; wholly agreed. I mention similar sentiments above:

@CAM-Gerlach

One caution, though—its very hard to see any kind of newsletter reaching or appealing much to the ≈95% of Python users who don’t already care about packaging—even with heavy promotion and tailoring it accordingly, it’s hard to see it reaching more than a tiny fraction of “regular” end users beyond those who already have some interest in packaging, or are in a particular position where they feel they need to stay updated on it.

Likewise,

@takluyver

I periodically leaf through the NumFOCUS newsletter and ‘This week in GNOME’, and they both fall into this pattern. These little isolated events don’t make a great overview, and if there’s one project I’m really interested in, I’m probably better off following it directly.

Indeed; the NumFOCUS newsletter also came to my mind as an example of what not to do, for all the same reasons:

@CAM-Gerlach

NumFOCUS, perhaps the closest rough equivalent for scientific Python as the PyPA is for packaging, has a similar newsletter they send out giving the latest updates for NumFOCUS sponsored and affiliated projects […] I’m not sure it’s really that useful or that many people actually read it regularly since there’s just way too many projects (usually several dozen each newsletter), only a handful of which any given user actually care about, and for which the updates vary in significance with little central editorial control.

As to what we should do, I do indeed find

@takluyver

What I think would be more valuable - but harder - is to trawl through discussions on here and various crucial repos (like pip & warehouse) and try to give people a ‘state of play’ on where packaging more broadly is going.

much more useful. To quote my previous comments on the subject,

@CAM-Gerlach

To avoid this issue and be have a high value to time ratio for readers, such a newsletter should be reasonably selective about which projects and updates it includes, focusing on those which have a meaningful impact on a substantial number of users, and organizing them in a way that’s quick for the user to skim to just what’s relevant to them.

and summarized my overall suggestion as

@CAM-Gerlach

I could see a monthly newsletter with a curated and concise roundup of the most important updates, changes and discussions for PyPA projects and the packaging ecosystem being a valuable resource, at least for PyPA project maintainers, packaging community members and other devs interested in keeping up to date on packaging.

@takluyver

LWN does this kind of summary at times - in fact they have one or two (2nd paywalled) on Python packaging at the moment.

Yeah, I’d been following that. If we’re going to appeal to a wider audience, particularly on at least a semi-regular basis, we’re going to need to cut it down to a handful of bullet points with links to more information/full discussions, as opposed to LWN’s propose, and focus much more on the final outcomes than on the course of the discussions.

@takluyver

the writer has to make judgment calls on how much weight to give different opinions.

On one hand, this takes somewhat more effort on their end due to typically lacking the depth of subject matter expertise that we do. On the other hand, the writer there is generally an impartial observer, whereas very likely whomever would be writing it would have been involved in some way, or at least a potential vested interest, which makes that trickier.

[petersuter]

Maybe the Discourse plugin to automatically create summaries would be of interest.

[smm (Shamika)]

I am happy to co-ordinate/help in any way possible. I realize the frequency has not been decided yet, but is there any value in having a cadence of every three months?

[lwasser (Leah Wasser)]

i think this is a great idea!!

[lrq3000 (Stephen Karl Larroque)]

I also think this is a great idea, especially considering how active this topic is currently.