A Python packaging newsletter?

pradyunsg · February 13, 2023, 9:17am

This is an idea that came up in a chat I had in early 2021, according to my notes. Not taking names since I have not checked if they want me to call out to them in public.

The core idea is that PyPA projects don’t have a consistent unified communication channel to end users for telling them about what’s happening and, by virtue of that, we also collectively don’t have a shared understanding of what’s happening across various projects.

A regularly scheduled “here’s what’s happening” update would (done well) give us:

a place to tell users that we need their feedback
a place to announce things that we expect would require significant user-side work
a place to “see” where the major discussions are happening within the community

An irregular newsletter done poorly would serve as busywork on our end with not a lot of value to ourselves or the subscribers/readers.

There’s a few open questions about this:

Who will coordinate and manage something like this?
Do others think something like this would be valuable?
Are there any pros/cons to this that I’ve not mentioned here?

pradyunsg · February 13, 2023, 9:21am

FWIW, I do think this should not be a “single person manages this” thing — at least 3 people being interested in helping run this is likely necessary for this to actually happen and make this sustainable within a volunteer group.

Also, I’m well aware that this isn’t software development but it is still vital and valuable community management work that we have collectively not done. I’m hoping that we (at least) end up with a consensus that we should be doing more of that, even if we don’t end up going with this idea and/or specific individuals to run this specific sort of thing.

CAM-Gerlach · February 13, 2023, 10:55am

I initially was a little skeptical when I first read this, but I could see a monthly newsletter with a curated and concise roundup of the most important updates, changes and discussions for PyPA projects and the packaging ecosystem being a valuable resource, at least for PyPA project maintainers, packaging community members and other devs interested in keeping up to date on packaging. It’s also something I’d personally be pretty interested in both subscribing to and contributing to the editorial team, since it falls pretty squarely within my wheelhouse.

One caution, though—its very hard to see any kind of newsletter reaching or appealing much to the ≈95% of Python users who don’t already care about packaging—even with heavy promotion and tailoring it accordingly, it’s hard to see it reaching more than a tiny fraction of “regular” end users beyond those who already have some interest in packaging, or are in a particular position where they feel they need to stay updated on it.

However, IMO it isn’t the end of the world, as we can still have the news trickle down to those users indirectly so long as we can attract enough prominent community members, “go-to packaging people” for their org, and folks with enough time/interest in packaging to care somewhat even if not enough to follow all the discussions regularly. With that in mind, publishing should of course be on an existing (relatively) high-profile blog/website in addition to email, Discourse, etc., which gives those folks something to share with others, helps people come across it later, reduces the investment to skim it and allows third party content aggregators (Reddit, Hacker News, etc) to pick it up if there’s a particularly interesting item that captures wider attention.

NumFOCUS, perhaps the closest rough equivalent for scientific Python as the PyPA is for packaging, has a similar newsletter they send out giving the latest updates for NumFOCUS sponsored and affiliated projects (basically all the core packages in the scientific Python ecosystem, and lot more), where each project sends in a few bullet points plus NumFOCUS mentions any major happenings on their end as an organization.

It’s nice in theory, and we (Spyder) regularly contribute to it, but I’m not sure it’s really that useful or that many people actually read it regularly since there’s just way too many projects (usually several dozen each newsletter), only a handful of which any given user actually care about, and for which the updates vary in significance with little central editorial control.

To avoid this issue and be have a high value to time ratio for readers, such a newsletter should be reasonably selective about which projects and updates it includes, focusing on those which have a meaningful impact on a substantial number of users, and organizing them in a way that’s quick for the user to skim to just what’s relevant to them.

konstin · February 13, 2023, 6:18pm

yes please!

Something like This Week in Rust but as “This Month in Python Packaging” would be invaluable. What’s also neat about TWiR is that it’s just a markdown page you can put in your rss reader or even make pull requests against.

Personally, i’d be interested in seeing the following sections included:

PEP status changes or expected status changes
Calls for testing/feedback
Updates from major packaging projects (cpython/pypy, pip, pypi, virtualenv, tox, poetry, hatch, pdm, …)
A edited selection of new guides/tutorials

CAM-Gerlach · February 13, 2023, 10:37pm

Sounds like a good starting point. I’d also suggest including major PyPI/warehouse updates and PyPUG additions/overhauls (which might be what you’re referring to with “a edited selection of new guides/tutorials”).

hugovk · February 14, 2023, 11:45am

Something for issue 1:

GitHub now detects dependencies and usage from pyproject.toml, no need for the empty setup.py:

github.com/dependabot/dependabot-core

Standard Python support

opened 12:15PM - 17 Mar 21 UTC

closed 09:09PM - 24 Oct 22 UTC

flying-sheep

T: feature-request python

There’s exactly two standard ways of specifying dependencies, none of which depe…ndabot supports. 1. [PEP 517’s `prepare_metadata_for_build_wheel()`](https://www.python.org/dev/peps/pep-0517/#prepare-metadata-for-build-wheel). The legacy way to specify Python dependencies is `requirements.txt`. The standard way is having a PEP 517 build system configured. That build system can optionally define a function called `prepare_metadata_for_build_wheel`. If that function exists, the dependencies can be obtained (from the metadata) without building a wheel, otherwise `build_wheel()` needs to be called. Code to just do the above would look like: ```py import build deps = build.ProjectBuilder("path/to/project").get_dependencies("sdist") # or deps = build.ProjectBuilder("path/to/project").get_dependencies("wheel") - {"wheel"} ``` 2. [PEP 631](https://www.python.org/dev/peps/pep-0631/) a new standardized way to specify dependencies in a Python project (now merged into [PEP 621](https://www.python.org/dev/peps/pep-0621/)). It’s supported by [`flit`](https://flit.readthedocs.io/en/latest/) and a bunch of other new build backends. So if you want to continue your (incomplete) way of just parsing files and not calling into Python, this needs to be supported.

pradyunsg · February 24, 2023, 9:26am

As a casual nudge: this still needs one/two more people to volunteer to help with this, before I’d feel comfortable saying that we can do this in a sustainable manner with volunteers.

konstin · March 1, 2023, 4:15pm

I’d also be interested in contributing/editing!

takluyver · March 6, 2023, 1:28pm

I think it would be relatively easy to put together lists of small concrete things that are mostly uninteresting for people not already in the packaging world - ‘Project X released 1.3’, ‘Project Y merged support for auto-Z-definition’, etc. I periodically leaf through the NumFOCUS newsletter and ‘This week in GNOME’, and they both fall into this pattern. These little isolated events don’t make a great overview, and if there’s one project I’m really interested in, I’m probably better off following it directly.

What I think would be more valuable - but harder - is to trawl through discussions on here and various crucial repos (like pip & warehouse) and try to give people a ‘state of play’ on where packaging more broadly is going. E.g. ‘Status of [project] metadata (PEP 621) support’, or ‘Debate reignites over local packages folder proposal (PEP 582)’.

LWN does this kind of summary at times - in fact they have one or two (2nd paywalled) on Python packaging at the moment. Of course, there’s a reason they charge readers - it takes quite a lot of time and effort to read what we’re all writing and produce a coherent summary, and the writer has to make judgment calls on how much weight to give different opinions.

CAM-Gerlach · March 6, 2023, 2:50pm

Yup; wholly agreed. I mention similar sentiments above:

Likewise,

Indeed; the NumFOCUS newsletter also came to my mind as an example of what not to do, for all the same reasons:

As to what we should do, I do indeed find

much more useful. To quote my previous comments on the subject,

and summarized my overall suggestion as

Yeah, I’d been following that. If we’re going to appeal to a wider audience, particularly on at least a semi-regular basis, we’re going to need to cut it down to a handful of bullet points with links to more information/full discussions, as opposed to LWN’s propose, and focus much more on the final outcomes than on the course of the discussions.

On one hand, this takes somewhat more effort on their end due to typically lacking the depth of subject matter expertise that we do. On the other hand, the writer there is generally an impartial observer, whereas very likely whomever would be writing it would have been involved in some way, or at least a potential vested interest, which makes that trickier.

petersuter · March 6, 2023, 4:55pm

Maybe the Discourse plugin to automatically create summaries would be of interest.

smm · March 8, 2023, 7:31pm

I am happy to co-ordinate/help in any way possible. I realize the frequency has not been decided yet, but is there any value in having a cadence of every three months?

lwasser · March 9, 2023, 7:40pm

i think this is a great idea!!

lrq3000 · April 2, 2023, 1:19am

I also think this is a great idea, especially considering how active this topic is currently.