Is PSF losing Python?

PSF
1

Hello, all!

Disclaimer: I would like to apologize for the clickbait subject:) However, it’s something I’m afraid could happen. Please, presume that I have only good intentions in writing this. I’m just trying to raise an issue that really concerns me as a Python user, a Python contributor, and a community member.

I’d like to compare two facts.

  1. PSF has spent more than 75% of the money to PyCon in 2019 https://www.python.org/psf/annual-report/2019/.
  2. There are 1333 open PRs in CPython at the moment and they go back to Feb 16, 2017 https://github.com/python/cpython/pulls?q=is%3Apr+is%3Aopen+sort%3Acreated-asc.

So, there is a very important all-hands gathering of the developers and participants interested in the most popular programming language in the World. Looks logical.

On the other hand, the core of this programming language lacks community contributions management. PRs are being stuck for years. People making the first contribution get zero encouraging and support except “don’t bother” and “you’ve been asking too much” replies from Core Developers on Twitter.

Comparing these two facts I’m getting really concerned about the Python future as a Community. And I have no idea how to fix this apart from PSF should do something about it.

Thanks!

4 Likes
2

I have a book recommendation on this topic. From reading this and what I know is cooking in the PSF I am not so worried.

1 Like
3

I will also recommend the book (disclaimer: I’m quoted in it). I will also say there is stuff going on that is not public, but which have been hampered by the pandemic. But the right people are aware of the issue and are working on solutions.

4

So, basically, @guido and @brettcannon, you are saying “it’s ok, don’t’ bother”. Again.

Honestly, those answers couldn’t be less helpful. This: “right people are aware of the issue and are working on solutions” is just another example of “we are doing great” and no one knows what this means.

Also, it is clear that this has nothing to do with the pandemic. We all were working and connecting online like we continue to do. And unless there are a public plan and a roadmap that states the problem and ways it is going to be solved, “the right people” don’t exist and nobody does anything.

I understand that you believe that it’s ok to do it this way but from an outside look (which is the point that everybody has except “the right people”) it’s just another example of hypocrisy.

Sorry, if this sounds rude, It’s just the concerns I have. Being really passionate about the Python community, I don’t see another way to express it.

3 Likes
5

I certainly don’t think no one knows what this means. For example, I’m sure @brettcannon knows, so do the “right people” who are mentioned in his sentence.

However, it’s also true that not many people know. I, as a core developer (however inactive), don’t know what it means. I guess neither do many/most other core developers. And I’m a bit startled by the secrecy, given this is mostly a volunteer-based open source project.

2 Likes
6

The right people in this context is the Steering Council and the PSF board. And the denomination of “right people” is based on the fact that given our current governance, those are the people with the power to do structural changes and use those resources you mentioned before to achieve the goals. Even if this does not state what those changes are I think there is a failure to establish a parallelism between “we know is a problem and there is some plan to solve it” and “we say we are doing great”.

I think there some misunderstanding there. What Brett is referring to is the impact on the PSF finances due to having to cancel PyCon US (the main source of revenue) due to COVID. This has impacted the resources of the PSF and therefore the possibility of using those resources to projects that could solve this problem. You can read more about this here.

This is clearly subjective and cultural, but I do find the way you state the problem rude. And being passionate about something does not justify making people feel uncomfortable, especially when we are talking about ways to improve the community. If you don’t see any way to express it better, please, try harder because being rude with people that are devoting their time and efforts to the community is the most counter-productive usage of your time and a very small favour to the community you claim to be helping.

As an active core dev, I spend several hours every day reviewing Pull Requests and sometimes a single review takes multiple hours of review, test and design considerations and is a very complex process that I do for free. Even if I do very much agree on with the fact that we certainly could use some help with the PR situation (that the PSF and the SC are working on), having to read community members state that we say is “hypocrisy” and that we say is “don’t bother” and “you’ve been asking too much” clearly makes me not want to do it anymore, so please, consider that next time you plan to justify being potentially “rude” based on the fact that you are stating “hard truths that people don’t want to hear” because the net contribution of that could be negative.

16 Likes
7

I could continue replying and explaining myself here trying not to be rude. Know what, I’m done.

There is a difference between “I spend several hours every day reviewing Pull Requests and sometimes a single review takes multiple hours of review” and “we have a process that ensures nothing being staled or ignored”.

There are obviously a lot of issues with Python community management including but not limited by PRs. I was polite, I wasn’t polite, it doesn’t make a difference. Maybe you should stop doing because of some random post, maybe you don’t. What I know and see that a lot of contributors were lost and continue to stop trying to contribute. Yes, one of them is me and there are a lot of others out there.

I don’t know what is more valuable for the community a single Core Developer or hundreds of contributors. I know that I’m done. I’m leaving all of you in your cosy and positive place without my ugly rudeness.

This was my last contribution in any form to CPython community, CPython repos or whatever. There are a lot of other thing possible to do using Python. So, I’m going to be ok.

Please, keep up the good work!

8

Is not a dichotomy, you can have the core developer and hundreds of contributors happy. Regarding core devs, often is enough to just be respectful and not being rude with people that are giving hundreds of hours of unpaid labour to the community, for the second (contributors), as I mentioned, we are working on it.

I will kindly suggest waiting until the PSF/SC announces whatever plan they have to fix this situation and also to take into account the recent financial challenges of the PSF when looking at a possible delay with respect with their original schedule.

2 Likes
9

There’s no secret, there’s also things we just haven’t bothered communicating publicly because it has not gone anywhere and there is no need for anyone to know (same reason you all don’t need to know about every crazy idea that enters my head; if it isn’t actionable then it doesn’t need to be public :wink:).

We were considering trying to line up some paid staff work to help with things such as the PR queue (there was that poll we sent out earlier in the year to core devs on this topic). The coronavirus put that all on hold due to financial constraints from there being no PyCon US 2020 and thus the PSF losing its major funding vehicle to help make such a thing happen.

I didn’t go into details because there’s zero timeline right now as to when things will get back to a place where the PSF could financially pull this off and thus the SC taking the idea up again; the idea is on indefinite hold until the situation changes. Since it’s not being actively discussed due to the current situation and there isn’t much anyone can do right now, there isn’t any public discussion to have to report on from the SC and thus “stuff going on that is not public” because there’s no point is making it so.

4 Likes
10

I’m not sure that it will affect your decision to continue contributing, but one of the best ways to help with issues such as the PR review bottleneck is by helping to review PRs. It’s a common misconception that only core devs can perform PR reviews. Even if we make the final decision in merging it, having a detailed review from any contributor helps a lot in reducing the time it takes for us to assess each PR. It also helps significantly to make smaller PRs, and to focus your efforts on existing issues (rather than opening a new change proposal issue and a PR to add the feature/change to CPython, as is more commonly done).

I can empathize with the current situation as someone that only started contributing to CPython around a year ago, and had to wait quite a while at times to receive responses on my PRs (even I still have some open PRs waiting on review). But directing that frustration at those trying to do as much as they can in their spare time will not help it.

Compared to some of the other popular languages, we have a rather unique ecosystem that depends far more upon voluntary efforts. If you are unsatisfied by the current situation, the best way to work towards a solution is by trying to directly help yourself. Providing constructive feedback as you’ve tried to do can also be helpful, but it’s imperative to avoid unhelpful commentary. As stated above by @pablogsal, being rude only serves to discourage others in this situation.

6 Likes
11

I’d also submit that having a lot of open pull requests, or PRs that have been open for a long time, isn’t necessarily a serious problem by itself. Any sizable project tends to accumulate PRs which need more work, and PRs where not everyone agrees on what should be changed. It’s easier to propose a change than it is to make a consensus decision, and there are far more people who can make PRs than who can merge them. Some projects automate closing ‘stale’ PRs, but that can easily annoy contributors.

1.3k is a bit extreme, and I’m sure it would be good if review could be made easier. But there aren’t many projects on Github that are as widely used as Python and have such a broad scope (thanks to the ‘batteries included’ standard library). So it’s not easy to say what’s a healthy number of open PRs.

3 Likes
12

While I am neither a core dev nor a contributor at this time, I am both a Python developer and considering contributing against open issues, thanks to the timely post by @lig .

If funding is an issue, is there an active ongoing GoFundMe or IndieGoGo fundraising mechanism operated directly by the PSF?

I don’t see any “in my face” attempts at nonstop fundraising, like monthly subscriptions at different support levels, and while I see a vague reference to JetBrains, I don’t see a mutual effort between PSF and JetBrains to promote a subscription that benefits PSF directly (I a very early Jetbrains PyCharm subscriber).

While I don’t intend to be rude or ruffle feathers, this subject is not suitable for closed door discussion if lack of funding for the PSF is going to be the thing that kills Python interest.

(and in case you’re wondering, the “Click here to make a one-time donation” link on the q2-drive page is broken)

This is one of those issues that needs to be handed off to a team that knows how to do it, and not managed by core devs. If it’s available now, point me to where I can subscribe, donate, or otherwise contribute.

If not, it really needs to be one of the top three priorities.

Best regards and thanks for all your years of hard work… it does not go unappreciated.

rwdim

p.s. I just made a $1,000/year donation commitment and I urge all others who benefit from the work of these fine people to do the same. If you are a full-time Python dev, you NEED the PSF to be flush with cash… Donate NOW at this url: https://psfmember.org/civicrm/contribute/transact?reset=1&id=2

2 Likes
13

Can you elaborate on this? I found this page on python.org (under PSF → Donations), and the link seems to work fine (admittedly I didn’t actually submit the donation form; maybe that’s broken?):

You can also choose to become a supporting member or a sponsor following instructions on python.org (there are a lot of stuff under the “PSF” section). I believe the PSF has been promoting these programs for multiple years now, and there’re periodic fundraising drives each year to raise awareness on one-time donation methods as well. There’s a balance in how aggresive these drives can be, and I honestly don’t think the PSF can do much more to get in people’s face before getting too annoying (which harms the foundation). But apparently the message does not reach the right people right now. Do you have suggestions how the PSF can improve outreach?

The JetBrains collaboration sounds like an interesting idea, but I’m neither in the position to comment nor have the legal expertise to understand its implications (I know the PSF have constraints being a non-profit). I’d suggest checking out other contact methods on python.org as well. While the PSF staffs (which is a different group of people from CPython core developers because it requires different sets of skills to write programs and running a foundation) do read this forum, contacts like email are more likely to reach the more relevant people faster.

14

@uranusjr - Thanks - I found it after I posted, but the email I had from the Q2 drive had a broken link. I’m just saying that this drive needs to be public and have a face (or faces), preferably showing the diversity of the Python developer base both culturally and geographically.

15

How would you suggest the PSF make the message more public? I know they currently use the newsletter and Twitter to send out the messages, and cross-post to some Python-related mailing lists (not sure about other places e.g. Reddit since I don’t use them). Are there past examples to take inspirations from for the “faces” idea? How should those faces be selected and presented?

16

I’m glad you asked (but I’m sure you’re going to regret it) :slight_smile:

Currently, the PSF operates in a cocoon of it’s own making. It should make use of social funding sites to allow people to donate any way they want to.

Also, there should be ONE, and only ONE way to donate on any python dot org or PSFMember dot org or other affiliated site, and only one instance of the code for that mechanism, not multiples.

As it is, if I donate as a PSF member, I can’t make a recurring donation from my account, because the form is different than the one I used on the public site before I became a member. The PSFMember.org site only allows me to specify a $99 membership, a $99 renewal, or a different amount, but no recurring charges because, as I said before, the form used here is completely different than the public form.

The form I used at https://psfmember.org/civicrm/contribute/transact?reset=1&id=2 has multiple amount choices, plus the option to make the amount recurring.

Additionally, it does not allow me create my PSFmembers account and attach the donation to my account at that time, so now I have my PSFMember account which is unfunded because there was no logic to match the email of the account with the email of donations when they occurred in anything other than the expected order.

I’m not criticizing, I’m reviewing. I understand how most developers work and the logic behind design decisions made in near vacuums, when there isn’t a physical team nearby to ask about resources one can use to accomplish a task without deep diving git repos.

Just saying, a team focusing on the process of fundraising would absolve some other team of having to track these things, allowing integration with the systems the rest of the world uses to as a matter of business, and not a feature-request for the cores to deal with.

As for the “Faces of Python”, I would solicit videos from Python developers, up to 1 minute each talking about “Why Python”, why it’s the language of choice for them, and maybe a brief dialogue about the successful projects they’ve implemented in Python. Videos that are suitable are used on the public side and in promo pieces.

Cheers - R

p.s. Willing to jump in to generate a PR fix if there’s nobody working on the site right now.

17

I can confirm this from personal experience. Not CPython but Python infrastructure, nevermind. It feels like there is some elite in power that doesn’t allow you to ask the “wrong” questions or, with PRs, do the contributions they’d prefer to come up with among themselves (even if that means deferring a solution for a year or more, “because we also have a private life”). People are not willing to guide you along, try to make you part of “their community”.

It’s okay to confirm, the “right people” are aware and do “something”. But the secrecy of such a statement is disturbing. There should not be secrecy. There should not be elite. There should be open-mindedness, transparency and the will to collaborate.

You guys should take this criticism seriously and clarify what needs to be clarified. We love Python for some reason. It’s certainly not elite-thinking and secrecy. This should really stop. Please.

18

You are misinterpreting Brett’s short reply. We (core devs and PSF) have discussed our plans to improve the situation in public, for example Brett posted this on the public python-committers mailing list. Our plans are not a secret and there is no conspiracy. There just has not been an official announcement because everything still in an early stage. Our plans to hire additional staff to deal with PRs and bugs were put on hold by COVID-19.

For now the PSF does not pay any core developer to work on CPython. Money is spent on infrastructure, PSF staff, grants and funds for our community (local conferences, meetups, PyLadies etc.). Core devs and developers of other core pieces of Python’s infrastructure are not getting paid. In the past five years I have received travel reimbursement (economy class tickets) and accommodation reimbursement (with room sharing and no breakfast included) for three events from the PSF. That’s it.

To clarify what needs to be done: The PSF has to recover from the financial impact of COVID-19 and increase its revenue stream. Eventually the PSF may be in a position to hire and pay developers to work on Python core and infrastructure full time. The PSF cannot hire somebody for a permanent position until the PSF is sure that it is financially sustainable.

5 Likes
19

I also replied directly in Is PSF losing Python? - #9 by brettcannon to the comment suggesting there was some “secret” thing going on here. I don’t know why people are choosing to interpret a “short statement lacking details” as a “secret”. Had someone just asked for details I could have provided them.

While I do understand why people would like to stay informed, the accusations of secrecy simply make me not want to bother trying to keep anyone informed about anything until there is something very concrete as lacking the “proper” to avoid being accused of secrecy. Ironically that will lead to secrecy as it means I will purposefully deflect questions until there is something to discuss to avoid the accusations and spending time trying to manage this sort of fallout.

Yes, my comment was short and lacking detail. But as Christian pointed out, this has been brought up in other places. All I was trying to do was assuage any concerns that we are ignoring the problems we have as a very large, popular, old open source project. Unfortunately that attempt to be comforting has instead led to me regretting saying anything which is I think the opposite of what those who are frustrated by my initial lack of detail want. But I simply don’t always have the time to reiterate what has been said elsewhere every time someone asks because they choose to follow things somewhere else; the Python community is simply too big for that and my time is flat-out limited (both in the Python sense and as a living, breathing person).

I also hope people realize how costly communicating actually is. This post alone is taking up my time away from other things I could be doing for Python (such as reviewing PRs or fixing bugs, or specifically working packaging-related standardization projects I’m spearheading or participating in). It’s going to take up the time of all people reading this. That total amount of time in the world spent on this topic is not insignificant and a loss of time put into Python itself. That means I have to make a judgment call as to how much time and effort I put into responses which sometimes leads to short ones, sometimes long ones.

Anyway, at this point I don’t have anything more to add to this conversation beyond what I have already said here and other places so I’m going to be unsubscribing from this topic.

10 Likes
20

For what it’s worth, as someone who has contributed on-and-off to CPython and the PSF infra over the last few years I don’t think there’s a cabal of core devs trying to hinder contributions and questions. However, I’ve certainly felt the way you and the OP are feeling and I think here’s a few reasons why:

Many parts of the code-base are ancient, their original developers gone with no one actively maintaining them. If you happened to send out a pull request or reply to one of the issues touching these areas, it can feel like you’re shouting into the void and that you’re being ignored but it’s simply down to a lack of developers interested in that area.

There are also many unwritten rules as far as contributions go, for example the question of format XYZ to PEP7/PEP8/some standard comes up from new contributors every few months and receives the same set of apathetic responses. Refactoring bits of old and stable code to use new python features or for the sake of readability is another common one. There’s a barrier to entry as far as understanding what these unwritten rules are and you won’t pick up on them without reading up on the mailing lists etc.

With a growing number of pull requests and issues, it does seem like ones by core developers get prioritized. From their perspective, it likely makes sense as reviewing code from other core developers is a lot easier since there’s already an established chain of trust. However, as a contributor this can be super dissuasive, seeing stuff get merged while you get no attention. Ultimately, I think this is just down to a large reviewing bottleneck.

All of this isn’t to paint a bleak picture though, I’ve found that once a core developer actually looks at your pull request, it gets a lot better. Almost all reviews I’ve received have been super encouraging, well-written and guide you in the right direction. If you ever get a chance to talk to some of them one-on-one, there’s a similar feeling of mentorship and guidance.

The developer guide is one of the best I’ve seen of its kind, it answers almost any technical questions related to development work. Then there’s developers like Tal and Mariatta who are actively working on getting more contributors and reducing the barrier of entry. Not to mention the developers who work in each sprint to try to cull the backlog.

7 Likes