That’s fair, but I don’t know him, and I’m sure I’m far from the only person in that position. Trust is very hard to win and planning to barge on forward without people’s trust is not filling me with confidence.
1 Like
I don’t have access to my previous email (it’s from my previous job and it’s been deleted). I removed that google account from my phone but that did not copy over the info to my other google account in google authenticator. The authenticator is empty. My biggest mistake was to not use my personal email and bitwarden for everything. I did everything wrong, I am a great example of bad practices.
1 Like
Hi @tarekziade, you are not personally being denied, it is just not our means of promoting folks up through the various levels of “trust” as contributors to PyPI over time. It provides not just a basis of trust in the community, but with the PyPI administrators as well. The primary trust I concern myself with in these situations is one of commitment, it is not trivial to onboard and provide context and access to someone who is unlikely to remain engaged in the role after they find relief from their specific issue.
We look for folks who have made consistent contributions in some combination of discussions, issues, development, documentation, etc when bringing onboard moderators and admins.
All I can commit to at the moment is that I will be spending some time on the backlog Friday mornings for the next few weeks. I won’t commit to clearing it as I do not have time, and it undermines the requests I have made for a paid support role for PyPI internally at the PSF year over year. I have bent over backwards or spent weekends doing this work in the past and as soon as the squeaky wheels quiet the discussions stop.
To anyone: if you or your employer is interested in funding a paid PyPI support role for an inaugural year, please reach out: ee [at] python [dot] org.
14 Likes
I don’t think I am asking for trust from you or anyone. I am asking for the admins to change the process if possible so people YOU don’t trust can still volunteer.
By the way, it’s interesting that you use the word “barge” here, knowing that some folks have been waiting for over a year to get a resolution.
2 Likes
Please don’t mischaracterize the current status. There is about a 4 month backlog of unaddressed account recovery requests, and many more that were initiated, responded to, and have not met the necessary requirements to proceed with recovery that may be over a year old.
I was talking about this one PEP 541 Request: Kconfiglib · Issue #2526 · pypi/support · GitHub
Is there any pre-work I can do to help you ?
(see my previous proposal)
1 Like
Thanks for answering, this is useful. You are right, I am not planning to contribute on this for years to come – just sporadically as a volunteer.
But until you find paid staff for this, there should be a way for you to use volunteers to help you, anything that does not require admin access and just interacting with issues and investigating the users profiles would be based on public info.
I won’t commit to clearing it as I do not have time, and it undermines the requests I have made for a paid support role for PyPI internally at the PSF year over year. I have bent over backwards or spent weekends doing this work in the past and as soon as the squeaky wheels quiet the discussions stop.
This part worries me a lot. Sounds like rolling out mandatory 2FA without that support role is not going to improve your situation.
1 Like
I’ve been worried for years 
4 Likes
My snap take here, after following the packaging discussions for the past 6 months or so, is that either Python the project needs to treat packaging as a core concern that falls under its responsibility and is a top candidate for funding, or packaging needs to split into its own organization with its own not-for-profit corporation seeking corporate sponsorships and paid employees.
I do understand that I’m new here and that this is a very “rest of the f—-ing owl” thing to say. I also don’t have much insight into the functioning of the PSF. I’m just very much aware of packaging being treated as a secondary concern by core developers and the SC, at least historically. NPM was its own organization for 6 years before GitHub acquired it. Crates.io seems to be managed by the Rust Foundation. So it seems like there is precedent for this kind of thing working - but packaging can’t stay in this weird middle ground where it is simultaneously critical and not critical to the Python ecosystem.
It’s a serious indictment of something that this is the situation. You’d think with so many software engineers involved we’d all agree that something working isn’t a reason to not maintain it, and we shouldn’t have to display how bad the support need is for something like this.
4 Likes
It’s a serious indictment of something that this is the
situation. You’d think with so many software engineers involved
we’d all agree that something working isn’t a reason to not
maintain it, and we shouldn’t have to display how bad the
support need is for something like this.
You’d think that, but as an employee of another similar open source
software foundation representing projects relied on by many
billion-dollar/Fortune 50 companies, I can say without hesitation
that squeezing funding from those corporate purses is extremely
hard, and finding enough to pay (not even going rate) for just one
talented engineer is often insurmountable when you take the other
costs of operating a non-profit into account. It’s much easier to
get interested companies to pay someone directly (i.e. assign one of
their employees to you), but more often than not you’ll see them get
promoted or reassigned into another division the moment you’ve
finished training them up to do the tasks you need; and then you’re
back at square one again.
Thing is, this situation has only been getting worse… again not
just for Python, but everywhere in open source. It’s a classic
Tragedy of the Commons: companies are unlikely to fund open source
communities that are critical to their own business because “someone
else will do it,” so it ends up falling on a handful of volunteers
and people donating their own time after hours because they’re
already being paid to work full time on other stuff. The few
companies who do donate get asked for more and more favors to make
up for all the ones that don’t, and execs/investors start to ask why
they bother to shoulder the load while their competitors are getting
a free ride. Every year I find myself wondering which companies are
just going to stop sponsoring our community infrastructure efforts,
and whether we’ll be able to find enough new donors to take their
place. It shouldn’t be like this, and I get the impression most
users are comfortably oblivious to the problem until it critically
impacts something they rely on.
10 Likes
I don’t know how much money flows through tidelift, but one could argue that PyPI is a dependency of all projects hosted on PyPI, and there are a number of PyPI-hosted projects being lifted. Is it worth reaching out to them to see whether PyPI as a project could be lifted and whether that would bring in enough to to have any impact?
3 Likes
Thank you for your insights here. As an outsider I’m very unfamiliar with how sponsored open source work happens in practice. It feels obvious that all the companies that use so much should see how contributing back benefits them, but as you say it’s very much a “tragedy of the commons” or “prisoner’s dilemma” situation. How frustrating.
1 Like
Is there a misunderstanding here?
The way I read this, the assumption is that the trusted people working on PyPI have tasks A and B to fulfill, where tasks A require high level of trust, while tasks B do not. And @tarekziade (and others?) are offering to work on tasks B so that the trusted people can spend more time on tasks A. Right?
The next question would be whether or not there are indeed such tasks B that do not require a high level of trust and were they being delegated would free up enough time for trusted people to work on tasks A.
4 Likes
It’s some of that, but IME it’s also largely a failure of bureaucracy. Corporations know how to procure things and services to obtain well-defined deliverables. But open-ended maintenance of open source software is not a thing or a service and the deliverable is a nebulous, ongoing result outside of their control. Companies simply don’t have processes for “purchasing” whatever OSS maintenance is. [1] This is an issue to the extent that, as mentioned, it’s often more feasible for companies to just pledge development hours from their existing employees (not ideal for the projects, as also mentioned).
There are some companies explicitly trying to bridge this gap but it’s unclear to me that any of them have been especially successful to date. ↩︎
3 Likes
Yes, it seems like some of the discussion may have missed that, but I think that’s what @tarekziade clearly said:
So it seems like he’s offering exactly what was asked for earlier in the thread (to do the “grunt work”).
That makes sense, but. . . I mean, it also seems like a problem in the big-picture view. If it’s only possible to contribute in large amounts over a long period of time, that will unavoidably mean that the number of potential contributors is tiny. In the long run it seems like it would be useful to have some sort of system that provides a smoother onramp for newcomers to contribute in small ways.
I hasten to add that I realize creating such a system would itself be yet another drain on the time and energy of the already overburdened key people here. So no pressure. 
But just saying, hey, if at some point there is funding for a position, maybe good to make it explicit that part of that position would be laying the groundwork for spreading the work out a bit more down the road.
That said, do we have a clear idea of what these “grunt work” tasks are (if any) that non-trusted, non-committed people could do here and there to take some of the burden off the more trusted, committed people?
Yes, it’s a problem, and not even restricted to for-profit corporations. Non-profit institutions often have a similar issue of getting donors to fund “unsexy” maintenance work rather than flashy stuff like getting their name on the door over some physical space. I remember in college I went to an event where students had breakfast with some of the UC regents, and one of the regents I ate with mentioned, “It’s hard to get anyone to make a big donation just to pay for keeping the lights on.”
2 Likes
This is really the key.
Take a step back and look at large, successful, top tier open-source projects. They merge hundreds of lines of changes every day. Each of these lines could contain a vulnerability (intentional or not). Write access to the main branch is reserved to a tight bottleneck of highly trusted, top-level maintainer(s). Yet the immense majority of these changes been carefully reviewed, are not malicious and get merged. Most changes are even bug-free! Things are in motion and the project is alive.
So how can some projects achieve such a miracle? De-centralization and delegation, plus some level of redundancy. They also acknowledge that trust is not black or white and that mistakes can and do happen. Processes can get better and safer. But the only perfectly secure way to do anything is to do nothing.
CPython is one of these, and we do in fact spend years vetting (most) new contributors and ensuring they can be trusted to merge commits into a public record where anyone else can review the commits and we can trivially undo any damage.
Here we’re talking about processes that have actual access to protected user data, and legal responsibilities surrounding that. I’m not totally familiar with the setup, but I would assume that GDPR makes it harder to give access to new people rather than easier. And the damage that could be done even while a new contributor is training/practicing is real and potentially difficult to revert.
Ultimately, the grunt work in the slow processes has already been automated or written up and people follow it. But (in the package name claim case) it’s probably not even legal everywhere for PyPI to give out a private email address so that someone else can contact the current owner and wait for a reply. When you reach out to the current owner yourself and negotiate a transfer without involving PyPI, you are doing the grunt work and saving them effort.
I really don’t think we’re going to be able to reduce the existing tasks any more than we already have, so all we can do is make them slower by writing 60-post-long threads that those people then have to read because it’s about them and their job. They know that there’s a lot of work to do and they need more people, and are in a much better position to make it happen than we are.
So when Ee says what they need, we should just listen and do what we can, rather than arguing about it.
12 Likes
I dare everyone to find a better example - or “worse” depending on how you look at it.
The Zephyr project (among others) depends on kconfiglib and funded all its development. It has offered to take ownership. Zephyr is incredibly active, successful and sponsored by some of the biggest tech companies, one which used to employ the disappeared kconfiglib owner. In case that matters, you can meet in person Zephyr maintainers regularly travelling around the world at tech conferences. In person or not, they will answer any question or request about this kconfiglib topic in a matter of days.
The main developer and owner of kconfiglib suddenly went off-grid. There’s no recent trace of him on the Internet - at all. His former teammates were worried something bad happened to him (it wasn’t the case). The initial mistake was to have a single kconfiglib owner/point of failure. Lesson learned.
It’s incredibly easy to find evidence of all the above; it “only” takes time. So I can’t imagine how an ownership transfer could be more of a “slam dunk” than this one.
Yet this PEP 541 request was filed not 4 months ago but 1 year ago.
Thank you so much.
2 Likes
I don’t know how CPython works but projects that scale have more than one level of trust.
I bet there are exceptions but in many cases the email address of the project owner can be found in most git commits.
This also sounds like a small part of the whole research process but what do I know.
Maybe the documentation should be clearer about how non scalable and dysfunctional the process is.
PyPI does.
However, PyPI is also different than most OSS projects. Most OSS projects are not services, they’re pieces of software, so they don’t have service level concerns where there are some actions that require a high degree of access to private data and broad powers to affect nearly every Python programmer in the world.
Trusting someone is a lot easier when the damage they can do is limited to committing something nefarious to a code repository (which can later be reverted before release). The kinds of support requests that PyPI gets simply do not exist for the majority of OSS projects, and their processes for dealing with trust do not map well to those types of requests.
Sometimes. Other people explicitly use a private, high value email address attached to their PyPI accounts so that they can get communication from PyPI dropping in a “high priority” inbox while their general public email address is a “low priority” inbox.
In fact, we’ve had this blow up in our face in the past when we transferred a seemingly defunct project from one person to another, when the original author didn’t respond to an email from the email address listed in their public projects when they had purposefully setup a private email on their PyPI account that they monitored more closely.
Maybe something like this when you open a new issue?
7 Likes