Removing the mention of "Microsoft" from PEP 458

I don’t think it’s necessary to send a secret memo. It’s not a secret that Azure provides Notary image signing, which implements the TUF spec.

Which compliance"requirements are you referring to that you think TUF doesn’t fulfill?

1 Like

I would categorize that as “offered as a service” rather than “used in production by Microsoft”, and even then it all appears to be done through Docker, including key generation and maintenance. So we hardly implemented the spec.

Our own internal ones that I am responsible for helping our internal Python users meet. I can’t share all the specifics, I’m afraid, but certificate generation, handling, and use are covered in a lot of detail.

I’m not sure if these goalposts are worth moving.

I’d assume that a premium, paid for product should be production-ready quality — specially if you are charging for it on the premium SKU. Also, unless you’re familiar with the internals with how azure does Content-Trust (which I admittedly don’t have access), just by looking at the TLD, it appears to me that Microsoft.com-owned domains are used for azure registry and content trust. Client-side tools appear to be Docker’s, yes, which I assume is a positive note as you guys don’t have to re-implement everything. Whether you wrote your own notary server/client code or deployed Docker’s copy I don’t know, but I hope you agree that the line you’re trying to draw is rather thin…

I’m afraid this just reads to me as corporate fist-slamming on a public conversation about an open source project and I honestly don’t know what to say about it. I’m trying to use my imagination as much as possible, but I can’t really tell if you’re concerned about the cryptographic primitives, the two independent security audits (does it need a third?) or the lack of specification about key management the PEP side.

I would like to hear at least some hints though, so we can actually try to address your concerns within the proposal, be it TUF or otherwise.

1 Like

I don’t believe the PEP is justified in saying that Microsoft has chosen to use TUF in any production systems in any way that implies an endorsement. This is not moving any goalposts.

These concerns don’t relate to this proposal, I merely mentioned it because I was surprised that we were allegedly using it in production (and it turns out, by my definition at least, that we are not).

PEP 458 will not make packages on PyPI trustworthy enough for us to use them without manual review and internal mirroring, which is currently the case, and most likely we’ll continue to approach that by building from source. We are far more concerned about malicious publishers than compromised infrastructure. But I’ve already raised these concerns multiple times and had them dismissed, so I’m not planning to spend more of my time trying to affect this particular discussion.

1 Like

I’m trying to stay constructive here. Is the goal here to say “please remove Microsoft from the list of deployments because this is an outright lie”? If so, yeah we should probably move this to the TUF development mailing list because there is plenty of places where this is mentioned, not just this particular proposal.

I didn’t mean to take this derailment either, but this is exactly what I find very confusing. Your definition of using TUF in production is one that would at least, to me, also disqualifies Kubernetes and many other open source technologies you use as being used in production as well. I agree this is a digression, so I’d understand if you don’t want to continue discussing it.

Yes, I’m aware of this. This is part of the reason I am also working with Microsoft in many other initiatives (like Reproducible Builds and such). I know many teams within Microsoft are concerned about consuming open source in a trustworthy fashion and, interestingly enough, some of them believe that TUF is a good part of the puzzle (and I don’t mean to say this as having a formal endorsement from Microsoft)

I’m curious though, when you say “building from source” you mean pulling in from some random VCS and hosting your own repository? or does it mean pull from PyPI and then re-host the packages? or both? I believe there have been plenty of realistic compromises on both fronts for this to be considered a non-issue and probably more damaging than most of those typosquats…

I can’t talk for everybody here, but a compromise of PyPI does not refer to a malicious nobody publishing python2-dateutil to hack people with sloppy dependency handling practices (I would at least call that an exaggerated claim) . I also believe that having strong, cryptographically-enforced integrity and authentication primitives does not preclude having defenses for e.g., typosquatting. on a separate proposal.

I really want to underline that, although you think this is not a realistic threat model, most of the packages that people use fall within 10% of popular packages, so being able to ensure the integrity of those in the PyPA infrastructure as well as mirrors is paramount, as those are the ones that a compromise in PyPI would probably change when there infrastructure is actually compromised.

1 Like

I don’t know for sure that it’s an outright lie, which is why I asked for examples. But if the only example that’s known is that our container registry service provides an interface that will work with existing client tools, then I would say please remove Microsoft from the list. If you have more example, you have a way to email me and we can take it off this thread.

Potentially, though I do have the added visibility of how we run things internally. The best approach is to request a direct quote from an employee specifically endorsing the project, rather than guessing from what can be seen publicly.

I’ve been offered the video of that, as I’m one of the people closely involved in the effort. Thanks for coming and speaking - I’ve heard that it was appreciated by a number of people.

The gold standard is to have an internal clone of the original source repository, build from that on internal infrastructure, and publish to an internal feed. It’s still not clear which corners we will cut yet :slight_smile:

Last time I read through the PEP and the linked material, I didn’t come away with the impression that there were plenty of realistic compromises involved here. Maybe that’s true? But they aren’t being reported to the PSRT - the typosquatting is.

I agree completely. When I refer to a “compromise of PyPI” I’m explicitly excluding anything uploaded via normal processes.

I’m not actually opposing PEP 458 at this point, just trying to make sure that it’s as clear as it needs to be. It’s a complicated and somewhat unintuitive proposal, which doesn’t make it the wrong thing to do, it just makes it hard to explain what it does, why we should do it, and what the extent of its impact will be.

Ok, I’m personally more comfortable with erring on the side of caution when it comes to these public perception situations. I do think it being used in the wild as a “success” story in the Azure Container Registry helps build some good background. Re-reading your original post I realize I gave too much weight to the wrong sentences. If you’re concerned about this looking like official endorsement then I understand why it needs to go.

The CNCF sig-security is building a list of supply-chain-related compromises (which I maintain, so I’m biased). I don’t think there has been a compromise of PyPA infrastructure (and hopefully there never is), but there’s other plenty of examples in similar ecosystems. I imagine we can pull in some of the more relevant examples as background here as well.

Agreed. I think if it isn’t clear enough then it’ll be very hard to find people who 1) can see its value and 2) it will be hard to implement and scrutinize.

@mnm678, what do you think? should we 1) drop Microsoft from the case (or rephrase it to be clearer) and 2) probably borrow from the compromise list here examples that are relevant to the PEP?

1 Like

Steve, thanks for raising this issue. I’d like to second Santiago’s recommendation that we move this over to the TUF developers mailing list. This is an error I should the blame for as my language has been copied by others in the community. To be frank, in some cases we have limited visibility into what companies are doing with TUF apart from what ends up in blog posts, tweets, etc. We’d like to give an accurate picture of TUFs actual adoption and use and didn’t know the scope was as limited as you say. I’m glad that we now have someone on the inside at Microsoft who can give us a more complete picture and help us tweak our language.

My advice is to at least strike “Microsoft” from the list and move on, and potentially just the entire list of companies and just say “TUF has been used other places” and link to some public list that’s maintained elsewhere. The fact that I got a request to split this out to its own topic I think suggests way more effort has gone into discussing whether to keep “Microsoft” versus the simple solution of just taking it out.

3 Likes

Thanks for the split, Brett.

Glad to be able to help out, I do recognise how hard it is to get insight into big companies. Feel free to add me to a thread on your mailing list - steve(dot)dower(at)microsoft(dot)com.

Sounds good. I moved the conversation to our mailing list.

1 Like