Towards a `pip audit` subcommand for vulnerability analysis & management

[dustin (Dustin Ingram)]

@pradyunsg

TBH, I’m not too keen on this development model. I’d much rather that pip-audit’s code move into pip as pip audit, and then evolve like any other pip command beyond that point.

FWIW, I’d be happy to trust one or more of pip-audit’s maintainers with the commit bit on pip, to help with continued maintainance of this (and other pieces, if they have the bandwidth/interest) of pip.

I think finding a way to do this this is fine with me as long as we can figure out what should happen to the features that I’m not expecting pip to want to adopt, like support for non-PEP 691 APIs, but I think these are minor enough that the option to just drop support for them is on the table.

@bernatgabor

I personally feel that pip does one thing: discover, download and install packages. IMHO the audit feature sounds very useful but orthogonal to its goal.

I’d suggest thinking about pip audit as tooling to determine what to install and whether to install it.

@bernatgabor

I’d prefer to live alongside it, perhaps as package-audit package?

This is the current status quo.

@bernatgabor

Is this package at all pip related?

Yes, this project essentially has a direct dependency on pip.

@fungi

it seems like the end goal is to have security
advisory information served from PyPI and reported to end users at
the time they’re trying to install things

I’d say that running an audit at install time is explicitly not a goal for pip audit: I think via npm’s attempt to do the same thing and the reaction to that feature, we can conclude that that’s not a good idea (npm audit: Broken by Design — overreacted).

@dstufft

Though if we want this to be available to more than just pip, maybe the right approach is a library for the “core” parts, then a pip command that uses that library

I think this makes sense as well!

@dstufft

As part of that I think using pip-api seems wrong as once it’s part of pip, it can just use pip’s internal APIs directly.

This is still in draft but essentially ‘use pip’s internal APIs directly’ is exactly what pip-api would do if it was vendored: Support being vendored by di · Pull Request #138 · di/pip-api · GitHub

@pf_moore

What are your reasons for thinking this should be added to pip rather than living as a separate application?

I kind of addressed this in OP, but the main reason is that we want to make this “canonical and easily available to every Python user”.

(A minor reason is that this depends heavily on pip, for now we can sustain that maintenance burden but I think long term it would be much easier to maintain if it were part of pip itself in some way, similar to other tools like pip-compile that depend on pip.)

@pf_moore

As a separate pip audit subcommand, I’m not convinced it fits within the scope of pip. I see pip more as “managing packages on my machine” and this is more about “querying information from a package index”. It’s a fine distinction, and one that’s far from hard and fast. But if we do consider it as in scope, then I’d want to follow our usual principles, of using standardised interfaces and avoiding implementation-specific approaches. As I understand it, pip-audit uses the PyPI JSON API to get vulnerability information, so I think we should look very seriously at standardising the JSON API[1] before we integrate this into pip. That way, other indexes have a clear target to work to (for example, I could imagine devpi wanting to mirror vulnerability data as well as packages from PyPI).

I think this is addressed in the roadmap, we’re not expecting pip to take a dependency on non-standardized APIs.

@pf_moore

Apart from anything else, I don’t think we want to be in a position where pip-audit gets fixes added and released and they take up to 3 months to get into pip (which is what vendoring could result in) - that just gives a bad message about using the pip subcommand rather than the standalone version.

This is a really good point that I failed to mention – thanks for raising it.

@pf_moore

One other thing I think we should be careful of here, is whether the pip team has the resources to deal with security-related issues. I am no security expert (which demonstrates my point ) but if pip started flagging a project as having a vulnerability and we got an angry project developer asking for urgent help “because it’s a security issue”, which we couldn’t give, we could very easily end up in a bad situation where “pip developers cannot respond to security problems” ends up all over twitter… I, for one, don’t want to be put in a position of having to deal with something like that. And yes, I know that in the scenario I describe, “speak to the owner of the project being flagged, or to PyPI if you think the alert is wrong” should be a reasonable and sufficient response. But people aren’t like that, sadly.

Based on our experience with the project over the last year, this seems unlikely: given that pip-audit only reports on known vulnerabilities that have previously been disclosed to the project maintainers, and that lots of other tools and technologies will report on the same vulnerabilities (in the same format), I think it would definitely be unreasonable to expect pip developers themselves to be responsible here in any way.

That said, users will do just about anything if you let them so it probably will happen. The pip-audit maintainers can definitely help with this along with related maintenance, and if there’s other resources that would make this better I feel confident we can find a way to make them available.

@pf_moore

I haven’t looked very much at pip-audit, but one concern I’d have here is people thinking that “no warnings from pip audit” equates to some form of approval or assurance from pip that things are OK. On the other hand, if we bury the information in disclaimers, it’s going to be a pain to use. It’s hard to tell from the pip-audit issue tracker, but how much “real world” use has this seen, and what sorts of end user issues get raised? I’d like a better feel of the expected support burden that this is going to add to pip, and if that means collecting data from pip-audit as a standalone project for a while, I’m fine with that.

We address this somewhat in our security model. I think the biggest risk here is that users confuse ‘audit for known vulnerabilities’ with ‘do some static analysis’, and think that pip audit will protect them against unknown vulnerabilities in novel code.

Adoption is definitely much lower than I would imagine we would get if this was a pip subcommand instead, but so far we’ve found that being very clear that this is about known dependencies and doing a little education about the security model has resulted in users having a pretty good understanding of what pip-audit does and how it can protect them.

So far the most common end-user issue over the last year is that they don’t want to remediate known vulnerabilities (for a multitude of reasons) and want some means to essentially ignore the results from an audit. But since this is effectively the same as never running an audit in the first place… we’ve pushed back on this. Otherwise, the support burden has been fairly low.