Towards a `pip audit` subcommand for vulnerability analysis & management

[woodruffw (William Woodruff)]

@pf_moore

What about pip install pip-audit is too much to expect people to do?

This isn’t a totalizing reason, but one good reason that I can think of (as one of the primary pip-audit developers): users often have remarkably messy local development environments. That means multiple copies of Python, multiple copies of pip, virtual environment and pipx and pyenv wrappers, etc.

When a user does pip install pip-audit at the moment, they’re effectively communicating to us that they want pip-audit to be in the same environment as the surrounding pip, and that pip-audit’s functionality is possibility limited to however old the surrounding pip is (since we use pip-api and some custom shelling out of our own). We have workarounds for this, but (IMO) the developer experience of pip audit is much cleaner than pip install pip-audit && pip-audit. Essentially, fewer impedance mismatches to worry about.

@dustin

That said, I also have the ability to pay for development work. Almost all development on pip-audit was paid work and ongoing maintenance of pip-audit, regardless of where it lands, as well as related work is already in a statement of work with @woodruffw’s team that will continue into 2023. It would be very reasonable for me to include additional pip maintenance in that if we’re working towards a pip audit subcommand.

And to second: my team at my company (Trail of Bits) is more than happy to perform maintenance work on pip, both within the context of pip-audit and in a broader sense!

@sumanah

It sounds like part of what we’re figuring out is: what do users want/expect when they’re working with pip, and how much additional friction would it cause for them if they have to invoke the audit command one way versus another way?

I’m definitely biased, but I’ll say this as a user of other package management ecosystems: I more or less expect some amount of auditing functionality in my package installer. npm audit is probably the most widely used example (and has plenty of flaws, as described upthread), but I also regularly use cargo audit. That’s a funny case since it’s technically a third-party command, but it has a major Rust WG behind it (RustSec) and doesn’t have the same DX problems as a separate pip-audit command since cargo has allowed third-party commands from the beginning (not that I think pip should!)

The point about additional friction is a great one: I don’t want a prospective pip audit subcommand to suffer the same security fatigue fate that npm audit does. My first blush idea for integration was to have pip audit be 100% explicit, at least to begin with – users should have to explicitly invoke it, rather than it being a side effect of a pip install ... invocation. That might make sense to change over time, but I think that would be minimally disruptive for an “MVP” while also getting auditing functionality into as many developers’ hands as possible.

@sumanah

Additionally: beyond maintainer capacity, what should our criteria be for including particular commands and not others within pip? Do we need to support consistency in the user’s mental model of “this is the kind of thing one uses pip for”, and if so, what are our users’ mental models about that?

Again showing my bias , but I think of pip as a “package management system.” It already has functionality that’s strictly outside of package installation, but all current functionality (AFAIK) has something to do with querying, managing, or checking the state of Python packages and package distributions.

From there, I think pip-audit would qualify for subcommand inclusion on the basis that (1) it operates entirely on the same objects and state as pip ordinarily does (i.e., it does not broaden the scope of things pip concerns itself with, even though it adds new code), and (2) it exposes functionality that exists in other package management ecosystems, like npm and cargo.

I recognize, however, that “other package tools do it!” is not necessarily a precedent that the pip maintainers wish to establish. But I think the combination of prior art and the limited domain of interest make pip-audit a reasonable candidate in particular for inclusion.