Proposing a community maintained database of PyPI package vulnerabilities

It would also make legacy eggs arbitrary code execution during pip install or import a no longer a problem in term of security

1 Like

Just for inspiration/precedent, the Rust ecosystem has a similar system for packages on crates.io: Advisories › RustSec Advisory Database and GitHub - RustSec/cargo-audit: Audit Cargo.lock files for crates with security vulnerabilities acting as an auditing tool for dependencies.

1 Like

Indeed!

Ruby also has a community maintained database of vulnerabilities: GitHub - rubysec/ruby-advisory-db: A database of vulnerable Ruby Gems

And there is a Go proposal/prototype to do the same: vulndb - Git at Google

1 Like

Thanks Oliver! I like the general idea. I’ll try to find some time to read your proposal and give feed after I have dealt with 3.10 feature freeze frenzy.

@brettcannon I think @oliverchang’s proposal belongs in the packaging category to draw in more attention from packaging folks.

There is already a commercial security database for Python packages at https://pyup.io/. They release their database once a month at GitHub - pyupio/safety-db: A curated database of insecure Python packages

1 Like

@tiran depends if this is more for the project on GitHub or coming to a general conclusion. I read:

as the key question. If it’s more for the idea then I can move it back.

1 Like

@brettcannon Yes that’s the key question from me :slight_smile:

For more general packaging/warehouse discussion, I opened an issue at github/pypa/warehouse/issues/9407 (I’m not able to post the full link), or should there be another discussion topic on here for that?

Just a note, the psf and pypa orgs are managed by different people and have different criteria to join. If joining psf does not end up making sense (I don’t personally have an opinion but don’t have a say anyway), it may make sense for you to join pypa instead if you are inclined to. See also: PyPA Members, And How To Join — PyPA documentation

3 Likes

Hi Oliver,

I love the idea of such a database! I hope either the PSF or the PyPI guys will love it too…

Thumbs up!

Cheers, Dominik

2 Likes

I talked to @EWDurbin about this offline and we both agreed that this probably would be more appropriate under the pypa org instead.

Can a mod move this back to the Packaging category so we can give this a little more time for discussion with the members there?

Assuming this doesn’t get any significant opposition there, I’ll probably kick that off sometime next week.

1 Like

Moved over to Packaging

3 Likes

Here’s an example of the repo @oliverchang is proposing to create: GitHub - oliverchang/python-vuln-examples

Here’s a sample of what an advisory would look like: python-vuln-examples/PYSEC-2021-63.yaml at main · oliverchang/python-vuln-examples · GitHub

I’ll be starting the pypa-committers vote to create this as a new repo, pypa/advisory-db on Monday!

1 Like

Hi Dustin,

cool! I in particular like " The goal is to have the pip install (and an additional pip audit ) command automatically report vulnerabilities out of the box." - that would be supercool :slight_smile: :crazy_face:

Cheers, Dominik

1 Like

And flexible for some don’t care about it at some point in time too.

1 Like

Per PEP 609 I’ve started the PyPA vote to create https://github.com/pypa/advisory-db (borrowing the same name as https://github.com/RustSec/advisory-db)

1 Like

The advisory repo looks like it’s being kept up to date by osv-robot · GitHub which as far as I can tell looks like a Google-run bot. Is the source for that bot available somewhere? Will a non-Googler be able to continue to maintain this repo if Google chooses to stop sponsoring the work?

6 Likes

:wave: python friends. I help maintain the advisory-db for RustSec, and also contributed a smidge to the OSV spec. If there’s any stuff I can share around our experience running a vulnerability DB, please just ask!

6 Likes

@ehashman Great question. The source for the robot (and the rest of the OSV tooling) is here: https://github.com/google/osv (the robot specifically is here: https://github.com/google/osv/blob/master/docker/worker/worker.py)

The goal of the repo is that while it can be backfilled/bootstrapped by this robot, individuals can also submit advisories directly to it as well.

The project is in its infancy but Google is making an effort to ensure it is not entirely Google-driven, e.g. there is a proposal that helped define the vulnerability format that had many community contributors: https://tinyurl.com/vuln-json

3 Likes

Thanks for the positive feedback everybody!

This is now public at pypa/advisory-db on GitHub (I’m not allowed to post the full link here for some reason) and currently contains 550~ advisories dating back to 2019 (more will be backfilled soon).

2 Likes

The repo is here: GitHub - pypa/advisory-db: Security advisory database for Python packages published on pypi.org

If you would like to use the CVE database, Thoth’s resolver can automatically remove packages with vulnerabilities stated in the database - see the demo:

Currently, we are ingesting dependency data for other operating systems than Red Hat Enterprise Linux 8 and UBI 8 (with Python 3.8).

If you would like to consume recommendations for other operating systems, feel free to let us know to eventually prioritize data ingestion based on the user base.

And last but not least, thanks for the database!

1 Like