In particular we’ve started to build a database of vulnerabilities that affect PyPI packages. CVEs are notoriously difficult to match to open source packages and versions, so our goal is to define a standardized shared vulnerability interchange format with precise version/naming that makes them much easier to consume.
An example vulnerability entry would look something like this (more examples here)
id: PYSEC-UNDECIDED-2021-0001
package:
name: httplib2
ecosystem: PyPI
summary: Vulnerability in httplib2
details: httplib2 is a comprehensive HTTP client library for Python. In httplib2 before
version 0.19.0, a malicious server which responds with long series of "\xa0" characters
in the "www-authenticate" header may cause Denial of Service (CPU burn while parsing
header) of the httplib2 client accessing said server. This is fixed in version 0.19.0
which contains a new implementation of auth headers parsing using the pyparsing
library.
severity: HIGH
affects:
ranges:
- type: GIT
repo: https://github.com/httplib2/httplib2
fixed: bd9ee252c8f099608019709e22c0d705e98d26bc
- type: ECOSYSTEM
fixed: 0.19.0
references:
- https://github.com/httplib2/httplib2/security/advisories/GHSA-93xj-8mrv-444m
aliases:
- CVE-2021-21240
modified: "2021-02-12T14:56:00Z"
published: "2021-02-08T20:15:00Z"
We’ve built out a proof of concept for a workflow that automates most of the work necessary to generate these entries from existing CVE feeds. Once this gets going it should result in very minimal ongoing human maintenance work, and we are happy to contribute time to bootstrap this.
Our ultimate wish is to see this community database flow into PyPI’s API/UI and eventually the pip command so users can tell if their dependencies are vulnerable. We’ve already started engaging with the PyPI team on this.
I like this idea. My simple mind take it as whitelist server that pip install could support so by default it only install package that in the list. But user should also able to disable or use difference server of their own. So event other company could have their own whitelist server and tell pip to use it instead of default when needed
This also always difference level of security could the introduced
Thanks Oliver! I like the general idea. I’ll try to find some time to read your proposal and give feed after I have dealt with 3.10 feature freeze frenzy.
@brettcannon I think @oliverchang’s proposal belongs in the packaging category to draw in more attention from packaging folks.
For more general packaging/warehouse discussion, I opened an issue at github/pypa/warehouse/issues/9407 (I’m not able to post the full link), or should there be another discussion topic on here for that?
Just a note, the psf and pypa orgs are managed by different people and have different criteria to join. If joining psf does not end up making sense (I don’t personally have an opinion but don’t have a say anyway), it may make sense for you to join pypa instead if you are inclined to. See also: PyPA Members, And How To Join — PyPA documentation
cool! I in particular like " The goal is to have the pip install (and an additional pip audit ) command automatically report vulnerabilities out of the box." - that would be supercool
The advisory repo looks like it’s being kept up to date by osv-robot · GitHub which as far as I can tell looks like a Google-run bot. Is the source for that bot available somewhere? Will a non-Googler be able to continue to maintain this repo if Google chooses to stop sponsoring the work?
python friends. I help maintain the advisory-db for RustSec, and also contributed a smidge to the OSV spec. If there’s any stuff I can share around our experience running a vulnerability DB, please just ask!
The goal of the repo is that while it can be backfilled/bootstrapped by this robot, individuals can also submit advisories directly to it as well.
The project is in its infancy but Google is making an effort to ensure it is not entirely Google-driven, e.g. there is a proposal that helped define the vulnerability format that had many community contributors: https://tinyurl.com/vuln-json
python friends. I help maintain the advisory-db for RustSec, and also contributed a smidge to the OSV spec. If there’s any stuff I can share around our experience running a vulnerability DB, please just ask!