PEP 771: Default Extras for Python Software Packages (Round 3)

Packaging Standards
9

I’m not sure who might have done this originally[1] but I just had a go at looking at this - I used the following BigQuery to find all packages that have a dependency in the metadata that contains extra == 'default':

  SELECT
    name,
    ARRAY_AGG(DISTINCT version ORDER BY version DESC LIMIT 5) AS sample_versions,
    MIN(upload_time) AS first_seen,
    MAX(upload_time) AS last_seen,
    COUNT(DISTINCT version) AS n_versions
  FROM `bigquery-public-data.pypi.distribution_metadata`,
    UNNEST(requires_dist) AS req
  WHERE REGEXP_CONTAINS(req, r'(?i)extra\s*==\s*["\']default["\']')
  GROUP BY name
  ORDER BY last_seen DESC

This returns 126 projects which have at some point had this as an extra – of these, 80 still have it in their latest release, and I think around 60 are independent projects (the remaining 20 are forks or copies of these).

Some of these packages prominently feature [default] in the installation docs, for example networkx and in cases like this, it seems to be what most users would want. It looks like about a third of the projects fall into this category.

For some such as asyncdb, the [default] option is mentioned in passing but not prominently shown in the main installation instructions. For some like pytreeclass the default extra is defined but never mentioned in the docs. There are also several packages where it seems that [default] is not what should be installed by default: for example TopoVelo uses it to list exact pinned versions of the dependencies.

So could we use [default] as the ‘special’ name for default extras instead of introducing a new metadata field? Technically it seems that many (but not all) uses of [default] align with this, but in practice we don’t actually know what the various package maintainers would want - if we did adopt this, we’d be opting them all in, and this would also apply retroactively to any release that used [default] in the past, with no way for any of those releases to opt out. For a small number of packages it would definitely not be what is wanted. So if any authors would not be happy with [default] actually being installed by default, they could not change this for past releases.

I personally think it would be preferable to make this new feature be truly opt-in for maintainers via a new metadata field, and let packages migrate to this for future releases once they have had a chance to digest the implications of the truly default extras. After all, as we have emphasized in the PEP, package authors should consider carefully if it is the right fit for their package, so it would not seem right to just opt in a number of packages. The benefit of the new metadata field is that it also allows multiple extras to be listed, and doesn’t require existing packages to rename whatever extra(s) they want to have included by default (default, recommended, all, etc.).

I won’t repeat all the analysis above for [recommended], but the numbers are a little larger: 137 packages have [recommended] as an extra in their latest release. I haven’t checked the intent for those though.

  1. Maybe you were thinking of Paul’s analysis for PEP 685 (PEP 685: Comparison of extra names for optional distribution dependencies - #17 by pf_moore), which looked at normalization clashes in extra names rather than their intent? ↩︎

2 Likes