Killing off the `egg=` fragment once and for all?

Packaging
1

I’m proposing this to the packaging community, with the hopes of getting thoughts and feedback. I’m interested to hear what you all think!

Background

The egg= fragment is one of two ways to specify the name of a package in a requirement specifier that doesn’t “natively” contain the package name (e.g. a URL, where the base component of the URL may not match the project’s name).

For example, here’s how you can tell pip that a URL requirement installs package foo:

git+https://github.com/some-org/some-repo.git@some-rev#egg=foo

This is equivalent, as far as I can tell, to the preferred requirement syntax:

foo @ git+https://github.com/some-org/some-repo.git@some-rev

Motivation

The egg= fragment is only weakly specified, and currently enables some surprising behavior:

  1. The value of the egg fragment is parsed as a requirement in its own right, meaning that the following happily parses:

    git+https://example.com/foo.git@rev#egg=foo[extra1,extra2]==1.2.3

    This is surprising, since the version constraint has no actual meaning here and the extras (extra1, extra2) actually do end up getting installed.

  2. It isn’t immediately clear what a requirement like this does:

    foo @ git+https://example.com/repo.git@rev#egg=bar

    It would be reasonable to expect it to install either foo or bar (or both?), but it’s not specified which is actually preferred. Experimentally, recent versions of pip install foo rather than bar.

Proposed solution

Deprecate egg= fragment support entirely, and eventually remove it from pip and related tools.

If I’m correct (and I might not be!), then egg= is entirely vestigial in the current packaging ecosystem: it’s functionally equivalent to the less ambiguous and easier to parse name @ url syntax, is (currently) poorly specified, and references a long obsoleted packaging format (Eggs).

Further context

I currently have a PR open (https://github.com/pypa/pip/pull/11617) on pip to refine pip’s existing behavior around egg= fragments: the current surprising behavior around extras is being deprecated, with the current plan being to remove it after two full releases.

4 Likes
2

It would be great if pip could start showing deprecation warnings that would include how the specifier would be rewritten using foo @ – both to encourage updating and to verify how pip parsed ambiguous input.

Is far as I can see, the egg fragment is not part of the standard, so, it is vestigial and limited to tools that need to work with older conventions. But, pip is one of those tools. If these specifiers are on PyPI or in popular requirements.txt files, I don’t think two pip releases is long enough as a deprecation period.
I wouldn’t be surprised if places we “can’t see” – outside PyPI – used this syntax much more commonly than public code.

Can/does PyPI disallow uploading packages with non-standard specifiers?

3

How long ago did pip’s parser gain suppport for the foo@ syntax? Not
all users will (or can) upgrade to the latest versions of pip, and
many use the what’s provided by their stable server distributions
which may be many years old.

Some projects I work on try not to alienate those users and would
likely include both foo@ and #egg=foo together in order to support a
broader range of pip vintages. It may make sense to not print such
deprecation warnings in situations where both package name
indicators are present and contain the same value.

4

Agreed – the PR I currently have open only nudges users towards the name @ ... syntax, but a full deprecation of the egg= fragment should provide the exact equivalent replacement so that a user could drop it in.

Just to clarify: the “two releases” in the top-level comment doesn’t refer to this proposal, but to the current PR open (which doesn’t remove the egg= fragment at all). The current PR only restricts the egg= fragment to valid PEP 508 project identifiers, which should have far less impact :slightly_smiling_face:

I 100% agree that two releases wouldn’t be enough of a period for a full deprecation/removal of the egg= fragment!

I don’t believe it’s something that PyPI can check for in the general case, unfortunately, since doing so would require executing arbitrary setup.py code to grab the requries metadata.

I’ll find a more precise answer in a bit, but pip probably gained name @ url support around 2015, since that’s when PEP 508 was finalized. So, roughly 7-8 years most likely.

1 Like
5

Searching sdists of the top 5k projects (4,753 had sdists, 2022-11-02):

  • found 1,061 matching lines in 154 projects:
    • 231 in requirements.*txt
    • 51 in tox.ini

Searching for #egg= on GitHub, there are:

3 Likes
6

FWIW, to confirm: #egg support in pip is vestigial, from before PEP 508. PEP 508’s URL requirements format support was added in pip 18.1 (2018-10-05).

I’m in favour of removing all special casing we have around #egg within pip, including around some places where pip presents data it has using that. I do think it would make sense to also error out on requirements with #egg in them, at some point in the future but we should start by clearing out the special casing and replacing it with a warning.

3 Likes
7

Projects I work on are definitely still making new releases (for bug
fixes from stable branches) consumed by users of older versions of
pip than that. Ubuntu Bionic (18.04 LTS) carries python3-pip 9.0.1
for example. Our projects likely need to come up with a workaround
in order to support both users with pip too old to safely parse foo@
syntax while also supporting users with new enough pip that it
doesn’t respect the #egg=foo syntax. I’m guessing we could make it
work with forked requirements files for old vs new pip, but will
need to look into our options.

At the very least I need to light a fire under the other developers
to start replacing all the old syntax in our newer branches, a quick
search turns up around a thousand occurrences across hundreds of our
repositories just in master branches alone (not only requirements
files and tox configs, but in integration testing scripts,
configuration management, documentation, and so on). I don’t think
any of them saw this coming, but probably should have.

8

See also:

chart of PyPI uploads: exponential wheels growth, eggs are flat

2 Likes
9

Commenting on a very old thread, i know – but apparently it hasn’t happened yet (killing #egg).

Note: the documentation is very sparse for the package_name@ syntax – I hadn’t done this in a while, and found a bunch of (not all that old) info about using #egg=, but suspected that should have been long ago deprecated, but a fair bit of searching didn’t find the more modern solution.

This thread is where I eventually found it.

So; yes, a depreciation warning sooner than later is good, and perhaps a reveiw of docs to make sure the @ sytax is well documented.

2 Likes