What's up with scripts and hyphens/underscores?!?!

I have a package that’s been working for years – its name includes a underscore:

adios_db

(GitHub - NOAA-ORR-ERD/adios_oil_database)

It includes a number of command line scripts – defined in the setup.py (I know, old fashioned) like so:

scripts = ['adios_db_init = adios_db.scripts.db_initialize:init_db_cmd',
          ...
           'adios_db_process_json = adios_db.scripts.process_json:run_through',
           ]
# and then, in the setup call:
...
entry_points={'console_scripts': scripts}

This has worked just fine throughout a couple years worth of Python and setuptools updates.

But suddenly, today (Python 3.9.12, setuptools 63.2.0), I get, when trying to run the scripts:

$ adios_db_process_json ./
Traceback (most recent call last):
  File "/Users/chris.barker/miniconda3/envs/adios_db/bin/adios_db_process_json", line 33, in <module>
    sys.exit(load_entry_point('adios-db', 'console_scripts', 'adios_db_process_json')())
  File "/Users/chris.barker/miniconda3/envs/adios_db/bin/adios_db_process_json", line 22, in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
  File "/Users/chris.barker/miniconda3/envs/adios_db/lib/python3.9/importlib/metadata.py", line 542, in distribution
    return Distribution.from_name(distribution_name)
  File "/Users/chris.barker/miniconda3/envs/adios_db/lib/python3.9/importlib/metadata.py", line 196, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: adios-db

WTF? why is it looking for a package with a hyphen in it???

I know that for years, the ecosystem has wanted to use hyphens, rather than underscores, for some kind of normalized package(distribution) naming. I NEVER understood that – you can’t have a hyphen in the importable package name – why would you ever want one in the name in PyPi? or anywhere!!!

Anyway, I can see replacing hyphens with underscores automatically, but why would it ever go the other way around?

SO:

  1. How do I fix this?
    (I confess, I’ve googled a fair bit, but not gone to the current docs and updated to the latest best practices – but I am very annoyed and frustrated that this would happen to a long-functioning project without even updated to the latest Python or anything else)

  2. Why is this happening at all? It seems quite crazy – I really think the whole packaging world had gotten away from the “easy things should be easy” philosophy :frowning_face:

Finally: My next step is to re-install everything from scratch, and see if this is some weird result that got accidentally introduced by some out-of-syc update – but it’s really, really weird!

The hypen and underscore versions of the name refeer to the same package, as they have the same normalized name. There are different normalizations, in some places _ is preferred while in others - is, this is mainly because of historic baggage.

So, I don’t think the issue is the package name. As an exercise, update the package name in the generated adios_db_process_json script, it should result in the same error.

That said, I think we should probably be using the original name, not a normalized version, in the entrypoint scripts we generate, to avoid situations like this.

The hypen and underscore versions of the name refer to the same package, as they have the same normalized name

This in my concern – a name with an hyphen is not a legal python identifyier. So it cannot be a “package” name – at least in the definition of “python package” – it can (and apparently is officially) be a “distribution” name – where a “distribution” is the thing one installs (usually from PyPi via pip), and a “package” is the thing one imports in Python code.

This distinction is particularly clear with “things” (not sure what word to use) with clearly distinct names for the distribution and the python package – e.g.:

pip install beautifulsoup4

then

import bs4

and as a distribution can, in fact, install more than one python package, this is a pretty important distinction.

All this is intro to the idea that while I’m still confused as to why any normalised name would include underscores – it’s absolutely critical that the python package name not include them – so why would the script wrappers (or specification) use them ???

Again, this may be a bug that’s due to something that got out of sync in my install – and I’ll go investigate that now – but if not – this is an issue to bring up with setuptools development, yes? – I’m still a bit confused about how much of this has made its way into the Python standard library …

OK: for anyone reading this thread in the future:

My bug was in fact due to things getting out of sync – between multiple conda environments, editable intalls, and multiple git branches – something got out of wack.

After rebuilding the whole stack everything works fine.

However: looking at the wrapper script - it does use the hyphenated version of the name, which seems really odd and bug-prone – but I’ll take up that discussion with the setuptools devs.

1 Like

I’ll admit that I was put off by the tone of the OP.

That said, I’m glad that you were able to get a working setup again. I’m actually very curious what went wrong here – since nothing in the toolchain should be trying to change the entrypoints for scripts to contain -.

There’s normalisation for package names, ala PEP 503 and friends, but all of those are internal-to-tooling behaviours that users should not see (other that pip install foo.bar and pip install foo-bar and pip install foo_bar being equivalent).

I’ll admit that I was put off by the tone of the OP.

Sorry about that – literally years of frustration over these kinds of issues … but that’s no excuse.

I’m actually very curious what went wrong here – since nothing in the toolchain should be trying to change the entrypoints for scripts to contain - .

well, the console scripts entry point certainly is – but only for the “editable” aka “develop” install. I posted on the setuptools discussion, and apparently that’s due for a refactor, so we’ll see.

I think what went wrong here is that in my mangling of environments, I ended up with a editable install that did work – i.e. the package was importable, but the “distribution” was not properly installed – and the script was using the Distribution mechanism to try to find the module to import. So couldn’t find it.

Turns out that this is only an issue with editable installs – the console scripts for regular installs are nice and simple and don’t have this issue.

And as mentioned, this is due for a refactor (due to PEP 660)

Thanks for your interest,

-Chris