Some questions about PEP 561 stub-only packages

Packaging
1

Hi all! I was glancing at the stub-only packages section of PEP 561 and, after opening an issue on flit to better support them, @takluyver raised some concerns that I couldn’t answer. I copy them here, plus some other questions of my own:

  • “The PEP makes stub-only packages sound like a largely temporary workaround which shouldn’t be needed longer term. Is that still the case, or are there a lot of stub-only packages which look likely to stick around indefinitely?”
  • “Is it actually a problem to have a __init__.py in a stub-only package? Glancing at the PEP, it doesn’t seem like it mentions this, and I would normally take ‘package’ in a Python context to mean a directory with a __init__.py file.”
  • Is there a standard way to collect stubs? I’ve seen projects implement their own find_stubs-like function (see grpc-stubs, django-stubs, and many others from awesome-python-typing). I tried package_data={"": "*.pyi"} (plus several combinations of include_package_data, MANIFEST.in, and other git commit -am 'Fix packaging' stuff) to no avail, but this approach worked.

Thanks!

2

I think those first two points are best asked of the typing-sig as they are not packaging-related.

3

This seems to reveal a broader problem regarding __init__.py. It does not matter whether a directory contains __init__.py or not from the perspective of a distribution (either sdist or wheel), only the importer. So the problem is whether a “package” in a Python context implies the existence of __init__.py. And the answer is no! Because PEP 420 (namespace packages).

Flit can decide whether it wants to support namespace packages or not. Whether a stub-only package can contain an __init__.py wouldn’t matter if it does. But you’ll need to ask the typing folks for clarification otherwise.

I don’t think they get special treatments. From packaging’s perspective *.pyi files are like any other data files, and expected to be picked up by normal data file collection methods.

It seems to work for me to set include_package_data, and add the following line to MANIFEST.in:

recursive-include foo-stubs *.pyi

(Replace the *-stubs directory name to suit your need.)

1 Like
4

Thanks for the clarifications!

And regarding MANIFEST.in… you’re totally right, apparently I got it two days ago but I had forgotten :joy: (global-include *.pyi in my case, but yours is more explicit).

5

And this answer also reveals a broader problem :grin:

Namespace packages are not just a way to save a file. They’re a complicated mechanism for collaborative installation that is difficult to explain and leads to even more difficult side effects (such as the entire standard library being importable as “Lib.antigravity” on Windows).

I’ve taken to referring to install packages as packages, and all .py files as modules, then clarifying how directories work. And I recommend people always use __init__.py files in module directories - sometimes even if they’re doing collaborative installs.

Is this something we should add to setuptools as a default? I don’t think pyi extension has another use right now, so no harm in including them by default?

1 Like
6

You are technically correct (the best kind of correct) - a package doesn’t have to have __init__.py. But a bit like Steve, I see namespace packages as a weird, somewhat confusing corner case. Normal packages have __init__.py, and if you say ‘package’ with no qualification, I assume we’re not talking about a namespace package.

1 Like