Poll: Public/internal handling — what behavior do we want?
I’ve noticed that different people have different ideas about what public/internal should mean. I’ve therefore created a poll to get a broader picture of the actual needs, hoping that this will help the discussion of specific solutions.
Please take a moment to share your view.
Vote here: Poll: Public/internal handling — what behavior do we want?
This is a good PEP and I support it, but in my ideal workflow, I’d never have to manually write an __all__. It always feels like a hack or like I’m using a beta-feature somehow. I’m just supposed to repeat the names of my symbols as strings in this magical module attribute? That just doesn’t happen in other languages.
So I think what I’d like is PEP 842 but without runtime enforcement and with writing to __all__ instead of __export__. (I’d actually like runtime enforcement but it seems difficult to add in a backwards-compatible way and it might not be all that important if everyone’s using an IDE/LSP client when writing code.)
This PEP is forward-compatible with that, so that could be a future extension.
I agree 100%.
Most of the projects I work on use hubs, so you would never write an __all__ for them. PEP 842 and 844 are for when you don’t want to use hubs.
Personally, I think all projects, in the long run, end up benefiting from hubs because the implementation and public layouts inevitably diverge. For that reason, even for a small project, I just bite the bullet and make a hub.
Astropy has a structure much like numpy, with lots of adding together of __all__ from different submodules, and this would definitely simplify things.
I think a DeprecationWarning is appropriate if __all__ is not a list (but don’t let that detail hinder acceptance!).
Marten
100% agree too. My way of controlling my public APIs (even the tiny ones) is to declare everything under an _internal sub-package, and export what I want in the top-level __init__ module. Even bigger packages (think the size of Django) can use this approach, just with more layers (everything still under _internal and the public API composed of N hub layers).
To me PEP 844 was interesting for class-level names, but unfortunately that was deemed out of scope.
I think more thought should be given on the patterns that might emerge from this. You say hubs are a good idea in general (and I agree, as things currently stand), but how many? Should one maintain one hub per module? Per package? I see a risk of this encouraging people to underscore every module apart from __init__ and then exporting everything - sink and tap included - in __init__. This isn’t great, because not only does it erase meaningful namespaces, but it also increases import time for consumers (in a lazy module-less world) who might only use part of a library’s public API.
If the export semantics described in this PEP (which I think are a great fit for Python) could also be expanded to other symbols and not just imports, wouldn’t that obviate the need for hubs a lot of the time?
Valid points. Most packages are small enough, so a single hub would be fine most of the time. For packages that grow bigger, or that want to preserve meaningful namespaces (using modules), they can always do so. But yeah this would have to be made clear to (new) authors.
The advantage of hubs in opposition to a fully public module layout where you use whatever mechanism to declare what’s public and what isn’t, is that you can reorganize code as it evolves much more easily. Sometimes (often?) your internal API structure doesn’t mirror exactly what you want your public API to be. Being able to reorganize your code while keeping the public API decoupled from it is a clear advantage to me. Even if the internal and public structure match, I find it much easier to write backward-compatibility code in the public hubs (legacy names, deprecations, __getattr__, etc.) to keep a clean internal structure. So no, personally I don’t think anything can obviate the need for hubs a lot of the time 
That’s fair: Whenever you introduce a new feature, maybe that feature will be overused. A lot of people worried that the moustache operator would be overused. Personally, I don’t think that people are goinig export things they don’t need to export just because it’s easier.
You should create one hub per package and module in the desired public layout.
As Tim explains in detail: Not always because hubs allow your implementation and public layouts to diverge. I agree that PEP 842/844 do make it easier not to use hubs, which nice for a lot people though!
Aren’t the contents of most hub __init__.py’s mostly imports and re-exports? Okay, perhaps including a few things like __version__ or other metadata. In that case, 844 does work but 842’s from ... export syntax is definitely better suited.
I think you might have missed the word “don’t”?
“PEP 842 and 844 are for when you don’t want to use hubs.” I.e., in the non-hub code where you want to add non-imported names to __all__.
Packages with hub layouts are specifically the easiest to maintain __all__ by hand today because linters can partially help:
from foo import Foo
from bar import Bar # linter error: unused import
__all__ = ["Foo"]
You can combine this with a test to catch mispellings:
def test__all__imports():
import package
for name in package.__all__:
getattr(package, name)
If you don’t care about __all__ at runtime and use a hub layout, then all you really need is:
from foo import Foo as Foo
from bar import Bar as Bar
Frankly, I don’t care about __all__ at runtime. I find it hard to explain and magical, with an incredibly specific purpose. I don’t even think star imports and the public API should be two separate things, but I digress. I only use __all__ because I don’t have something better.
I don’t want to see language level support for __all__ because it’s not respected by tooling (LSPs /autocomplete/etc) today and I doubt this PEP will change that (Has any LSP author chimed in?)
For my personal projects, this PEP won’t really help:
# mypackage/__init__.py
from typing import Final
from .foo import FooClient
from .bar import Bar
__version__: Final = "0.1.1"
def get(a, b, c):
"""convenience wrapper. Use FooClient.get directly for advanced use cases"""
return FooClient(a, b).get(c)
__all__: Final = ("FooBarClient", "Bar", "get", "__version__")
Even with this PEP, I can’t get rid of __all__ entirely, and because I can’t get rid of __all__ entirely, all the issues remain. Not to mention, I’ll probably lose the tuple.
Lastly, this will also force any future proposals with stricter or different semantics to pick a new keyword, because changing export to, say, raise an error would be a backwards-incompatible change.
If you want to get rid of __all__, you do that by writing a pure hub:
__version__ and get into your source tree (e.g., ._src, which is arguably where foo and bar should go too)from ._src export ...)Your final hub could be:
# mypackage/__init__.py
from ._src.foo export FooClient
from ._src.bar export Bar
from ._src export __version, get
All of the objects live in mypackage._src.
When reading the docs, I noticed an inconsistency wrt. imported symbols
7. Simple statements — Python 3.14.7 documentation states
If
__all__is not defined, the set of public names includes all names found in the module’s namespace which do not begin with an underscore character ('_').__all__should contain the entire public API. It is intended to avoid accidentally exporting items that are not part of the API (such as library modules which were imported and used within the module).
Whereas Distributing type information — typing documentation states
Imported symbols are considered private by default. A fixed set of import forms re-export imported symbols.
i.e. by default (without __all__)
__all__ as a mechanism to change that)import X as X as re-export syntax)Considering export just as syntactic sugar to add exports to __all__, is compatible with the first definition. So the PEP seems like a small independent and consistent improvement. But it has an awkward interplay with __all__: In using an explicit keyword we (i) simplify management of __all__ and (ii) make the public intention clear at the definition site - from x export y clearly marks y as public.
However, the inverse is not true: In from x import y, y may be public or not depending on whether __all__ is specified.
Is it reasonable to change the default semantic of imports to private within the scope of this PEP? Of course, there needs to be a transition phase, but one could add something like “from Python 3.xx on imported symbols are considered private. Use export to explicitly mark them as public”. IMHO that would make export semantics more intuitive and solve the above inconsistency. It’s however a mor fundamental change than just syntactic sugar.
The import docs document what import x does: it will pull in x.y even if y was simply imported in x. Imports are public as far as importing behavior is concerned.
Type checkers consider ordinary imports private semantically. So they’ll flag it.
I don’t see how this can be reconciled. The type checker is catching a useful error and you can’t change the behavior of import without breaking code.
What you can do if this PEP were accepted is to ask linters to encourage the use of export.
You’re not proposing a change to export semantics though, are you? I think you mean “exporting” semantics?
Yes “exporting” semantics.
Not sure I understand what you mean by public importing behavior. If you mean it’s becoming part of the namespace, that is also true for _internal_name, and that is fine, the semantics is defined via a convention, not via technical differences.
My concrete question is: What is the language’s standpoint on import semantics?
I have a module mymodule and inside do import x and from y import z, are x, z considered public, i.e. is it safe for a user to access mymodule.x, mymodule.z?
Typing has the clear standpoint that it’s private, but typing is optional and a relatively recent addition to the language. Maybe it’s a documentation issue. I haven’t found a comparable statement outside of typing docs (and they are quite recent). So I’d either conclude that public/private semantics of imports is not defined by the language, or it’s implicitly public because behavior is public and we haven’t stated any deviating semantics. If however the language would already consider imports semantically private, that should explicitly documented also outside of typing.
Yes, I think they are. You make a good point.
I see your point about how in a world with export we may be able to just say that imports are always private unless they’re in __all__. The implicit behavior is no longer necessary since people can painlessly use export.
For projects I’ve been a part of, imported names (except in __init__.py) have been treated the same as underscore-prefixed names. Removing one because it is no longer used internally has never been worthy of a changelog entry, deprecation cycle, or minor/major version bump.
I have no intention of prefixing imports with _ or supporting downstream users who discover import locations by hitting TAB, and I won’t begrudge an upstream project that breaks something because someone working on my projects has done so.
As an aside, I’m all for making the public API more easily discoverable (I’m +1 on this PEP as originally written, but I haven’t reread the latest version), but this round of PEP discussions make Python programming sound like an absolutely joyless endeavor for a lot of people, where VSCode’s autocomplete behavior can override project documentation and drive maintenance tasks.
Pylance uses it for auto-complete, but only when it’s statically declared or under very specific manipulation scenarios.
If your code is dynamic enough to require runtime analysis instead of static analysis to extract API data for documentation purposes, then this convention doesn’t work anymore: the information of “Foo is public” and “Bar is public” is lost at runtime. Yes one could run both static and runtime analysis and combine results, but I think it’s important to remember this limitation of the “redundant alias” convention 