Export Statement for DRY Re-exports

Ideas
1

Building a public layout API from a library’s internal layout usually means one of two things today:

  1. Writing every exported name twice, once in an import and again as a string in __all__.
  2. The reflexive-alias idiom, from x import y as y, which type checkers understand but which reads like a typo to everyone else, and which leaves __all__ empty, so you lose dir() cleanliness and wildcard-import control.

I’d like to propose a small addition that removes both: a from <module> export <name> [as <alias>] statement. It does exactly what from <module> import <name> [as <alias>] does, and also appends the name to __all__ in the same statement.

# spam/__init__.py
from ._internal.core export PublicAPI
from ._internal.widgets export Widget as PublicWidget

Nothing is left to sync by hand, and no alias needs decoding. It composes with control flow the same way import does, and supports a wildcard form for the common two-tier layout (an internal module curates its own __all__, a hub module does from ._internal.core export *), plus a lazy variant that builds on PEP 810.

Relationship to PEP 842: this grew directly out of that thread. Both proposals land on the same syntax for the module re-export case, from MODULE export NAME, arrived at independently, which I take as a good sign it’s the right shape. Where they differ: PEP 842 also adds export forms for standalone names, assignments, def, and class, backed by a new __export__ variable and an ExportWarning on non-exported attribute access. This proposal deliberately covers only the re-export case.

Relationship to PEP 844: just posted, and worth reading alongside this one. It proposes public()/private() builtins for names a module defines, the same ground atpublic covers today. Its own text notes that it doesn’t solve re-exports well (public(Widget=Widget) still names Widget three times) and calls this proposal a good companion for exactly that gap: public()/private() for names a module defines, export for names a module passes through. Its Open Issues section asks how PEP 842, 843, and 844 should be reconciled, so that’s probably the central question for this thread.

Quick comparison:

PEP 842 PEP 843 PEP 844
Mechanism New syntax (five forms of the export keyword) New syntax (one form of the export keyword) New builtins; no grammar change
Scope Definitions and re-exports Re-exports Definitions
Export list A new __export__ list, and __all__ __all__ __all__
Runtime enforcement Yes: ExportWarning on access to non-exported attributes None None

Full draft of the proposal with explanation is here: PEP 843 – Export Statement for DRY Re-exports | peps.python.org

Feedback welcome, especially on the one open question in the draft: should export * require the source module to define its own __all__, or fall back to import *'s no-all behavior (bind everything without a leading underscore)?

7 Likes
2

For what it’s worth, I like this one.[1]

Could it be worth also adding some behaviour to export that helps IDEs and type-checkers, like perhaps modifying the metadata of exported objects, or removing plain imports from dir?

  1. More than the other two, by a significant margin. ↩︎

3

I don’t think those are the only two common options.

For example, rich exposes an API that’s distributed across a number of submodules. packaging does the same. Similarly, scipy and sympy have a module structure to their public API. To be honest, I think a nested structure is probably more common than a flat one for larger projects.

While I’m not saying it isn’t worth improving things for projects that do follow the pattern you describe, I think it’s overstating the case to suggest that people “usually” adopt a pattern that will benefit from what you’re proposing.

4

Sorry if this was confusing, but the “hub-and-internal model” I described doesn’t imply flatness. It often does have submodules. You can see an example of this in the PEP where we talk about lazily importing a submodule. Each submodule has its own hub (an __init__.py) that can expose various objects and even more nested submodules.

SciPy is one of the many examples of this, and if you look at its various __init__.py files, you can see how this PEP would collapse most of the __all__ maintenance that it does.

For example: See in a sobmodule, replacing the imports that are exported with export would eliminate __all__'s maintenance. But, nicely, one import would remain—thereby distinguishing semantically the exports from a true import.

Or if we look at its top-level module, this code:

from scipy.__config__ import show as show_config
from scipy.version import version as __version__

submodules = [
    'cluster',
    'constants',
    'datasets',
    # ...
]

__all__ = submodules + ['show_config', '__version__']

def __dir__():
    return __all__

def __getattr__(name):
    if name in submodules:
        return _importlib.import_module(f'scipy.{name}')

could all be replaced by a more elegant:

from scipy.__config__ export show as show_config
from scipy.version export version as __version__
lazy from . export cluster
lazy from . export constants
lazy from . export datasets

No clever machinery needed.

I have to admit, I didn’t understand what Rich was doing when I looked at it. It has a variety of modules that define objects. All of the modules are exposed in dir, but not all of them are part of the public interface in the docs. Within the modules that compose the public interface, there are various objects. These are explicitly exported. They’re just there and documented. Therefore, Rich won’t benefit from any of the PEPs: 842, 843, or 844.

However, if Rich did want to set it’s __all__ lists, it could do that maybe more nicely with PEP 844 or 842. This PEP won’t help it.

I looked at about 20 projects (including Rich and SciPy), but if you want to survey more projects, please feel free to share your results.

This is a really good idea, but I wanted to keep the proposal as modest as possible while hitting all the objectives. Feel free to discuss in this thread any ideas you might have about modifying metadata though. I’ve seen people edit the __module__, but I haven’t thought through all of the consequences of doing that unconditionally.

1 Like
5

It’s a minor point, but I think I’d find the meanings of import and export sort of muddled. In normal language they are antonyms, but here export means “import and add to __all__”.

The proposed syntax is satisfyingly succinct, but I don’t think that it conveys the meaning of the statement well.

1 Like
6

Apologies to all native speakers for me discussing English terms, but “reexport” seems to me a good description of what that statement is doing.

1 Like
7

You didn’t comment on packaging, which doesn’t use that pattern. I’m not going to do a more extensive survey, though, as it’s not really about the numbers. I think the point stands, not every project would benefit from this and I’m not sure it’s “common”. That doesn’t mean it’s useless, I just think you should be careful not to overstate its applicability.

1 Like
8

Very nice chart, thanks!

2 Likes
9

Open questions:

  • Which is the better logic? Marking specific elements as “public” (e.g. through export) or marking other elements as “private”.
  • Do we need or want to create a dedicated namespace for the public interface? The design here is based on the existing two-tier pattern of public “hub namespaces” that do not contain logic, but only import and re-expose symbols from internal namespaces. An alternative would be that there is only hierarch hierarchy and every module contains a set of public and a set of private variables; e.g. one imlementation option would be that __dict__ contains the public variables and __private_dict__ contains the private variables. The latter would in addition only be checked when looking up from within the module scope (actual storage and design up to bike shedding; and yes opt-in external access to __private_dict__ is still possible either directly or through some syntactic sugar).
10

Nested or not, every project will in fact benefit from this, as soon as they declare an all somewhere.

Thankfully now ruff take into account this automatically (at least in init.py files) but i’d rather have a solution like your proposal instead of depending on a tool to handle my exports

1 Like
11

TBH, I don’t think it’s really meaningful to compare the proposals like this given that they have different goals. PEP 842 seeks to address 3 concerns:

  1. Functions/classes/variables that aren’t supposed to be used but are still visible in the namespace
  2. Imports leaking into the namespace of packages that import them
  3. This idea that the underscore prefix convention isn’t enough and that we need something more aggressive

(Concern 2 is the only one that gains any traction for me personally, and mostly only for single file modules where the import sys as _sys workaround is quite annoying.)

843 focuses only on 2 but only for packages[1] and only handling __all__[2]. requests.urllib3 is still there and still looks like public API to anyone using completion to explore a library.

Incidentally, if you compare the contents of the numpy namespace with numpy.__all__, there’s nothing in there that would have been exported without __all__.

>>> set(dir(numpy)) - set(numpy.__all__)
{'_NoValue', '_array_api_info', '__all__', '__getattr__', '__config__', '__cached__', '__future_scalars__', '__spec__', '_specific_msg', '__builtins__', '_core', '_expired_attrs_2_0', '_utils', '__package__', '__dir__', '__doc__', '__expired_attributes__', '__loader__', '_pyinstaller_hooks_dir', '__NUMPY_SETUP__', '_type_info', '_int_extended_msg', '__file__', '_distributor_init', '__numpy_submodules__', '_msg', '__path__', '_CopyMode', '__name__', '_typing', '_globals', '_pytesttester', '_mat', '__array_api_version__', '__former_attrs__'}

So numpy is actually a perfect example of going to great lengths to define __all__ when you could just delete it and your problem is solved.

Whilst I’m not completely dismissing your arguments for still defining __all__ even when you’ve already taken advantage of the __init__.py to create a clean, curated namespace, I don’t find them nearly compelling enough to justify a language change.

  1. which need it much less than single file modules ↩︎

  2. which I consider to be overused and largely unnecessary ↩︎

12

Since my explanation didn’t convince you, you might want to actually try this: delete __all__ and see what extra things get pulled in from a wildcard import, and what’s missing. :smiley:

13

Nice PEPs! Thanks for bringing this topic forward :slightly_smiling_face:

Me when I started reading PEP 844:

image
image889×500 66.9 KB

I wish I had found this paragraph earlier. It deserves much, much, much more visibility.

I end up agreeing with PEP 844: combined with 843, they form a nice way of declaring public APIs. 843 for re-export specifically (flat/nested layouts, but esp. flat ones), 844 for layouts where some modules are public and there’s no re-export (probably).

I’m not sold on how 844 declares public assignments though. None of public(SEVEN=7) or SEVEN = public(SEVEN=7) look good to me (SEVEN = public(7) would look better, but requires horrible tricks to get the symbol name).

Also, it doesn’t offer a way to mark class symbols as public. Those still require the underscore prefixing convention. It’s a bit like PEP 702: you can mark a few things as deprecated, but not all, and not in a very flexible way… So IMO it should stay a third-party library. I’d prefer the standard library to incorporate complete solutions, not half ones :confused: In that regard, 842 is maybe a way to extend __all__ to support class symbols too.

As usual, writing as author of Griffe, a tool that extracts API data from Python source and understands __all__ and other conventions.

14

What does “class symbols” mean? Because @public can definitely be used with class definitions:

% cat foo.py
from public import public

@public
class Public:
    pass

print(f'{__all__=}')

$ python3 foo.py
% python3 foo.py
__all__=['Public']
1 Like
15

That wasn’t clear, sorry, I meant symbols under a class definition:

from public import public, private

@public
class Public:
    @public
    def hello():
        print("hello")

    @private
    def goodbye():
        print("bye")

The use case is documentation. For example “auto-docs” tooling needs to know if something is public or private to decide whether to render it in an HTML page.