PEP 843: Export Statement for DRY Re-exports

PEPs
1

PEP 843 is ready for discussion. It adds a small statement, from x export y [as z], that does what from x import y [as z] does and also appends the name to __all__ in the same statement.

# spam/init.py

from ._internal.core export PublicAPI
from ._internal.widgets export Widget as PublicWidget

The problem it targets is narrow: it simplifies “hub modules”. Hub modules are modules whose job is gathering names from internal submodules and re-exposing them. Today that means writing every name twice, once in the import and again as a string in __all__, with nothing keeping the two in sync but a careful reviewer. This statement folds them into one.

Full text: PEP 843 – Export Statement for DRY Re-exports | peps.python.org
PR: https://github.com/python/peps/pull/5078

Relationship to PEP 842 and PEP 844

This PEP sits alongside PEPs 842 and 844. In short, PEP 843 just focuses on re-exports. Here’s a quick comparison:

PEP 842 PEP 843 PEP 844
Mechanism New syntax (five forms of the export keyword) New syntax (one form of the export keyword) New builtins; no grammar change
Scope Definitions and re-exports Re-exports Definitions
Export list A new __export__ list, and __all__ __all__ __all__
Runtime enforcement Yes: ExportError on access to non-exported attributes None None

Open issue

One question the PEP leaves open: should export * require the source module to define its own __all__, or fall back to import *'s behavior? Feedback on that specifically is welcome.

Thanks

This went from an idea to a PEP because of a lot of people’s time. Peter Bierma, PEP 842’s author, is sponsoring this PEP despite it being a rival proposal to his own, and pushed on getting the header fields, CODEOWNERS entry, and process details right. Barry Warsaw, Jelle Zijlstra, Hugo van Kemenade, Guido van Rossum, and Éric Araujo reviewed the draft, catching many errors. And in the earlier Ideas thread, Brénainn Woodsend pushed back on whether NumPy actually needs __all__ at all, Paul Moore was skeptical of how common this pattern really is, and James Webber and Xitop objected to the word “export” itself.

11 Likes
Use a with block to control export (public/private/__all__)
Poll: Public/internal handling — what behavior do we want?
2

I feel it should behave the same as import *. Why introduce an exception that’s going to trip someone over?

6 Likes
3

Thanks for publishing this! I still support it and think it would make a good companion to PEP 844. I did just release atpublic 8.0.0a1 which improves a few things, especially related to this use case. With this new version, today I could write:

from ._internal.core import PublicAPI
from ._internal.widgets import Widget as PublicWidget

public(PublicAPI)
public(PublicWidget)

so slightly better than before, but it still repeats itself, which is why I like the from ... export ... format proposed in 843.

2 Likes
4

Curious how public(PublicWidget) retrieves the name of the variable.

5

In Interaction with __all__, I think it would be useful to state plainly that an export statement implicitly creates __all__ if it doesn’t already exist, and show that the following is valid:

from .foo export Foo

__all__ += ["Baz"]

(The current wording is somewhat indirect about this)

3 Likes
6

The full resolution algorithm is here, but essentially in this case, it first gets PublicWidget.__name__ and then looks that name up in the module’s dictionary. If the object it gets back is the object passed (i.e. PublicWidget in this case), then that’s the name it uses, and it’s mostly fast-pathed.

If that isn’t the case, then it searches for an object in the module’s dict that is the object passed, and uses the matching key in module dictionary as the name.

This is how both from ... import X as Y works to add “Y” and why you can confuse it with

class Fiddle:
    pass
Violin = Fiddle

public(Violin)

which adds “Fiddle” to __all__. Fully explained in this section.

1 Like
7

This is awesome, thanks Neil!

I have some suggestions that I think could make it even better:

  1. As far as I can tell, this PEP currently does not disallow

    def f():
    from spam export ham

    But this would be desugared into an __all__ within the scope of f, which isn’t useful and only confusing. But even if this would have updated the module-level __all__, such dynamic exports would be problematic for static typing.
    So I propose we only allow export statements to be used at the module-level, for example by having it then raise a SyntaxError like in PEP 842.

  2. There’s nothing in PEP 843 that mentions or disallows using export in an if typing.TYPE_CHECKING: guard. I can imagine that being very useful for type-check-only utilities in stubs packages (notably _typeshed), where an export exists in the stubs but not at runtime. So I think it would help to to mention that this is an intended use-case.

5 Likes
8

Does this apply to conditions as well? How do people feel about this:

# Lib/concurrent/futures/__init__.py
if _interpreters:
    lazy from .interpreter import InterpreterPoolExecutor  # noqa: F401
    __all__.append('InterpreterPoolExecutor')

Should this be okay?

if _interpreters:  # Okay to conditionally export?
    lazy from .interpreter export InterpreterPoolExecutor

Other examples:

# os.py
if 'posix' in _names:
    name = 'posix'
    linesep = '\n'
    from posix import *
    try:
        from posix import _exit
        __all__.append('_exit')
    except ImportError:
        pass

# cpython/Lib/socketserver.py:135–142:
if hasattr(os, "fork"):
    __all__.extend(["ForkingUDPServer", "ForkingTCPServer", "ForkingMixIn"])
if hasattr(socket, "AF_UNIX"):
    __all__.extend(["UnixStreamServer", "UnixDatagramServer",
                    "ThreadingUnixStreamServer",
                    "ThreadingUnixDatagramServer"])

# cpython/Lib/asyncio/__init__.py:44–49:
if sys.platform == 'win32':
    from .windows_events import *
    __all__ += windows_events.__all__
else:
    from .unix_events import *
    __all__ += unix_events.__all__

# cpython/Lib/pickle.py:40–43:
try:
    from _pickle import PickleBuffer
    __all__.append("PickleBuffer")
    _HAVE_PICKLE_BUFFER = True
except ImportError:
    pass

I didn’t find anything with using export in functions though. And it’s always easier to block something and allow it later if need be.

9

Only the if sys.platform == "win32" example can be expected to be understood by static type-checkers according to the typing spec. So it’s probably easiest to follow typing spec here, and therefore disallow the other examples.

10

So, all of these conditional exports would have to be written the old way—and therefore would already not be understood by type checkers?

Wouldn’t it be easier to allow using the new way of writing them just for the succinctness (DRY) benefit? Either way, type checkers would be confused.

1 Like
11

I’m not convinced this would actually be problematic. Why should type checkers care about the value of __all__as long as it’s a sequence of strings? What problems are actually caused by dynamically modifying __all__ at runtime (something that currently is possible and will still be possible)?

12

There are several reasons for that. One is that type-checkers needs to be able to statically determine which of the module’s symbols are imported when doing a star import. Another is for IDE features such as LSP autocompletions.

13

But they are already unable to do so statically. It’s not like this PEP is introducing the ability to dynamically modify __all__. These problems already exist and I disagree with the notion that this feature should be artificially limited to cater to an unrelated limitation of type checkers.

If anything, I suspect that dynamic uses of export are easier for static tools to reason about than direct mutation of __all__. It would be a perfectly reasonable heuristic to, for example, treat all reachable export statements as being used. That’s much simpler to support than trying to keep track of direct mutation.

1 Like
14

Or other kind of static analysis tools. I can agree on that. As long as an export in a class or function body doesn’t mutate the module’s __all__, I don’t see why we should prevent them (except maybe to avoid confusion).

Not sure to understand this part. What do you mean by dynamic use of export? Why would it be simpler than understanding operations on __all__?

15

I’ve updated the PEP based on the review so far:

  • Resolved the export * open question in favor of matching import * exactly in response to Guido’s comment.
  • In response to Eneg, made the __all__-creation rules explicit (creates it if missing, copies it into a list if it exists but isn’t a list, otherwise uses it as is), and added an example showing __all__ += still works after an export statement.
  • In response to Joren, called out if typing.TYPE_CHECKING re-exports as an intended use case for stub-only packages.

I also noted that export is usable anywhere import is, with no restriction of its own out of consistency with import. If we want to add restrictions to export, we should add them to import as well, and that can be done in a separate proposal. That’s far too ambitious for this proposal.

4 Likes
16

This still leaves the issue I mentioned in https://discuss.python.org/t/pep-843-export-statement-for-dry-re-exports/108687/7:

So to clarify: Following the semantic implementation from the PEP, this f will be desugared as

def f():
    # from spam export ham
    from spam import ham
    __all__ = list(globals().get("__all__", [])) + ["ham"]

So it doesn’t actually update the module-level __all__ as you’d expect, and instead creates a useless __all__ within f’s scope.

2 Likes
17

Yes, I figured that it would be too confusing to add limitations to export that we don’t have with import.

Nice catch! I’ve updated the PEP so that it updates the global __all__ no matter where it is.

18

I’ve clairified the behaviour of this PEP given @jorenham’s recent comments and added an open issue due to Guido’s comments in PEP 844.

19

This actually feels wrong, especially since (unless the user also adds a global statement) there imported item is not in fact visible at the top level, and hence not importable.

It seems to me that export functionality (in PEPs 842, 843, 844) ought to be syntactically disallowed in non-global scopes. (Similar to how return and yield are only allowed inside function scopes.)

8 Likes
20

Oh, right, good point. Yes, that makes sense. I’ll update the PEP tomorrow.

1 Like