Add the export keyword to Python

Ideas
1

Let’s discuss whether it makes sense to add an export keyword and possibly the syntax for from ... export .... I am okay with making only a portion of this proposal into a PEP and I would like to write the PEP and implementation of export for CPython if we decide to accept any part of it.

Rationale

The existing method of defining __all__ in Python modules involves manually listing all public attributes and functions which a module exposes. This method is a bit prone to errors and not intuitive for newcomers.

The proposed export keyword will make this process more straightforward. This keyword aims to improve readability and maintainability of Python modules, especially libraries and their __init__.py files. It seems to make sense to take a convention that we have been following for decades and turn it into syntax.

Specification

The export keyword can only be used on the toplevel of the module. It can be used in three different forms:

  1. export <name>: This would add the name to the module’s __all__ list. Similar to import <module>.

# Inside my_module.py

def func1(): ...

def func2(): ...

export func1, func2

This would be equivalent to:


# Inside my_module.py

def func1(): ...

def func2(): ...

__all__ = ['func1', 'func2']
  1. from <module> import <name> would import the name into the current module and add them to the current module’s __all__ list.

# Inside __init__.py

from .sub_module export func1 as func, func2

This would be equivalent to:


# Inside __init__.py

from module import func1 as func, func2

__all__ = ["func", "func2"]
  1. from <module> export *: This would import all names from the module and add them to the current module’s __all__ list.

# Inside __init__.py

from module1 export *

from module2 export *

This would be equivalent to:


# Inside __init__.py

from module1 import *

from module2 import *

__all__ = list(set(sub_module1.__all__ + sub_module2.__all__))

References

2 Likes
2

This thread already has a number of excellent posts in it, notably referring to the atpublic package. Can you explain how these are insufficient?

3

Many criticisms of atpublic refer to its syntax which would be solved by introducing a keyword.

My main reason for having a separate syntax is to formalize the __all__ syntax as it still feels a little hacky. If you take a look at references, many developers of popular libraries have also been yearning for it. Hopefully, they’ll also come here and provide their reasons for wanting it.

4

Syntax is very expensive, particularly the creation of a new keyword. Your proposal would mean that anything using export as a function/variable name would instantly stop working. That takes a LOT of justification.

Personally, I don’t use __all__ - if something is private, I name it with a leading underscore. If I were to use __all__, I would most likely either roll my own decorator to populate it, or use something like atpublic. Other than JavaScript envy, what is the reason for making this an actual keyword instead of simply a name?

(For the record, I don’t think an @export decorator would be all that valuable either, but the bar for its creation is a lot lower than for a new keyword.)

1 Like
5

if something is private, I name it with a leading underscore

I agree and I do the same thing in regular modules. However, in __init__.py files, you have no option of doing that. You gotta import a bunch of stuff and then put it into __all__.

what is the reason for making this an actual keyword instead of simply a name

No reasons beyond the ones I have already listed, at least from my side :slight_smile:

6

I think this is a step in the right direction, but it still doesn’t solve all of the problems with the current exporting behavior.

In particular, many libraries try to control not only what gets exported, but what gets hidden. And they do that by moving all of their source files to private folder (e.g., _src) and then building a directory tree for just their exports with files named __init__.py. These files only contain from... import statements and __all__ lists.

This proposal merges the __all__ lists into the import statements, which is definitely a step in the right direction. However, there is still the hassle of creating a parallel structure just so that you can have a clean library interface. In my opinion, if we’re goin to add keywords, it would be nice to resolve all of the problems with exporting.

Just off the top of my head, how about creating __export__.py files whose attributes are the only thing that gets exported from a given directory?

1 Like
7

Strangely, I have never seen such libraries. Would you post a few examples, please? I’d love to take a look.

Just off the top of my head, how about creating __export__.py files whose attributes are the only thing that gets exported from a given directory?

Your proposal is pretty interesting but I’d stick to several small proposals instead of a single large one as PEP 1 instructs. Though it’s still valuable to discuss it here to maybe think of a roadmap and build a picture of where we want to go with the new export syntax.

8

A less elegant style is done by libraries like NumPy, which go through the trouble of deleting things from the interface.

I don’t see how I could make my proposal smaller? I’m suggesting __export__.py files as an alternative to the export keyword–an alternative that solves everything you wanted in your proposal and also allows creating concise public interfaces.

1 Like
9

We could almost certainly make export a soft keyword, meaning that no current code would break. We had something similar with PEP 695, which made type into a soft keyword.

Of course, even if it is backward compatible, the bar for new syntax is high.

10

As a soft keyword, in what contexts would it be valid? Although I must admit, I had assumed that export spam = "ham" was part of the proposal, and on rereading, that isn’t the case, so that might make it easier. But it’d be tricky to find valid uses that won’t get in the way somewhere. I guess it’d be okay as an attribute?

11

I’m pretty sure we could make from export export export work.

Note that export x as "y" would be fine too; that’s very close to the syntax for the type soft keyword.

12

Hmm, okay. In any case, the bar for new syntax is still very high.

If the problem here is __init__.py needing to import and reexport, maybe we can have a solution specific to that? I mean, looking at something like asyncio/__init__.py, it’s pretty clear that there’s a problem here, but I’m not sure what the solution is.

13

To be honest I’m not sure why asyncio/__init__.py is going to such great lengths to prevent users from accessing…asyncio.sys?

And also all the stuff from asyncio.taskgroups (not included in the __all__ = ... block) but I’m guessing that’s not on purpose? [1]

To be less snarky about it and more on-topic: I don’t use this pattern a whole lot and don’t find it useful when I encounter it, because I almost never use import * anyway. I personally wouldn’t want to encourage more of it, but that’s just me.

  1. I opened an issue for this ↩︎

14

Yes, there’s good reason for that. It might not be a big deal with sys since that’s very well known, but there have been other cases where people have used an imported module from some other module, and then been bitten by the fact that it stopped being imported.

More generally, the removal of any public attribute could be seen as a breach of backward compatibility, so excluding them in the first place prevents this.

Agreed, but __all__ is also good documentation as to what’s public. Though personally, I prefer to just underscore-prefix the ones that aren’t.

So, here’s a thought. Can we get a function that lists all public symbols in one or more modules? If __all__ exists, it returns that, otherwise it returns all symbols that aren’t prefixed with an underscore. That removes the part of the problem where other modules have to have __all__ or it doesn’t work (as is the case in asyncio).

1 Like
15

That removes the part of the problem where other modules have to have __all__ or it doesn’t work

Could you help me understand how your proposal solves that problem? Would users use that function? I am a bit confused as to how and where that would be used.

16

In my opinion, the problems with the current system are that:

  • producing an interface means specifying redundant information in
    • module __all__s,
    • __init__.py’s __all__, and
    • __init__.py’s import statements.
  • keeping the interface concise requires either
    • deleting unwanted symbols, or
    • having a clean parallel directory structure of modules (as described above).
  • also, keeping the interface concise means
    • being conscious that from .abc import def will create the symbol abc as well as def.

It’s a hard problem to solve since there are essentially two consumers: endogenous imports wherein the library itself wants to import its own symbols (from ..xyz import symbol) and exogenous imports (from some_library import symbol).

Currently, the external interface is specified by __init__.py’s __all__. In an ideal world, only those symbols would be available to exogenous imports. My suggestion is that we come with simple ways of ensuring this. By simple, I mean having none of the above issues (redundant information, using del, having a parallel structure, worrying about relative imports polluting the interface).

17

How about this, to piggy back on the original idea. Every module starts with an __exports__ empty list. We allow only these two syntaxes from the proposal:

But instead of populating __all__, they populate __exports__. When an exogenous import of the library happens, if it sees __exports__, it replaces __all__ with __exports__, and deletes every symbol that’s not in __exports__.

The deletion behaviour solves the three conciseness problems (having to delete symbols, having to create a parallel structure, worrying about pollution by relative imports). Thus, this new syntax proposal would not just be a bit of syntactic sugar for producing __all__. It would be a gateway to producing concise external interfaces.

I agree with @jamestwebber that the * form should be discouraged here. If the goal is to produce concise interfaces, it probably shouldn’t be provided. (After all, there is no redundancy problem since you typically either have an externally-imported module that exports its symbols or an externally-imported package that exports the symbols in the module.)

2 Likes
18

I am quite fine with losing “*” as I never use it and have linters prohibiting it in every project of mine.

I also like your proposal. It is a bit more complex than the original one but solves a lot more actual issues.

1 Like
19

It feels like there’s an assumption here that packages re-export symbols from sub-modules in the top-level __init__.py. I’m not sure why that’s a common pattern - in my experience, package structure is part of a package’s API and importing names from a subpackage is normal and expected.

If you want to export a carefully selected set of names from the top-level package, then that’s fine, but I don’t think that

from .core import main_entry_point
__all__ = ["main_entry_point"]

is that big of an issue. And even the assignment to __all__ is optional, as it only affects import * usage, which shouldn’t be the recommended way of accessing the package in the first place.

I’m not saying that it’s not a little clumsy, but “fixing a small clumsiness in a relatively uncommon pattern” isn’t anything close to sufficiently beneficial to justify new syntax.

Really? asyncio/__init__.py looks pretty clean to me. All this proposal would do is remove the assignment to __all__. If that warrants new syntax, your tolerance for a bit of redundancy is a lot lower than mine is…

4 Likes
20

No, I don’t think it warrants new syntax; I was thinking of a convenience function, possibly in importlib, but not syntax.

1 Like