Making imports private by default in a future version of Python

Ideas
1

Summary

I’d like to propose a change that would, by default, make from mod import * ignore names that were themselves imported in mod. This is combined with a new syntax, from .submod export name which, when used in mod does expose name to from mod import *. In addition to affecting wildcard imports, this change is also intended to affect the semantics of type checkers: a normal from .submod import name should be treated as a non-public symbol while an export defines a public symbol.

The goals are:

  1. Make __all__ unnecessary in almost all cases
  2. Give type checkers (and humans) a clear, unambiguous signal about which imported names are part of a module’s API and which are not.

__all__ is today mostly needed in order to hide imported symbols and to be a signal to type checkers. Both these jobs will now be taken by dedicated syntax instead.

Since there has been some doubt in previous discussions about whether “private imports” can even be implemented reasonably, here is the mechanism up front: under a new from __future__ import, the import statement additionally records the name it binds in a module-level list, __imported_names__, and the fallback rule for from mod import * skips the names listed there. That’s the whole runtime change. The new from mod export name form imports without recording, i.e. it keeps today’s behavior.

Nothing else about imports changes: the name is still bound in the module namespace and still reachable as an attribute.

The proposal

Add a new future import:

from __future__ import private_imports

Enabling it does two things in that module:

  • The import statement becomes a private import: the bound name is recorded in __imported_names__. Names recorded there are excluded from from mod import *, and type checkers should treat it as private.
  • It enables a new statement, from mod export name, which behaves exactly like today’s from mod import name: the name is not recorded anywhere. from mod export * is allowed too, and is the natural replacement for wildcard re-exports in hub packages. Only symbols can be re-exported this way, not whole modules.

Without the future import, import keeps its current meaning and __imported_names__ is not created at all, so existing code is unaffected. Eventually, once the future import becomes the default, import would always be private and export would be the only way to re-export an imported name.

The semantics of __all__ are unaffected

The runtime change only affects the “fallback rule” of wildcard imports. Currently, from mod import * has two rules: if __all__ is defined, use exactly that; otherwise fall back to every name in the namespace that doesn’t start with an underscore. This proposal only touches the second part. The fallback becomes: every name that doesn’t start with an underscore and isn’t listed in __imported_names__.

__all__ keeps overriding everything, and no existing module that defines __all__ changes behaviour in any way.

Where this fits into Python’s existing semantics

PEP 8 already says that imported names should always be considered an implementation detail and that other modules must not rely on indirect access to them unless they are an explicitly documented part of the containing module’s API. The typing specification starts from the same default: imported symbols are considered private by default.

In practice, though, the language and the typing spec don’t agree with each other. At runtime, from mod import * will happily pick up every non-underscore name, imported or not, unless the author maintains __all__ by hand. Type checkers, on the other hand, have had to invent conventions like import X as X to figure out what is intended as a re-export. The proposal solves this inconsistency by making re-exports explicit with the export statement.

Example: “hub” packages before and after

The pattern most affected is the very common one where a package’s __init__.py collects the public API from private submodules.

I’m distinguishing here between Variant 1 and Variant 2 of the hub model, as defined here.

Variant 1: submodules define __all__, __init__.py uses wildcard imports

Today:

# pkg/_core.py
import os
from typing import Any

__all__ = ["Widget", "make_widget"]

class Widget: ...
def make_widget(name: str) -> Widget: ...

# pkg/_utils.py
import re

__all__ = ["slugify"]

def slugify(s: str) -> str: ...

# pkg/__init__.py
from . import config_init  # noqa: F401  (imported for its side effects)

from ._core import *
from ._utils import *

A small bug here is that config_init will still be exported. This is easy to miss. One fix is to import it as _config_init.

With this proposal:

# pkg/_core.py
from __future__ import private_imports
import os
from typing import Any

class Widget: ...
def make_widget(name: str) -> Widget: ...

# pkg/_utils.py
from __future__ import private_imports
import re

def slugify(s: str) -> str: ...

# pkg/__init__.py
from __future__ import private_imports
from . import config_init  # noqa: F401  (imported for its side effects)

from ._core export *
from ._utils export *

All three __all__ lists are gone. os, Any and re are private automatically, so the wildcard imports only pick up what was defined in the submodules, and pkg is private in __init__.py without anyone having to notice the problem.

Variant 2: __init__.py imports everything explicitly

Today:

# pkg/__init__.py
from . import config_init  # noqa: F401  (imported for its side effects)

from ._core import Widget, make_widget
from ._utils import slugify

__all__ = ["Widget", "make_widget", "slugify"]

With this proposal:

# pkg/__init__.py
from __future__ import private_imports
from . import config_init  # noqa: F401  (imported for its side effects)

from ._core export Widget, make_widget
from ._utils export slugify

There is now a single source of truth: the export statements are the import and the declaration of the public API.

Implementation sketch

import X would compile to the current IMPORT_NAME + STORE_NAME, plus an extra step along the lines of __imported_names__.append('X'). The export form compiles to today’s import and records nothing. __imported_names__ is created only in modules that enable the future import, which keeps the wildcard rule simple: if the attribute exists, filter by it; if it doesn’t, the module predates the feature and behaves exactly as today. (An earlier version of this idea recorded names unconditionally, but then the fallback rule would have to consult the future flags as well.)

Because the mechanism is a plain module attribute, the state is inspectable at runtime — unlike the import X as X convention — and in unusual situations it can be manipulated directly.

Can a future import enable new syntax?

PEP 236 says a future statement may change the syntax of the module it appears in, and Python 2 had several that did (with_statement, print_function, unicode_literals). But that isn’t true of CPython today. I checked on 3.12: the flag mechanism still works, but only if the flag is known before parsing starts.

>>> import __future__
>>> compile("1 <> 2", "<s>", "eval", __future__.barry_as_FLUFL.compiler_flag)   # works
<code object ...>
>>> compile("from __future__ import barry_as_FLUFL\nx = 1 <> 2\n", "<s>", "exec")
  File "<s>", line 2
SyntaxError: invalid syntax

The PEG parser parses the whole module first; future flags are extracted from the resulting AST and only affect the AST-to-bytecode stage. So barry_as_FLUFL is effectively dead as a future statement, and a new future import can change semantics (as annotations does) but not syntax.

Fortunately this proposal doesn’t need it to. export after a from ... clause is a syntax error today, so the grammar can accept from mod export name unconditionally as a soft keyword, with no ambiguity against any existing code. The future import then governs only the meaning of import, which is a pure semantic change of the kind CPython already supports. If we want to forbid export in modules that haven’t enabled private_imports, the compiler can reject it at the AST stage, where the future flags are available.

A corner case

Because recording happens at the import statement, this:

import X
class X: ...

ends up with X implicitly private, even though the final X is a locally defined class. I think this pattern is rare enough that it’s a price worth paying. If you do need it, __all__ still overrides everything, and an even simpler fix is import X as _X.

Compatibility across versions

One scenario that might or might not be worth thinking about: a library that wants to support both a Python version that doesn’t have the future import yet and a version where private imports have become the default (and can’t be turned off). Such a library can’t use the new export syntax (it wouldn’t parse on the old version), and import means different things on the two versions. For that specific case, the library can use import and then clear __imported_names__ of all entries manually. Or, just fall back to __all__.

Alternatives I considered

Just use __all__. It works, but it duplicates every public name and gives us two sources of truth.

A decorator from a library such as atpublic. This makes building __all__ painless, but type checkers don’t understand it (fixable with a typing PEP, admittedly). But for me personally, I would prefer dedicated syntax instead of a decorator.

A general export keyword, i.e. export class C: ..., export def f(): ..., export TWO = 2, export type L[T] = list[T], and from mod export Foo. This covers the cases decorators can’t, but it has two problems. First, Python already has a way to mark names public or private (the leading underscore), and this adds a second, competing one. What’s proposed here keeps the underscore convention.

Treat all imports as private by convention and use import X as X to mark re-exports. This is roughly what type checkers do today. It works, but it feels like a workaround that inserts new semantics into existing syntax: there’s no logical reason why import X as X should mean “public”. And it isn’t inspectable at runtime, because import X and import X as X have exactly the same effect.

Treat all imports as public by convention and rename internal imports with a leading underscore (import os as _os). Too verbose and too ugly to be practical across a whole codebase.

Open questions

  • Is private_imports the right name for the future import?
  • Is writing __imported_names__ directly an acceptable escape hatch for libraries spanning the transition, or should there be something friendlier?
  • Is accepting export as a soft keyword unconditionally, and rejecting it at compile time without the future import, the right way around the parser limitation described above?
6 Likes
2

There are multiple PEPs in discussion to address this issue.

1 Like
3

I’ve been concerning myself with the cost of imports and I have wondered about the cost of having something imported in multiple namespaces like import mod and from mod import this in the same script.

I thought wildcard imports were considered one of the seven deadly sins and should be avoided at all cost, I guess that has changed.

It would be nice to make the single underscore prefixed functions and methods that don’t show up in help() not show up in dir() as well..

4

Almost negligible. Once a module has been imported for the first time, it’s stored in sys.modules, which means that import mod a second time is just mod = sys.modules["mod"] which is very fast. Similarly, a from-import will put the entire module into the module cache (which means that no, you don’t save anything by doing a from-import, it still imports the full module), and then copies the specified attributes into your namespace.

There aren’t really “deadly sins” in Python. Every decision has consequences. One of the big downsides of star imports is that you can’t easily see where something came from; and especially, doing TWO star imports in a single module can lead to confusion. But even that has its uses (mostly with module merging, eg if you have a top-level package that goes from .server import *; from .client import *; from .errors import * etc, making the entire namespace appear flat to callers.

1 Like
5

This looks like a reasonable change for default from module import * behavior. Possibly consider phrasing this topic just as improving import * default semantics rather than using the much bigger and controversial term “private”.

I think it needs a different name. With private_imports I would expect that imported names will not be reachable via the module namespace from the outside, i.e. for

# mymodule.py
from future import private_imports

from .utils import helper

import mymodule

mymodule.helper  # should not be accessible

I think your proposal is rather a module_wildcard_imports_disregard_imported_names or exclude_imports_from_start_imports. Name up to bikeshedding, but IMHO it’s important to note that this only affects wildcard imports of the module. It is not changing anything about the the API surface of the module itself.

6

After reading again, I’m not clear how export works and should be used. On the one hand, you write

On the other hand, your example uses export in the hub.

Following your pkg / pkg/_core example. There are two import levels

  • a user imports pkg: According to your quote, The export in pkg/__init__.py would only affect from pkg import *, that’s quite an edge case for an end user and generally discouraged.
  • the hub needs to import the submoduls. They could in general continue to use from ._core import Widget, make_widget and leave out the __all__ as soon as all the submodules do from __future__ import private_imports. It seems no export keyword is needed in this cases.

Are you maybe mixing two ideas?

  1. changing import * behavior to exclude imports. This would largely work without the need for a export keyword.
  2. Replacing import * in the hub by a new export keyword. This would work without the need to modify import * behavior.
7

It feels a bit weird seeing this thread when

exists.

But the OP of that thread very explicitly didn’t want to take the step you’re proposing here, of modifying what ‘normal’ import statements mean, which is where almost all of the value of even having this new syntax comes in for me.

So yes, I do consider this an improvement over PEP 843.

I’m not a 100% sure about the details proposed here, but it’s close enough that I think it’ll get to the right place in the normal process (with lots of clever people thinking about it :wink: ). Then again, I also thought that about PEP 843