PEP 842: Module Exports

PEPs
42

But that (controlling wildcard imports) is what __all__ is for.

That’s not all it does in practice. IDEs, static type checkers and linters use it as a hint and often warn when you import symbols from a third-party module that are not included in __all__. This is precisely the reason why I started adding __all__ to my modules in anyio. And having an export soft keyword would let me avoid having to remember to add any new public-facing functions and classes to __all__ in public submodules.

1 Like
43

I have updated the PEP to now emit a RuntimeWarning instead of raising an ImportError upon accessing private attributes. I hope this resolves the philosophical concerns from before.

44

I did follow the idea thread, and I am following this thread too, but I found __export__ very counterintuitive. The main reason is that the status quo is to export everything, but it tries to prevent discourage the use of private names. I even got confused at first. This muddles the real reason someone could be trying to export a name when it is already public. Maybe __private__ would be easier to reason about.

45

Could you elaborate a little bit on this? I’m not sure I follow your reasoning.

Maybe, but I think it would be more difficult to use. When writing a module, I typically declare a few public APIs, and then add private ones as I evolve them. __private__ would require me to think about everything that I don’t want users to use, which I feel is harder than saying what I do want them to use.

Would it help to think about this as encouraging the use of public attributes rather than discouraging the use of private ones?

1 Like
46

Thanks for explaining the difference with __all__. I understand that this proposal tries to solve a real problem.

But the ergonomics are similar to those of __all__, and those are bad. It’s too easy to forget to add (or remove!) something to the list, and it’s distracting to have to update the export info in a totally different part of a file than the definition of the exported thing.

If we just cared about classes and functions, a more ergonomic approach would be an @export decorator. If we also care about exporting data or type aliases, I’d much rather look for a solution that adds a soft keyword named export (or private, for a better default).

EDIT: The export keyword idea is briefly mentioned in the PEP, and rejected with an argument of urgency. But Python has existed without this feature for over 35 years – is it really urgent? Remember the Zen of Python, which says “Now is better than never. Although never is often better than right now.”

13 Likes
47

I like your existing alternatives. I like that you called out module level __getattr__ interaction.

I do not think my ideas below combined are necessarily good or even internally consistent, I’m just trying to seed broader thoughts:

Q: What happens when and after I assign to a non-exported name in a module that declares exports? Can I assign to a name without a warning yet get a warning when i read it? Shouldn’t my assignment also warn?

RuntimeWarning appears to application users instead of the intended audience of code owners. Consider a DeprecationWarning or PendingDeprecationWarning style derivative (with its own appropriate non-deprecation name) so that it gets similar treatment and is far more likely to only surface to the code owners. (even those are not userproof, we still get complaints)

Something exports will be used for is actual deprecations… we’ve never added an easy non-decorator or non __getattr__ ability to create those for module attributes. So could we meaningfully add an official lightweight way to declare deprecations for names in a namespace at the same time? those require more metadata. At least reject the thought.

Q: Why should this only apply to module namespaces? It seems like this could apply to any namespace. People have written classes without using _ prefixes on attributes and methods that were not intended for public use. Even more often than in modules.

I genuinely think in the long run if we don’t use a new syntax for this, we’re missing the point.

A new dunder attribute container is easy to implement and bolt on to existing things (as you found while implementing this), but is unexciting. The time of creation time of use difference between a name binding and adding that name to a, never validated by the runtime, str container elsewhere is something I think we’d all like to get rid of? Syntax could do this.

I’d personally love it if __all__ could be deprecated in the long run with an established proven syntax as its replacement. (long time horizon change)

Random thought, pun intended:

from __future__ import private_namespace
DIE_ROLL = 4
public rand = lambda: DIE_ROLL

should we consider require a special way to enable the syntax like that dunder-future that also flips the globals() namespace for the file to behave as “explicitly” private for names not declared otherwise?

How to teach this is currently brief, it should cover the FAQs we anticipate learners asking of “why do these exist?”, “what’s the difference?”, “what should i use when?”, “can i just ignore these weird dunders?”… and how educators and materials should frame things so newbies don’t think they need to use magic dunder containers of stringified names. it’s a rather advanced feature in absense of syntax.

What does not being a public name even mean when it comes to classes and instances? That could quickly get sketchy. Probably best to consider those individually rather than by file edict. (ugh, that’s C++ struct vs class now isn’t it?)

Beware private/protected/public terminology C++ism and our name-mangling wars (__name, _name, name). IMNSHO we should probably stick to only supporting two behaviors instead of that old trifecta. protected and public are pythonic, python’s private is actively weird and non-dynamic - painful for white box testing.

If we go the long term syntax route, how do extension modules declare this and which names are public or not? Under the hood is there a __export__ supporting in container checking regardless even though we wouldn’t allow assigning to that dunder name? or would we allow it to unlock a transition period? if so we need to define what happens if new syntax is enabled in a file and it is assigned to. disallow that mix.

How do tests doing mocking and such interact with all of this without warning noise up the wazoo?

3 Likes
48

Thanks for the good questions and ideas. I’ll do my best to respond to both of you.

Yeah, I agree that the ergonomics aren’t super convenient. I wrote the proposal on the conservative side in hopes that it would go over better, but I think that was a mistake in hindsight.

Anyway, I do still think __export__ is important to have even if we add new builtins or new syntax. It’s a good backwards compatibility hook (similar to how lazy import has __lazy_modules__).

After briefly thinking about it, my main concern with export is the verbosity, and some of the corner cases that involve defining names. In particular:

  1. export reads well for simple variable cases (export hovercraft = eels), but gets ugly quickly. export lazy import spam is pretty bad, for example.
  2. Do we want export for everything that can define names, or just the important ones? For example, do we want an export with?

(To be clear: I think these are solvable problems; I just haven’t figured out how to solve them yet :slight_smile:)

I thought about this, but decided it wasn’t too important because it’s very uncommon for someone to accidentally assign to a private attribute. Maybe it’s worth doing for consistency, though?

Perhaps we should just add an ExportWarning class.

Hmm, this is an interesting idea. Any thoughts on how this would look in practice? I’m having a hard time envisioning something that isn’t just:

__metadata__ = {"name1": ..., "name2": ...}

which isn’t too pretty.

I hadn’t really thought about this because there’s no __all__ equivalent for classes, and because it’s a lot harder to implement (with modules we can nicely hook into module.__getattribute__, but with arbitrary classes we’d probably need to change object.__getattribute__). I’ll think more about this if we go for new syntax.

Seems reasonable to me to stick to just public and protected, but I’m not sure I follow why privates are “actively weird”.

I was envisioning all export syntax or whatever using __export__ under the hood. I don’t think we’d need to deprecate it because we want to keep it around for the backward compatibility reasons I cited above.

Suppressing the warning seems like enough, I think. It’ll be particularly easy if we add the new ExportWarning class that I suggested above.

2 Likes
49

I would expect that export import would only occur in specialised files like pandas/pandas/__init__.py at main · pandas-dev/pandas · GitHub and pandas/pandas/testing.py at main · pandas-dev/pandas · GitHub. In those places, export lazy import spam is fine.

At the same time there needs to be good support for importing from the correct place if importing from the wrong place is going to result in a warning. I suppose Python could provide something like `IndirectImportWarning(“spam is imported from foo, but should be imported from XXX.YYY.ZZZ directly”). Linters could pick it up from there, but users can also fix it themselves if they read that.

I think radically different default behavior is appropriate for attributes and for imports. It would be annoying if we had to explicitly mark all classes and functions and constants that don’t start with _ as exportable. And even the attributes that start with _ need to be accessible in testing. But I don’t think there is ever any need to import a module from the “wrong” place.

50

FYI, I believe the approach that numpy takes is to tuck all the typing stuff in numpy.typing. This keeps it tucked out of the way without needing any special machinery.

1 Like
51

One use case that may be worth considering. As previously noted, pip considers basically everything as internal/private. And we do have a history of users accessing pip’s internals. So it’s entirely reasonable for us to consider using this feature[1]. But pip-tools is an extremely popular tool that deliberately accesses pip’s internals - entirely deliberately and with the full knowledge that it’s unsupported. What should we do here?

Actually, here’s another question. In code structured with multiple modules, how do I say a name is “private for external users, but fine for use within my code”? Having to jump through hoops to use your own internal code feels ridiculous…

  1. I’d personally be against doing so, but that’s not relevant ↩︎

5 Likes
52

Following the “consenting adults” principle, I am perfectly fine if private members are only
marked by one leading underscore. However, there are cases where this seams to be unfeasible.

First, module imports (in the std lib and elsewhere) often use code like import enum instead of import enum as _enum. This is done for understandable reasons, but the consequence is that if I import re and then write re. in the PyREPL and press the TAB-key, the autocomplete suggests re.enum, although it is private under PEP 387. That is unfortunate.

Second, the leading-underscore-convention is not very helpful when one wants to convert a private name into a public name or vice versa since adding/removing the underscore breaks existing code. This is particularly ironic if the maintainer of a library wants to legitimatize usage of private names by removing the underscore and breaks the code in the way.

I think that none of these two points justifies such a strong paradigm shift, like actually preventing the access to private names. At least, there should be a simple possibility to access private attributes, like making them available in a _private sub-namespace (like accessing a private name xyz defined in re via re._private.xyz or even via re._xyz). This would also help in the case identified by Paul:

2 Likes
53

Yeah, I think __lazy_modules__ is going to be very helpful for adoption of lazy imports.

Putting lazy to one side, export import spam isn’t too pretty either. Two lines isn’t great, but this seems clearer:

import spam
export spam
5 Likes
54

I also imagine export import spam being rather confusing to new users, since ‘export’ and ‘import’ are complete opposites :wink:

2 Likes
55

I think switching to a warning here makes the proposal worse for the use I had pictured.

This doesn’t actually help in terms of enforcing the “intentional with any use of things not exported” boundary; it just throws up a warning that might not be seen by or otherwise reach the right person.

Without a clear positive acknowledgement of some sort from the author of the code touching the internal that they are doing it intentionally, this doesn’t help with the “any exposed detail is potentially relied upon” issue any better than a documented breaking change policy.

Besides this, I have a general distaste for runtime warnings that don’t reliably indicate a problem. It’s not necessarily a problem that someone is using an internal; it’s just something I, as a library author, would appreciate a little nudge at the language level to have them actually notice and consider it before doing so.

Even in the case of someone actively monitoring for warnings in their own code, warning filters that would accurately catch exactly the intended internal use without catching anything more are significantly harder for users to get correct than intentional access via __dict__ at specific use of an internal.

I think this change stops serving the purpose I had for it, so I have to retract any prior support for the proposal that was based on the prior semantics.

1 Like
56

Yeah, and I’ve tried this in my own packages, but it’s usually very annoying to do because you have to change files every time you want to add a type alias.

Agreed. My initial thought is to not emit a warning when accessing private attributes in modules that are in the same directory or in a subdirectory as the current module, but I feel like that will have some sharp edges (and doesn’t solve the pip-tools case).

I’ll have to think about this. Maybe we want a sys.set_private_attribute_whitelist(__name__, ["pip-tools"]) function.

That’s unfortunate to hear.

We have to make compromises and trade-offs when writing a PEP. There was so much negative feedback on the ImportError that my only options were to switch to a warning or to withdraw the proposal entirely. I care more about the runtime documentation aspect of this over the enforcement, so I went with the former.

Would it help if the PEP came with a hook that made it possible to raise an error instead? We could use an audit event, for example:

import sys

def hook(event, args):
    if event == "module.access_unexported":
        name = args[0]
        attr = args[1]
        if name == "my_module":
            raise ImportError(f"{attr!r} is not exported by {name!r}")

sys.addaudithook(hook)
57

That wouldn’t work. I don’t want something that pip would have to do to allow pip-tools to access internals. What they do isn’t supported by pip - it’s acknowledged and accepted, but we don’t intend to make it “official”.

As I said, personally I’d argue against pip adopting PEP 842. It’s strictly worse for us than the existing underscore convention. And given that pip is precisely the sort of project that looks like it would benefit from the PEP, that is a significant cause for concern to me.

That seems arbitrary. What about a structure like the following:

pip
    core
    helpers
        _internal

Here, _internal isn’t “in the same directory as or a subdirectory of” core, so core couldn’t use the internal helpers.

I don’t think it’s workable to assume any particular directory (or module) structure.

4 Likes
58

There’s also namespace packages to consider.

4 Likes
59

Ok, thanks for the context. Is the __dict__ workaround not suitable for pip-tools?

It concerns me too, but to be fair, I do think that pip is somewhat special here, given that it has no public API, but still wants its internals to be usable from an arbitrary downstream library.

I think this is a good datapoint nonetheless, so what would resolve your concerns? Just having a clear escape hatch for pip-tools?

1 Like
60

I’m still concerned about the clutter of the proposed solution.

There’s an existing pattern that feels less cluttered: put all implementation in _internal.py (or in a subpackage _internal), mark exportable things there using __all__, and in the toplevel package do from ._internal import *. This results in a clean toplevel namespace, and also solves the problem of legitimate access to internals: just go through the _internals submodule/package.

This is similar to how asyncio keeps its toplevel namespace clean, except the submodules don’t have.a leading underscore (so you can do things like from asyncio.events import blah, which should be considered a code smell).

5 Likes
61

I love this proposal because it seems to be the most Pythonic approach to access modifiers (or, if you prefer, a strong access suggestion, since it can still be bypassed) that I’ve seen so far.

I don’t have particularly strong feelings about the syntax. Whether this ends up being a decorator, a keyword, or a global dunder matters much less to me than the idea itself. That said, I’ll use __export__ throughout this comment because that’s what the PEP uses.

On underscores

It doesn’t take much looking around before you find code where _name does not mean “private”. The first example that comes to mind is typing.NamedTuple (and collections.namedtuple), which expose several public methods that break the convention, such as _make, _replace, and _asdict.

I think Python code is beautiful. I genuinely enjoy writing Python. What I don’t enjoy is making my code noisier just to communicate that something is internal, especially when it isn’t even a strong enough convention for tooling to support.

I’ve ended up adopting a simple policy in my own libraries: anything not listed in __all__ is considered private, and I don’t bother prefixing those names with underscores.

On tooling

As of writing this, only pyright with typeCheckingMode = "strict" warns about this snippet:

from functools import _make_key

_ = _make_key((), {}, typed=False)

functools prefixed the function with an underscore and omitted it from __all__, yet mypy, pyrefly, and zuban (all in strict mode) do not report an error. ty also accepts this snippet, although it does not currently offer a strict mode.

Autocomplete is no different. Pyright, pyrefly, and ty will all happily suggest _make_key when you type functools..

I don’t know why most type checkers and language servers have chosen not to warn about or hide _name, but if I had to guess, it’s because a simple rule would produce too many false positives.

Like everyone else here, I spend a lot of time in my IDE. Consequently, I care quite a bit about the IDE experience. It’s one of the reasons I embraced typing so heavily. I can stay in flow when my IDE is helping me. What breaks that flow is having to stop and consult documentation for something that could have been communicated through immediate feedback.

On whether this is Pythonic

I don’t see how this is unpythonic. You’re still free to poke at internals. This proposal just makes it obvious that you’re stepping outside the supported API.

Good APIs should be harder to misuse. The stdlib has plenty of examples where “read the docs” wasn’t enough. datetime.utcnow() documented that it returned naive objects since Python 3.0, yet it was still eventually deprecated because it was too easy to misuse. Documentation alone wasn’t sufficient. I don’t think anyone would argue that we should keep easy-to-misuse APIs because people should just read the docs and not mess up.

On how libraries solve this today

One thing I’ve found interesting while reading this thread is the idea that Python has always had unrestricted access to internals. I don’t think that’s ever really been true. Library authors have always been able to decide what is part of their public API and what isn’t. The only difference is that pure Python libraries have had to build that machinery themselves.

JAX has already been mentioned several times in this thread, so here are a few more examples:

The unfortunate part is that none of these approaches can ever provide a great developer experience. Every library ends up implementing its own flavor of access modifiers, and tooling can’t reasonably be expected to understand all of them. At best, you get runtime behavior. At worst, you end up duplicating your public API under if TYPE_CHECKING just to keep static analyzers happy.

On extension modules

Extension modules have complete control over what they expose and can trivially prevent users from reaching their internals. Everything is private by default, and they explicitly choose what to expose to Python.

So I don’t really buy the argument that Python has never had access modifiers. Extension modules have effectively had them for decades. It’s only pure Python libraries, or the pure Python portions of mixed libraries, that lack an equivalent way to express the same intent.

__export__ is just bringing a capability that extension modules have had for decades to pure Python as well. I don’t think anyone will agree with me if I said extension modules are unpythonic because they don’t allow me to poke their internals.

One thing I’d change

The only part of the proposal I’d change is the warning.

Personally, I strongly preferred the original ImportError (or a subclass of it). If the end result is only a warning, then I honestly think most of the runtime behavior could be dropped entirely, and this could instead become a typing PEP that teaches type checkers, IDEs, language servers, and other tooling to understand __export__.

On LLMs

Whether you see this as a positive or a negative is up to you, but __export__ would improve LLM-based workflows. If they accidentally reach for an internal API, immediate and actionable feedback is much more useful than silently generating code against an unsupported interface.

6 Likes