For some background, I was looking at PRs earlier today, and I saw this comment by Hugo about underscored imports in argparse:
I see the import copy as _copy pattern in older CPython code, but don’t really know the reason; is it to avoid argparse.copy appearing like a public API?
This got me thinking about solutions beyond prefixing, because I figured this problem applied to most modules, and prefixing isn’t exactly a nice solution for maintainers. I came up with the idea for a module __export__ variable, similar to __all__:
__export__ = ["Public"]
class _Private:
pass
class Public:
pass
The idea is that if a name isn’t in __export__, then it won’t be included in the module’s dir() (and thus not in its help()), and attempting to access it will raise an ImportError. (If you still want to access something for monkeypatching or something like that, unexported attributes are still accessible through __dict__.)
The nice thing about this is that it mixes with __all__ really nicely:
__all__ = ["Public"]
__export__ = __all__
class _Private:
pass
class Public:
pass
Out of curiosity, I threw together a reference implementation for this, and I think it’s simple enough for CPython. You can play around with it if you’d like.
What do others think? Would this make your life easier?
When would __export__ be present and NOT equal to __all__? Does it ever make sense to have something in __export__ but not in __all__? What about the other way around?
(Minor quibble btw: I’m really not a fan of unparenthesized tuples at top level, they kinda fail the “screen grit” check. Could you make that into a list or put it in parens please?)
I go by the philosophy that static types are descriptive, not prescriptive in Python due to the gaps in expression[1] I leave type alises and typing only symbols out of __all__, but users may still benefit from importing them, and I wouldn’t intend on excluding that posibility, even if I don’t want them showing up as an import recommendation from tools that look at __all__
and therefore not part of the stability of code, the static types are always the current best expression within static typing and subject to change. ↩︎
I’ve always wanted real visibility modifiers so sign me up. I already have __all__ defined in all of my projects and I would prefer if __export__ implied __all__ allowing me to just migrate with find and replace. This will introduce a discrepancy though: modules can now essentially private functions and classes in their scope[1] but classes cannot private methods in their scope and will have to continue with underscores. I don’t think that’s a deal breaker though.
I’m curious about how this interacts with submodules and namespace modules.
I like that idea, although one serious downside of relying on it is the potential confusion across versions. Suppose that Python 3.17 introduces __export__ and you find-and-replace as @monarch is suggesting. That code, run on Python 3.16, will behave oddly, since __export__ is just a meaningless attribute, and now __all__ isn’t defined.
Another way to look at it would be that this isn’t actually a separate list, it’s instead a change of the behaviour of __all__ itself. Hmm. Actually, thinking about that… really, this is just a change to __getattribute__ to restrict to those keys that are found in __all__. Currently we can’t define __getattribute__ the way we can with __getattr__, so this can’t simply be written as from exclusive import __getattribute__, but the behaviour’s basically that, right?
The idea is that an explicit __all__ will simply turn into a legacy way to define what a module exposes. Modules can define both __all__ and __export__ without consequences, and when people want to drop support for older versions, they can just remove __all__.
I thought about that, but I have a feeling that this will eventually make things more complicated. I think Michael makes a good point that static typing constructs aren’t always supposed to be in __all__ while still being accessible outside the module.
I think I’d be fine with __export__ being defined, and __all__ not being defined implying __all__ = __export__, that direction of implication doesn’t limit users. As long as dropping __all__ was optional and modules with a reason to have them differ retain the capability to do so, I don’t see it causing any issues.
It might be worth enforcing that __all__ is a subset of __export__, but I think leaving that to linters is probably fine if the intent is that most modules won’t need to define them seperately.
This is a great idea on its own. It would eliminate the common pattern of having a parallel library tree just to control what gets exported.
Can you go a bit farther than the decorator and add an export soft keyword?
export class Public:
pass
export pi = 3.14
export from .some_module import * # Pulls in everything that was exported there
export import .some_module as some_name
Maybe later. There’s a really high bar for new syntax, and I think it would be helpful to see what libraries can do with __export__ before we add an export keyword. There’s also a bit of verbosity there; export lazy import would be Python’s first triple keyword trifecta.
I tend to use dir() a lot to investigate module/class internals (rather than finding the source code and reading that) so this would be good for making module contents more manageable.
I wonder if raising an ImportError is a bit much?
I get that it’s meant to be private, but if the name isn’t listed or autocompleted anywhere, then I don’t see why raising an error is necessary on access, since you can always work around it by accessing the __dict__ . I use monkey-patching a lot and can imagine it just becoming a hassle to access the __dict__ rather than being useful.
It’s also likely to add a bunch of ImportErrors to people’s code where they’ve used internals which were previously public, but not included in __export__ when it’s introduced.
Many libraries go out of their way to prevent you from accessing their internals, so giving them an easy way to do that is a benefit for them at the cost of monkey-patchers who have to do a bit more work.
I think it’s a losing battle to want to make it harder for libraries to block internal access. The libraries need do that because it leads to support nightmares. See, for example, Jax’s
