PEP 842: Module Exports

PEPs
21

I just want to echo this point. Based on @peterc and @NeilGirdhar’s comments, it sounds like people would like @private/@public as built-in functionality, and I think __export__ is good groundwork for that. If libraries like atpublic can take advantage of it to implement convenient decorators, then I think that will make a good argument for eventually adding them as builtins or adding new syntax for this someday.

2 Likes
22

For some of the programs I have written, and that have been found useful by others, I’ve either accessed some variables that were clearly identified as private by the library author or actually “monkeypatched” some library code, or both. I was able to do this because, while discouraged by convention, Python makes nonetheless this possible. As I read your PEP, I understand that such behavior could be prevented by a library author if your PEP were to be implemented. For that reason, I am extremely opposed to this idea.

2 Likes
23

It can be discouraged, not prevented. See this section. In short, the easiest way to do this is to access __dict__:

import module

module.__dict__['private']

I agree that internals are nice to have when you need them, which is why I’m not proposing some ironclad shield around private variables. The documentation/introspection aspect, as Barry said, is the more interesting part, and the attribute access enforcement is mostly a formality.

4 Likes
Module __export__ variables
24

from the pep I can’t really imagine where __export__ is different than __all__? What about some way to indicate that __all__ is strict instead? Pretty much what linters and pycharm will already complain about but as a runtime check instead. It would be rare, but I could see some use, especially for libraries.

__all__ = strict([
   "foo",
])
25

There’s a section in the PEP about that. The case I cited there was with static typing – there are often typing-specific APIs (such as type aliases) that you don’t necessarily want in __all__, but that are still valid for users.

1 Like
26

First things first: I also dislike the idea, and it my personal feeling is it disrupts one of the most important tenants in Python - but others had expressed that well.

Second: I don’t see where in the PEP it says what it actualy does. It says the motivations, what items should be put on exports, but I can’t find (it can be there, but I can’t easily find) what happens to the names that are not “exported”. There should be examples for that. Python has, in a straightforward way, always exposed whatever names are at a module toplevel upon importing.
The PEP asserts that all names conitnue visbible by going through the module’s __dict__. But what about normal, plain, dotted access?

If I do a Module b with

a = 23
__exports__ = []

Upon importing b, what do I get with import b; b.a - a NameError? What is the mechanism for that? (it should be on the PEP not just in the ref implementaion). Should I be allowed to do from B import a ? (If this ever goes forward PLEASE allow this: I am an adult, and I know what I do want from the module, and I don’t want to go through the __dict__.

1 Like
27

As shown in the PEP abstract (and later on), it would raise an ImportError. The Specification section answers your broader question of what it does with more detail (e.g. interaction with dir).

I don’t see this explicitly mentioned in the PEP text. I would personally be surprised if the behavior between mod.attr and from mod import attr differed as it seems to make it harder to teach.

2 Likes
28

I’ve been envious of the ECMAScript export syntax for a while. I don’t like maintaining __all__ by hand, as I feel it violates DRY. An explicit export soft keyword would handily solve that problem. A @public decorator might help in some cases, but not when re-exporting imports. The explicit export syntax would cover that case too. I don’t see any value in a separate __exports__ though.

3 Likes
29

Yeah, having thought about this PEP a bit more, I agree with people saying there’s not enough value here. I think the central problem with exporting is DRY.

I think this PEP could use some real life codebases illustrating exactly how they’re improved by it.

Edit: below is an old idea superseded by my PEP.

Summary

I realize I should probably start a new thread, but I’d prefer something like an __export__.py file that:

  • only supports import and from ... import directives
  • automatically sets __all__ to everything that was imported (but not locals, since those will get polluted
  • and maybe even deletes things not in __all__ as per this PEP (even though that seems unpopular)

This would make exporting DRY, and if you look at my linked project, it would reduce my exporting work by 50% plus the cost of keeping things synchronized. If I had something like this, I would probably never use __all__ anywhere and all of my __init__.py would be empty. This would supplant everything.

2 Likes
30

This is an amazing proposal. I think I differ from many by saying that I actually learned what the run-time use of __all__ was for, from this PEP! For me, the difference between it and the proposed __export__ is explained quite nicely.

I very much appreciate the fact that this solves the re-export of imported modules problem, without having to prefix every import with underscores.

I do have a slight agreement with @barry that using decorators or the like would be nice so that __export__ can be kept in sync automatically. However, this could be solved in a different way by the export (soft)keyword you proposed and I agree that should be deferred until after we see how users may use the functionality of __export__ from this PEP if it gets accepted.

1 Like
31

I’m going to be blunt here: I think an export keyword has very little hope at the moment:

  1. The overwhelming negative feedback here pushes me toward making the proposal smaller, not bigger. New syntax is about as “big” as you can get.
  2. export is going to lead to a lot of verbosity. I can personally tell from PEP 828 that people really don’t like triple keyword combos in Python, and export would add a bunch of those.

I would be very surprised if this proposal were to be accepted in its current state, let alone a proposal adding new syntax for this.

Anyways, I’m going to remove attribute access enforcement from the PEP and replace it with a warning instead. The __dir__ improves the documentation part (which is what I find important anyway), and I’m hoping that a warning provides similar expressiveness as an error without causing the “Python doesn’t have private variables” problem.

>>> import module
>>> module.unexported
<python-input-1>:1: RuntimeWarning: 'unexported' is not exported by 'module'
'whatever'
2 Likes
32

Could you give some examples of this? As a interested reader of the pep, I couldn’t really understand that rejected idea without a specific examples, which made me ask the question again in the first place.

33

A module might have a bunch of type aliases that it uses for whatever typing-related things it wants:

type Callback[T] = Callable[[str, int], T]

def add_callback[T](callback: Callback[T]) -> T:
    ...

Callback in this case is a very generic name and isn’t particularly useful if you’re just prototyping. The purpose of __all__ is to limit the amount of namespace pollution that occurs when doing a wildcard import, so the author only puts add_callback in the __all__.

34

I think it might be good to have more examples.

Because in my opinion, if you have a function in an API, I think it’s a mistake to have private types in a public API function. In the example you’re giving, you can’t wrap add_callback and type it in a way that won’t break without yourself copying (rather than importing) Callback. Copying is much more flimsy.

1 Like
35

The idea is that types aren’t necessarily “private”; they’re just not useful to include when doing a wildcard import.

1 Like
36

Do people really import modules from other modules (i.e., from requests import urllib3 rather than import urllib3)? This seems bizarre as well as being more verbose than the correct approach.

I’d really like to see strong evidence that this is a problem that needs to be solved. Specifically because it’s one of the main arguments for why using a preceding underscore to mean “private” is awkward, and I’m not convinced it’s actually a real problem.

2 Likes
37

But that (controlling wildcard imports) is what __all__ is for.

1 Like
38

Yes. I think it’s more of a problem when the user isn’t aware that an apparent submodule is actually a dependency. In your example, the user might do that if they didn’t realize that urllib3 is installable on its own and not just a feature inside requests.

That’s my point. You can’t always control namespace pollution and declare all public names simultaneously.

1 Like
39

But do you have actual public examples of this? It seems very speculative, and the sort of thing that’s easily fixed by education, without needing a language feature. And furthermore, why would requests bother going to the effort of hiding the import? All they have to do if someone raises an issue about it is say

urllib3 is a separate project, which is a dependency of requests. You should import it directly via import urllib3, not via the requests module.

That’s easy enough, helpful to the user, and educates them rather than making them feel like they broke some sort of rule. And anyway, in 99%[1] of cases, there will probably be no issue raised, because importing via requests will work just fine. So why make such a fuss about something that’s doing no harm to anyone in practice?

(Note: This is a separate question from the case of private names which are part of the project itself. The arguments for and against ways of marking those as private are different).

  1. Made up number, of course… ↩︎

1 Like
41

This kind of thing is hard to cite, because I think it’s particularly common among beginners. I have personally made this mistake when I was first learning Python, but I don’t have a GitHub URL to show you.

Python is supposed to be a language that’s user-friendly and easy to learn. A user shouldn’t have to make the mistake in the first place. Sure, it’s an easy fix, but it’s an avoidable problem. I think that the language telling you that you made a mistake is much nicer than having to go on an issue tracker or being told by a teacher/reviewer that your code is doing something suspicious.

1 Like