I would go even one step further and answer two questions:
each may have their use cases, and theoretically we could even have both.
.
Okay, as a compromise, how about we lock the feature behind a __future__ import?
For example:
from __future__ import private_names
export def hello():
...
private_names will make everything private by default, and export would only be allowed in modules that use the future import. I think that resolves things adequately, as modules that want to opt in to privacy are explicitly marked at the top of the file.
I almost suggested that, but it does come with an implication that at some time in the future, private will be the default. And I think that’s going to be a much harder sell[1] than simply having it be an opt-in feature.
We don’t really have an equivalent of __future__ for “module-level settings”.
At the very least, I’m strongly against private ever being the default ↩︎
Sorry for the omnibus reply. The PEP has come a log way! My major concerns are at the bottom. First some nits.
I feel that the previous discussion thread should be linked from this thread for future reference, so here it is: PEP 842: Module Exports.
A future import is supposed to be a temporary measure; at some point the future behavior will be the default, otherwise a future import is not appropriate (despite an existing a counterexample). However, we could use some other explicit marker if we feel a marker is necessary. (Random idea: we could require setting __export__ = [] before export is used.)
I fundamentally disagree ith Paul (and some others) that it is a “long-established principle” that everything in a module is public. That was simply the default behavior, stemming from the original implementation – when you have no way to express what’s public and what isn’t, the only sensible default is to make everything public. There is no “God-given right” to be able to open up a module at runtime. (You can read the source on GitHub etc. though!)
We all agree that there is established prior art whereby large, long-living projects have a need to clearly demarcate their public API, so users don’t get surprised when private stuff changes in a future release. Many such projects have their own custom implementation.
This PEP exists to offer such projects a standardized implementation that is more ergonomic than current approaches. I think the PEP should focus on the needs of greenfield projects rather than bend backwards on how to introduce this into an existing large project – each large project has its own ways of handling change, and we don’t want to tell them how to do. I an even see that some established projects won’t adopt export, and that’s fine.
I have a bikeshed with export <assignment>, but that can be discussed later.
My main gripes:
Nobody wants default-private for modules that don’t use export at all. The debate is about whether the presence of at least one export is sufficient notice or whether we want an additional marker.
I could go either way; but if we agree that we need an additional marker, my proposal is __export__ = [] somewhere at the top. (EDIT: Anywhere before the first use of export.) That would mean that if __export__ isn’t set, using export will fail. (It shouldn’t silently do nothing.)
I’m not sure if this would be too restrictive, but can we make it something known at compile time so a SyntaxError can be thrown? I know it’s not “best practise” but if your first export comes after an executed block of code in a module, you’ll want a compile-time error to know to fix it instead of waiting for the module to complete. Or, if there’s export statements behind an if-block (if that’s allowed), you’d also want a compiler error.
I personally think it’d be fine to say “this will never be the default” and still put it in __future__. Or, just say that “this will be default in Python 4”, and then also assert that there will never be a Python 4 
.
But either way, if we don’t want to add a future statement for this, then we should consider adding a new method for declaring compiler flags, as I doubt this will be the last time this comes up.
Paul brought this problem up on the other thread, and I had a similar idea. He noted that some libraries keep vendored dependencies under library._vendored or similar; in that case, library would have access to the private attributes of vendored dependencies, which we don’t want.
My best idea for this so far is to make this the job of the __init__.py file in some way. For example:
library/_util.py declares utility functions as public (or doesn’t use export at all), which allows library/main.py to import from it.library/__init__.py somehow marks _util as private, so it can’t be imported from outside the module.Why do we need that? Lots of things throw runtime errors when reached. Users learn about these when they test their code. Linters can diagnose these easily as well.
Again, why do we need a compiler flag? The export syntax doesn’t change. It’s just a runtime error to use it when no __export__ global exists. Just what would happen under your previous proposal if __export__ wasn’t iterable.
Deciding whether a module is private-by-default once up-font simplifies things. It gives the freedom to set up internal structures completely different (it depends on the implementation whether that’s reasonable, but is an option for a compile-time decision). It is simpler for linters to detect and for people to reason about. Doing this at runtime would in principle allow all kind of edge cases like the one shown in PEP 842: Module Exports (new revision) - #20 by pf_moore, or or even non-statically defined behavior “private only on mondays”, or more realistically “private only on linux”, which cannot be resolved by a linter.
Since this is a feature primarily for large libraries, making private-by-default a static decision seems reasonable. A compile-time flag is one way to achieve this. Lighter ways would be conventions that __export__ = [] must be put at the top of the file and must not be used in code logic or methods.
I find the list of real world cases the most convincing part of PEP 842. I find the additional syntax and the possible overlap with __all__ the least convincing. Perhaps better use of __all__ can reduce or remove the problems in the real world cases. If that fails the case for the PEP becomes stronger.
Here’s a contribution along these lines. Consider creating constants and putting them in __all__.
For doing this Barry Warsaw’s atpublic provides this overload of the @public decorator:
public(SEVEN=7)
The PEP suggests:
public SEVEN = 7
I suggest use:
@call
@public
def SEVEN():
return 7
and similarly:
@call
@private
def ULTIMATE_ANSWER():
return 2 * 3 * 7
where public and private are from atpublic and:
def call(callable):
return callable()
I hope this helps.
The updated PEP is significantly better IMO. However, I agree with @guido that the warning should be replaced with an ExportError.
The need to distinguish module-level public and private API and the shortcomings of the current situation are laid out very clearly. 
I’m missing the discussion and mentioning in “Rejected Ideas” of the inverse solution:
__private_dict__ (name up to bikeshedding)__dict__ lookup fails, additionally try to look up in __private_dict__ before falling back to module __getattr__.private that adds the symbol to __private_dict__ instead of __dict__.Well implementation-wise it’s not 100% inverse because __export__ is a “seqence” of str acting as a filter whereas __private_dict__ would be a mapping from str to the actual objects so that we partition the namespace content in __dict__ + __private_dict__.
This has two effects:
__all__, __dir__ behavior (and user defined __dir__), lookup of dunder methods are affected.I just think it’s the most natural solution for something that affects syntax. I’m fine with other options, but I think we should have a good reason for choosing something else, beyond just “we don’t have a good place to put non-future compiler flags”.
Barry has his own competing proposal (PEP 844).
My proposal is not a syntax modifier. I want to treat an export without preceding __export__ = [] as a runtime error. It’s of the same nature as using a name that you forgot to import. Requiring __export__ = [] serves as a reminder to future readers that this module uses default-private, and reminds the author that using export requires using it for all (toplevel) public names.
I also had an old response in my edit buffer:
That feels like a pretty uncommon edge case – vendoring, like monkey-patching, has a bit of a code smell. It’s not ideal, but not the end of the world either. Also, a project that vendors a dependency can already mess with that dependency in any way it wants to – in fact that’s sometimes the reason to vendor. (I understand pip has other reasons too.)
What form would importing _util.foo in (1) take? If you write from library._util import foo that would be blocked by (2). We could change the rules for from . import (etc.), but many packages seem to shy away from relative imports.
Your solution is also very verbose. It should not require extra effort to use an internal symbol from another part of the library. (Except maybe for very large libraries, where subparts are developed separately. We should have a solution that scales to such use cases, but requires no ceremony in the normal case.
Very common in the i18n world.
Yeah, I get that; I just think it feels weird to have syntax that relies on an otherwise-unused variable declaration somewhere at the top of the file.
I don’t really have any counterarguments beyond gut feeling, so I’m open to convincing. Do others agree that __export__ = [] at the top of the file is the right approach?
I find your arguments convincing for allowing members of the same package to access private names, but I’m not too sure that it should be the default behavior, as I think there’s still some benefit to keeping some names completely private to a single file. Maybe we could add a C#-inspired internal keyword for this?
I share your apprehension. How about making so that all of your export directives instead populate __exported__, and then at the bottom of the file, if you want to limit exports, you can type __export__ = __exported__. I guess you would be free not to do it if you wanted to opt out of that behaviour? Or you can configure a linter to remind you to add that directive when you have exports.
Underscore-names are commonly used for this in many projects, and this convention is already enforced by linters.
Seems like you’re trying to use your export directive to both control what library users can import, and what the library itself can import. I think it’s a bad idea for one directive to have two disparate goals.
But perhaps this was your goal all along? So it isn’t so much an “exporting” idea as it is a module privacy idea? You want to be replace the underscore-names convention with enofrced module-level privacy using a keyword?
No, I care more about the exporting. Library users shouldn’t have access to private names. It’s just that the library itself should have access to private names if it needs to.
I agree with this. My reservations about this proposal are almost all based on how I think it would work for pip, and pip is very much a weird special case, for reasons which don’t apply to other projects. I do think my reservations are broadly valid, but I’d definitely appreciate any feedback/support from people with experience on more “normal” projects.
On the other hand, vendoring isn’t the only way to have a library visible nested within another one:
# foo/__init__.py
import requests
# foo/bar.py
import foo.requests as requests
# Do I now have access to private names in `requests`?
Yes, this is again a code smell, and is clearly an explicit attempt to bypass the protections of PEP 842, so I’m not suggesting we have to deal with this case explicitly. But I am concerned that what looked like a “simple” rule has a bunch of weird corner cases. That also feels like a (design?) smell.