Pretty much what atpublic (@public) solves! 
2 Likes
It happens - rarely IME - but it’s usually accidental, and it’s definitely a code smell. ISTR that linters can find that kind of stuff these days, but I could be mistaken. It just doesn’t seem like an important enough problem to me to warrant a language solution.
1 Like
mypy --no-implicit-reexport catches this. In my experience, it flags some real issues in large codebases where people accidentally rely on import os.errno-style things, but there’s some edge cases to get right.
5 Likes
Pretty much what atpublic (
@public) solves!
No, it doesn’t help when re-exporting imports from my private modules. I practice this in pretty much all of my projects. The current form recognized by static type checkers is from X import Y as Y which I’d rather write as export from X import Y.
2 Likes
How would you write such a module __getattr__? It only fires for undefined attributes, not defined ones.
I think that having to explicitly state which parts of a module are not exportable is inherently non-Pythonic. I would be in favor of having the existing behaviour remain but adding functionality to mark something as non-exportable. Perhaps a __private__ list that includes only the things that can’t be exported.
I can see this being a major foot gun for people, especially in a transition phase. Either that or it not being used.
Sorry, yes, it’d be multiple dels. But that’s still a pretty small amount of code.
I’m guessing too, because I haven’t written the new revision yet 
. I need to come up with a way to solve this in the new design.
Going back to the module structure idea from earlier, would it work for exports to only apply when there’s no common parent package? For example:
pip
core
helpers
_internal
something_else
core could access private things from helpers because pip is a parent module for both of them, but something_else could not.
There was a level of implicit “already have a facade of some sort” bit going on with that line of thought. If you really wanted to do it without a pre-existing facade, it’s still possible, though ugly, and involves setting up all definitions of the module on first access within __getattr__, while keeping the definitions of anything private within a closure of some sort and not written back directly to the module object. This would end up looking somewhat like lazy module evaluation in the process, so it’s not without some of the same tradeoffs that inherently has on top of the unintuitive module definitions required.
There’s also already an auditing event for imports, and as long as you install the hook after the library’s own internal self-imports have happened, you don’t end up erroring for private things used in other files within the same library.
But to be clear, it’s more that the warning exists than that the access is no longer checked by default in the change that sinks my willingness to use this.
I said it before, but I don’t believe in issuing unreliable runtime warnings. There’s no way from the site of the warning with the new semantics to know if a given internal use is intentional, so if going that route, the warning always has to be issued, and the user now has to write a warning filter of some sort (including potentially just supressing the warning in a context, though that has issues currently inherent to the design of the context…).
I would even be fine with __export__ just existing as a standardized module documentation attribute, maybe coming along with a change in behavior for help and dir, but with no changes to import or attribute access.
The language defining __export__ to mean a library self-describing it’s own public api, not just wildcard import behavior, even if not enforced by anything in the language is still better than users needing to go find what a project considers public and not in policy documentation.
There’s then the ergonomic side for making this something library authors will want to use rather than the same ergonomics as __all__. I think adding syntax that allows the same definition, while adding the name to __export__ works here? At least now that syntax is actually being considered in the discussion. A decorator is fine too, but won’t work for everything a library might export.
You can use the function call form in that case:
from X import Y
public(Y=Y)
Not quite DRY, but the best we can do.
I decided to actually write up an example of how one could do it
__all__ = []
def __dir__():
return __all__
def closure():
re = __import__("re", globals(), locals(), [], 0)
def SomethingPrivateNotUnderScored():
...
class _ActuallyPublic:
...
class Foo:
...
# after the indenting of most of the module:
public_api = {
"Foo": Foo,
"_ActuallyPublic": _ActuallyPublic,
}
full_api = locals().copy()
for obj in public_api.values():
if qn := getattr(obj, "__qualname__"):
obj.__qualname__ = qn.removeprefix("closure.<locals>.")
__all__.extend(sorted(public_api))
def __getattr__(name):
try:
return public_api[name]
except KeyError:
if name in full_api:
msg = f"module {__name__!r} does not re-export {name!r}"
else:
msg = f"module {__name__!r} has no attribute {name!r}"
raise AttributeError(msg) from None
return __getattr__
__getattr__ = closure()
del closure
That’s a working module with a restricted public api without another module acting as a facade. Module author’s choice on error or resolve with warning (just change the handling of the KeyError) There’s various other ways to do it, including mutating the module at the bottom (type checkers are easier to appease going about it that way, but there are other things to consider)
This version still conflates public api with wildcard imports, but that’s not strictly required. This can be extended further in various ways, but just restricting/warning for accessing things not in the public api isn’t that enticing on its own, we can already do that.
1 Like
Would you be okay with
from X import Y
export Y
?
3 Likes
This would double the number of lines in the module (compared to adding export to the import statement) and requires duplicating Y. In the end, this would achieve nothing while adding the burden of an extra keyword to support.
1 Like
No, the best we can do is add the export from X import Y (or similar) syntax, eliminating DRY.
ECMAScript has:
export * from "module-name";
export * as name1 from "module-name";
Thus the shortest syntax would probably be:
export Y from X
But I’m fine with anything that expresses the right intent while eliminating the DRY.
1 Like
Ok, this has come up multiple times so I’ll clarify. My original comment was probably[1] too brief, but the intention was that you engage with anyone raising an issue because they used an unsupported API, and explain to them why you don’t support what they are doing. This, to me, is very much collaborative while still asserting boundaries on what you (typically as a volunteer) are willing to offer.
This is in contrast to simply configuring the code to give an error (or now, with the changed proposal, a warning) to the user, with no engagement or explanation at all.
The latter feels to me to be far more adversarial. YMMV, of course…
in hindsight ↩︎
5 Likes
We’re getting deep into pip’s specifics here, but IMO it’s worth it as it illustrates how messy real-world application structures can get.
No, that would not work, because pip has a pip._vendor namespace containing vendored libraries. This proposal would allow pip to use internal names from vendored libraries.
Of course, you could say “that’s fine, the mechanism doesn’t have to be perfect”. But we were already at a point where the proposal won’t work for pip, and now we have a situation where it won’t work for vendoring. If we aren’t careful, we’ll need a section in the PEP to cover all the edge cases where it doesn’t work…
3 Likes
Furthermore, given the existence of namespace packages, every directory is, in some sense, a package (unless it doesn’t conform to the naming requirements of a Python package, I guess )
So I’m not even sure what a “common parent package” means in any precise sense.
1 Like
I could not find anything in the PEP about the mutability of __export__. If a user can simply mutate it, what is the point, defense in depth?
The proposal sounds to me like putting a name in __export__ to make an already public name public again. It is not visible or obvious that this would make all other names private. What intuition does this follow? Are there any precedents?
1 Like
TLDR: Don’t fix what isn’t broken.
I think “sprinkling” code with underscores is a very acceptable solution for robust library development. While for lower quality / robustness libraries / recipes / scripts, things being public is not yet an issue.
E.g. I think below is a good solution - I have adapted this and am very pleased with it - no need for any special machinery - everything is simple, makes sense and achieves desirable effect.
# spam.py
import argparse as _argparse
import asyncio as _asyncio
import tabnanny as _tabnanny
If something is really really private and one wants to deter the user from importing, can just put 10 underscores in front - this will likely have an impact at least somewhere close to what a warning would do.
Personally, I (almost) never make use of stuff that has more than one underscore.
“Ideally, users shouldn’t be tempted to reach for private names from modules in the first place.” - PEP842
From users POV, I like things to be transparent and being able to see and tap into everything.
I often use private stuff when I find there is something C optimized (or just generally useful).
And I always appreciate that it is on me to have a fallback if it is removed or changed.
Also, if something is private, but useful, then the signal from users once you change it can be an indicator that maybe it is worth considering exposing useful functionality publicly.
Also, I think being able to inspect things easily without needing to learn a bunch of special cases and machineries is as important for more advanced users as it is for beginner level pythonistas.
Any extra friction here will inevitably, at least to some degree, hinder learning, exploration and valuable collaboration.
Furthermore, I like private things being underscored.
The one who writes the code would not be impacted much.
However, reading code where everything is non-underscore means seeing everything as public while having to simultaneously keep scrolling to the top to figure out what is what.
I think underscores is a good method to keep minds of developer, maintainer and user in sync.
Although I agree that it isn’t perfect in all aspects, I think what this PEP is suggesting is a move away from pareto optimum.
I think the referenced issues would much better be solved via soft methods such as communication, documentation, and agreement. Or if something to address these systematically is actually very desirable, then this needs much more serious cooking to figure out something that doesn’t take away so much from other aspects of experience.
4 Likes
Extension modules already go against that. They make inspecting harder (probably impossible for newbies) and can trivially block their internals from users with no practical way to poke their internals.
4 Likes
The whole concept feels over-engineered. The current PEP draft does not refer to packages or submodules or alike (please correct me if I am wrong), private means “private to the module”. That is fine since module namespaces are a well-defined concept of the Python language. Widening the concept of private to submodules, packages, modules that live in the same directory or whatever else seams to be a bit alien here - in every case, it will be implicit and have sharp corners. I would say, “explicit is better than implicit”, so there should be an explicit way to say “give me the private members of that module”, which can be used by whoever wants to use it.
1 Like