One of the problems with using __all__ to mean “public API” is that there’s no way to differentiate between “intended for use outside the library” and “intended as a symbol that’s visible to other modules in the library, but not to the consumer of the library”
This isn’t an invented problem, and it’s one that’s been mentioned in multiple of the related threads you’ve participated in.
The hub model doesn’t solve this piece of the puzzle.
Do you have examples of modern libraries (like numpy, scipy, etc.) that use __all__ to mean “intended as a symbol that’s visible to other modules in the library, but not to the consumer of the library”? In my experience, the latter concept usually is indicated with an underscore.
What’s wrong with just saying that __all__ is the way to indicate library-level visibility?
I can refer you back to one of your own examples with jax, where the libraruy authors found that insufficient, and named the module something absurdly long to act as a clear reason someone shouldn’t be importing the module that wasn’t _ prefixed
It controls other behavior, like that of * imports, which as was also mentioned, has negative side effects when considering typing names that users should be able to import from the related module, but shouldn’t clobber other uses of the same obvious short names
Sorry, I think you misunderstood the example. They named it that to prevent its use. The symbol is not even exported from the public API.
I don’t think I did misunderstand it. If it was sufficient to say users are on their own for using private things, and it’s sufficient to say something is private by _ prefix and non-inclusion in __all__, then what jax is doing is totally uneccessary. Clearly, their use isn’t aligning with your own argument here.
From what I’ve observed, libraries now don’t worry about what appears in __all__. An excellent example of this is that Numpy used to expose bool_ and bool. The Array API simply exposes bool and it’s free to be a dtype.
If you follow Numpy’s lead here, there should be no separation between the public API and the contents of __all__.
If you think that Jax considers symbols from within their library (not in __all__) as part of their public API, feel free to ask a question on their issue tracker!
I’m not going to follow numpy’s lead here. I would have no issue with an empty __all__ to make * imports effectively no-op (other than side effects of module eval) if there was a better way to express the public api. I shouldn’t have to be wary of naming a typing symbol Callback because it’s a common name to avoid such issues.
If that’s the case, your presentation of it in support of other proposals was deceptive. You presented that as if it proved the need for something else to be done.
Let’s keep this civil please. If you didn’t understand something I wrote, feel free to ask questions. Accusing people of being deceptive is not appropriate.
I’ve never considered the second item to be an issue. To me, other modules in the same library always have access to everything else in the same library, regardless of whether they’re listed in __all__, have underscore prefixes, etc. The latter might be mildly ugly, but to me, it’s not a violation of any contract.
And that’s how type checkers behave too: __all__ or import x as x only affect what they allow as imports in other libraries, it’s never considered within the project’s code.
I’ve never worked with it this way, and I’ve always found this to be a place python is lacking compared to other languages.
I’ve had to work with larger projects where different people are responsible for different submodules, so the need to differentiate between “module public” and “library public” has some up.
Inevitably, this has led to ugly solutions because __all__ by nature can’t do both.
Exactly the kind of thing @public is good at documenting 
I don’t see how that’s true. public just deals with __all__
Related to “consenting adults” philosophy, I think the fundamental issue is whether library authors should have a builtin/convenient/sanctioned way to indicate lack of consent for consumers to play with their internals.
We should not make things inacessible, but we should consider better ways for distinguishing the intended public API and warrenty-void-if-accessed internals. See also my reply A two-namespace model for Python modules - #18 by timhoffm.
To be clear, I am not arguing that we should make internal/private/whatever term we settle on inaccessible. I’m observing that library authors do not have a reliable way to indicate what level of consent they give. This has historically resulted in public pressure to open up what they wanted to remain private because it was not sufficiently clear what they were consenting to.
That’s the point: __all__ is just a list of names, not a namespace. A namespace is a mapping from names to objects, i.e. you can access the objects via the namespace. __all__ not being a namespace has two downsides:
__dict__), which means we have duplication, and one has to keep the names in __all__ in sync with __dict__. atpublic / PEP 844 try to improve this. Still it’s duplication, only a better managed one.__all__ is “only” a documentation of intent. mymodule.public_name and mymodule.internal_name structurally look the same and are handled equally in execution logic, even though only “public_name” is mentioned in __all__. I propose that “read what is written in __all__” or “linters tell you when you use variables not in all” is not enought and instead it should be directly visible in code when you access non-public elements (e.g. through mymodule.__internal__.internal_name).A true internal namespace could address both.
It’s a slightly different angle: Modules have one namespace, the global namespace. It has two responsibilities: It’s the namespace for implementation (e.g. you define helper functions in it and import other names that you need for implementation), and it’s also the namespace used by external users to access the intended public names.
These two responsibilities are too much for a single namespace, because we don’t want external users to access implementation details. Since we just have that one namespace, we we’re forced to come up ways to declare what part of the namespace is actually public (__all__) or create separate implmentation files and only re-import the acual public parts into the actual module. Both feel like mere workarounds and have the above mentioned disavantages. In contrast, one could elegantly solve them using two namespaces.
Okay, I understand what you’re saying now. Thanks for clarifying this. 
Makes sense.
I understand that you don’t like hubs, but for people that you use them, the private access is not far from what you have: something like from jax._src... import .
I see your point now. You basically want:
Your __internal__ idea is a bit of a tradeoff though because if you ever moved something from the private to the public API, users of __internal__ would have to change their code (whereas they don’t need to now).
Thanks, yes, I understand your idea now. If you do decide to write up a proposal, it would be nice to see how you address various tasks and issues like:
Seems all doable though so I guess it will ultimately come down to taste 
I’m glad I could clarify my main idea.
You are posing good questions on the details. While I have some thoughts, they are not set in stone and there are different ways to handle this, so take them as rough preliminary ideas, not as fixed proposals:
@public/@internal similar to atpublic/PEP 844 would work. The export keyword from PEP 842 would work. I currently have a slight preference for an internal keyword - basically reversing the logic of PEP 842: specify what is internal, not what is public. This is easier for incremental adoption: Libraries can just start adding internal for new things and/or deprecate public access. But there’s also the converse argument that people rather want to specify the public interface. This is certainly a point that needs thorough consideration and discuttion. On the plus side, I think all variants are compatible with the two-namespace approach and we can choose the best.internal import (or conversely a keyword export i.e. from my.submodule export foo). Again this needs careful consideration on whether we want to explicitly put imported names in the public or internal namespac.On a side note: Yes introducing keywords has a high bar, but adding a second namespace is a big step as well. If we decide this is worthwhile, we should at least consider using keywords to mark names. IMHO keywords are the clearest way to express scope for all relevant cases: imported names, classes, functions, module-level attributes.
I understand this can be a little annoying for library authors, but at least they have control over all import sites so this is a relatively simple adjustment.
For external users of __internal__ I would actually consider dismissing this as a problem outright; if you use the internal namespace, you sign up for potential breakage on any version change. Maybe it’s even a good signal that shows consumers “hey, this implementation detail you relied on is now part of the public API”, which is a very useful signal.
But just because there’s two namespaces, that doesn’t mean they habe to appear separately from the outside. We could also consider import being smart about the relative location of a file and, based on that, either allow or disallow importing from the internal namespace directly.