Namespaces are one honking great idea – let’s add one more to every module!
A module logically has an external API (the symbols users are intended to access) and an internal implementation (the symbols used to implement the module but not intended to be part of its API). But a module currently olny has a single namespace.
Current approaches to distinguishing external and internal symbols are:
The leading-underscore convention: distinguishing them by naming convention.
__all__: a separate declaration of the module’s intended external API.
The hub-and-spoke approach: creating separate namespaces, with a hub exposing the external API and additional internal modules containing the implementation.
They all have their merits, but they also have limitations. Recent PEPs 842 and 844 are exploring ways to improve this situation, primarily by improving the declaration or filtering of names in the existing namespace.
Before discussing how to mark a symbol as internal or external, I propose that we first consider the underlying structure into which we want to encode that information:
Would it be cleaner to represent the two scopes explicitly, so that each module has two associated namespaces: one for its public API and one for its implementation?
Users of the module would primarily concerned with the public namespace, executing the module would use the internal implementation namespace.
This is deliberately a proposal about the underlying model, rather than about a particular syntax or set of semantics. In particular:
The mechanism used to populate the two namespaces is an open question. It could use decorators, a keyword, or a top-level declaration.
The model does not inherently prescribe whether modules should be public-by-default or internal-by-default.
Separating the namespaces does not necessarily imply access restriction. An implementation could leave internal names fully accessible, merely making the distinction available to linters and other tools. Alternatively, the implementation namespace could be exposed through a separate attribute, requiring users to explicitly opt into accessing implementation details without preventing them from doing so.
This could be introduced as an opt-in mechanism, leaving existing modules unchanged. The leading-underscore convention, __all__, and the hub-and-spoke approach would all remain valid.
The proposal is initially about module-level namespaces. It does not attempt to introduce access modifiers for class attributes or methods.
The goal at this stage is therefore not to propose a particular syntax, but to explore whether a first-class distinction between a module’s public and implementation namespaces is a better foundation for expressing module APIs than continuing to represent both concepts in a single namespace.
I like the approach to first define what before going to how.
Must the public namespace necessarily and always be a subset of the internal namespace? I think so, but establishing that explicitly could bring merit (eg. when getting to how):.
And is visible/invisible sufficient? I could imagine names being read-only for outsiders, and mutable when used internally. Also see current ReadOnly PEP 767 (although that’s at attribute level).
I think organizing this as three approaches hides the current reality. In nearly all of the projects that I’ve contributed to:
A leading underscore means private to the module. No leading underscore means public to the library only.
Presence in __all__ means externally public. Not present in __all__ means externally private. This is irrespective of whether a name has an underscore or not.
The hubs are simply convenient places that contain __all__ variables. Other modules don’t have or need __all__. They’re not a separate idea. They simply use the __all__ mechanism.
I think this is pretty standard among newer Python libraries. The benefit of the hubs is that externally private symbols are not easily importable, and it is easy for the library structure to deviate from the exposed structure. I think the issue is that some people are reluctant to create these hubs.
And the downside is that these hubs are often just a poor substitute for actual visibility semantics.
Placing an internal in another file results in a logical separation that’s often not ideal for human authors, as the internals are often tightly related to something intended to be public.
The reluctance to create the hubs where I’ve seen it has been because the hubs would be detrimental to any goal other than controlling visibility.
Namespaces are essentially dictionaries where some name lookups happen, so my initial question is, if a module has two namespaces, how would name lookup order change? What happens if a name is in both namespaces?
Possibly “approaches” is not the right term. These are all established conventions and patterns for managing aspects of the “public/private” topic.
The fundamental point I’d like to discuss is: Are these conventions and patterns the right way to handle the topic, or are they just the best one could historically do without a dedicated language feature for “public/private”? And would having two namespaces on a module be a meaningful foundational structure to manage public/private?
I think what you’re saying is mainly true for smaller libraries. As libraries grow, they tend to benefit from the hubs because the hubs allow the internal tree structure to deviate from the one they expose.
Depending on the characteristics one wants to achieve, there could be different implementation strategies.
One would be to add __internal_dict__ and add names to it based on a keyword, decorator or function (conceptually the same markers as proposed in PEP 842/844, but rather marking the private parts, not the public/exported ones).names without that go into __dict__ as before. For name lookup in modules, one would extend the lookup order LEGB to LEGIB. The I standing for the internal namespace.
The nice property here is that nothing changes for external users: __dict__ and all related functionality stay as is. It would be an opt-in decision of module owners to move selected names out of the publicly visible namespace into the internal one. If desired, we could still make the internal names easily accessible through something like mymodule.__internal__.internal_name.
There are other possible choices, like having everything that is added to __dict__ also being added to __internal_dict__ so that for name lookup, you’d internally only look at __internal_dict__.
Another degree of choice: Instead of marking the private names, you could also mark the public names as in @public / export.
But these are nuances and we can choose the model we like best. Distribution of names to the respective namespace and where to look would be handled through the language, so that the exact content of the namespaces and public/private markers can be designed as needed.
I think there’s reasonable use cases where that isn’t true. As has been mentioned before, one of the big examples of the hub-and-spoke module system is that you can change the internal structure without breaking the API. With that you can also e.g. rename objects internally while keeping the name in the API the same.
For that to work when the public namespace is a subset of the private one, you’d need to keep two references to the renamed object around. Similarly, when you move an object to another module, you’d have to import it into the private namespace, where it would then never be actually used. Those are reasonable workarounds, but I think the elegance of having the namespaces be completely decoupleable is worth the added complexity.
This made me realize that we currently also have the warnings.depcretated decorator, which acts as a sort of access-modifier-but-not-really-and-mainly-documentation. Maybe the best approach wouldn’t be to formalize a public/private stratification through namespaces but have a broader method of attaching metadata to names in modules. I could imagine something similar to typing.Annotated where you can in principle attach arbitrary metadata to names and the standard library provides some set of predefined metadata tags like public/private, readonly or deprecated. These tags could then either just serve as an information source for documentation tools or IDEs, or there could be some hooks in the import or module access system that allows them to e.g. emit warnings. This also has the benefit that we now need to only define a limited core feature set and then can see what additional metadata tags get widely used in practice, which could then be added to the standard library.
For a start, I’d have the public names always be accessible internally - it’s still a design degree of freedom whether one wants to replicate __dict__ to __internal_dict__ (or however the internal namspace is realized) or whether you do two steps in name resolution (LEGB → LEGIB as a possible implementation suggested above).
It’s conceptually simpler to (1) not managing multiple references to the same object and (2) always use the same name publically and internally.
If there is need for extended functionality, I think that can be added later to the namespaces. After all, with the suggested approach the user only specifies semantically whether a name is public or internal and the technical implementation could be declared to be an implementation detail so that later changes or extensions are possible.
This proposal is primarily concerned with module-level visibility, for which namespaces are a straight forward technological solution with almost zero overhead. IMHO the need for module-level write protection is low. Most exposed names are functions, classes, or constants (using all-caps spelling), for which accidental modification should be rare. For the few module-level variables, declaring them internal/with underscore and exposing the value through a function may be good enough.
It’s good to discuss all related aspects so that we don’t render us in a corner and so that we don’t regret not thinking about an extenion. But overall, I’d try to keep the initially soution focussed. On module visibility.
By “module-level visibility” (or maybe it should rather be called namespace-level visibility) I mean that you can decide for every name whether it should be visible from the outside or internal; e.g. for each of the namespaces mypackage, mypackage.foo, mypackage.bar you can have additional internal names that are not accessible via [namespace].internal_name.
For simplicity, I’m inclined not distinguish “outside”, i.e. when importing/accessing mypackage.foo, an external user would see the same interface as mypackage.bar. Giving mypackage.bar direct access to mypackage.foo internals may be possible, but complicates things as we’d have to distinguish from which kind of “outside” the access comes. Currently and in the simple solution, we just lookup in mypackage.foo.__dict__. The internal lookup is already special through the LEGB and therefore easy to extend to handle additional module-internal names.
I would also say that this simple rule already gets us quite far. Package authors can still do from mpyackage.foo.__internal__ import internal_func[1], or import mypackage.foo plus mypackage.foo.__internal__.internal_func. Also all underscore prefixing remains a valid pattern.
For a start, I’d go with this solution. It’s possible to built-in cross-package visibility later if deemed helpful. The only change that brings is relaxed visibility between modules in the same top-level namespace, and I’d consider this a non-breaking change.
possibly with an internal import syntax so that the name stays also private in mypackage.bar↩︎
Right, so if I’ve understood you, you mean you want to control imports from within the same library.
So, currently, many (most?) projects use a leading underscore to indicate this “module-level privacy”. And linters already report access errors for it.
I’m curious why you think that’s not enough? Is it because:
non-underscore imports are typically not included in this:
# package/a.py
from .other import x # no one wants to import this as _x
# package/b.py
from .a import x # very weird, but no linter complaint -- why not import it from other directly?
you don’t want to name private variables with an underscore,
you sometimes want to name public variables with an underscore, or
you don’t think a linter error is enough and you want run-time access control?
Also, just to be clear to other readers, this is very different than any of the PEP 842–844: those are mainly concerned with “exports” (to users of the library) rather than “module-level privacy”. One of the major differences between these two problems is that library users can’t be trusted not to reach into a library and import private symbols. Whereas, you can selectively prevent your own library’s modules from importing module-level private symbols since:
someone from your own team is doing the reviews, and
you have control over the library code and can fix offenders!
Yes, all the aspects you’ve mentioned are correct and contribute to the idea.
There’s one more aspect I’d like to emphasize: There seems to be a relatively broad consensus that the current handling of public/internal (or private) are lacking. PEP 842-844 are primarily concerned with improving the way specify these (e.g. through decorators or new keywords). I’d like to put focus on how we want to store that information. __all__ as a list of strings feels a bit like a bolted-on solution and has some disadvantages. I’d like to discuss whether namespaces are the better underlying technical concept.
On an additional note, “export” focuses on declaring public elements, whereas the default notion here would be declaring internal elements. But that’s debatable. I believe declaring private elements is simpler to introduce in a backward-compatible way. But since we control how names are assigned to pubic/internal namespaces. The namespace approach would still allow to build the API around declaring public elements. So that’s not a distinguishing factor.
Yes, but as I was trying to say, those PEPs are concerned with library-level privacy. As far as I can tell, the motivations for those PEPs are not about module-level privacy.
Sure, but __all__ is about library-level privacy. What’s not in __all__ is private to the library, but happily imported from within the library itself. For example, many libraries that use hubs have no __all__ variables in their non-hub modules.
So, I’m confused again whether you are interested in library-level of module-level privacy. Which problem are you trying to solve?
I think it’s time I speak my mind on this, just to make sure it is spelled out clearly once. I’ve read the threads on the two PEPs you mentioned, and even after that, I disagree that the current ways of indicating that something is not part of the public API are lacking and I fundamentally do not want Python to offer a convenient way to create private symbols in modules.[1]
As the consumer of a module or library, I want to be trusted to decide about the tradeoffs of relying on or monkeypatching implementation details of a library to suit my needs. I am an adult and can make my own decisions. Maybe I want a library to behave just slightly differently and maybe I have to do something cursed to achieve that that breaks future updates. If that is so, it should be my decision to make, not the library author’s.
If we give library authors the power to make internals inaccessible with no escape hatch, we take away power from library consumers to leverage the libraries however they see fit.
I do agree that there must be a way to mark symbols for internal use only, but I believe the leading underscore (and double leading underscore if you’re very serious about not wanting something to be accessed from outside) is a great pattern for that, and a lot of tooling picks up on this pattern already.
I’m not really sure what a second namespace could buy us that we don’t already have.
of course there’s inconvenient ways already, you can do whatever you want if your really set your mind to it in Python ↩︎
To reassure you: This is not about making anything inaccessible. It’s about a clearer separation. You should still be able to access all internals if you want to. The exact API can still be discussed, but one way could be exposing the internal namespace via module.__internal__, so that you can still do module.__internal__.internal_name.
Because the @private decorator exists in the context ofd PEP 844 without implying inaccessibility, that that term still floats around. I try to avoid the term private because it has the inaccessible connotation in other languages and thus different people here mean different things with "private. Instead prefer the terminology public/internal.
As I’ve written somewhere in the other threads. The underscore prefix convention is almost sufficient. It falls short in two cases:
For imports all imported elements are public by default. To formally remove such imports you’d need import renaming such as from pathlib.path import Path as _Path.
Unfortunately, there are some - admittedly rare - cases where underscores are used but the names are not considered private; e.g. the methods on namedtuple where underscores were introduced to avoid name clashes and not to signal non-public.
I’m not 100% sure on the definition of these terms. Can you please clarify?
I nevertheless try to answer up front. Libraries expose their API through one or more namespaces (e.g. mylibrary, optionally also mylibrary.foo, mylibrary.bar). Those namespaces are defined through files mylibrary.py or __init__.py, foo.py, bar.py. I try to solve the issue that those files are also use to implement the code and therefore can contain more names that should not be public. Current structures do not provide a namespace separation between exposed names and internal names. There are serveral partial solutions to the topic like underscore prefixes, __all__ conventions and creating separate implementation files. But I propose that using separate namespaces for exposing and internal names is a structurally more sound solution.
Okay. This is library-level visibility—not module-level visibility.
Yes, but this is fairly rare. Most modern libraries simply use hubs. I agree that it’s fair to look for a better solution than hubs.
Sorry, but this is unnecessarily confusing. There are exactly two reasonable ways in modern Python to indicate library-level visibility, and they are:
using __all__, or
using the from .x import y as y pattern.
Underscores are not a separate solution. They have practically nothing to do with library-level visibility. Yes, underscores are the fallback when you don’t define __all__, but I think this is poor design and few, if any, large libraries rely on this.
Similarly, hubs are not a separate solution. Hubs just do one of the above bulleted things.
Why do we need a “separate” namespace from __all__? It’s literally the list of exported names. Why do we need another copy of it?
My guess is that you’ve started out with the premise: “There are two lists of exports”, and then argued that “we should have two namespaces”. But there are not two lists of exports. There’s just one.
The underscore-names are solving a completely different problem (module-level visibilty), and the underscore-names are a fine solution for that problem as far as I can tell—unless you want to argue that they’re not.
It really feels like you’ve started with a solution looking for a problem rather than starting with a problem and motivating a solution. If you were writing a PEP, you would have to write the problem description first. It might help to write out exactly which problem you’re trying to solve.
