Hi PJ! Can you unpack this for me? What does “exporting visibility” even mean? (Agreed that for class members we should switch to private, but that’s a discussion for a separate PEP, since the requirements are quite different.)
Sorry, I meant Python doesn’t have export visibility in the same way that other languages do. If I use export in JS/TS for example, that makes the property accessible. Something that’s not exported isn’t visible or accessible, and that’s not what __all__ is doing in Python.
In most languages, “exporting” means “make this normally private thing available for other modules to access”, but in Python everything is already “exported” in that sense.
2 Likes
This does look a bit magical to me:
public(PI = 3.141592...)
It would be great if this can be written like:
@public
PI = 3.141592...
Which looks less magical, is probably easier to understand by static analyzers and makes the PEP more consistent.
But to make this work, assignment-decorators (or how to call this?) would need to be added first.
1 Like
I’m going to reply on this thread with multiple hats:
As an OOP C++ teacher
This is a controversial hat because I know not everyone will agree with me:
I do not consider this style to be ugly. It is extremely expressive and you do not have any ambiguity of what the function is meant for. You don’t need to read the comments to know “this function is meant for internal uses only” for instance (comments that may be hidden because your IDE decided to do so when loading the file). It also prevents your IDE to autocomplete it depending on the context you are currently writing from (and this is definitely an advantage).
Quoting PEP 20.2, I would say Explicit is better than implicit, and then we’re back to square one because PEP 20 did not help in really deciding something.
I very much appreciate reading C++ prototypes because they tell me many things about who should use that function and how that function would work and on which kind of types it works on (and hence I don’t mind reading templates or creating ones, coupled with C++20 concepts). In particular, I would not be against a keyword instead of a decorator.
Having a decorator is reasonable instead of a keyword, however, a decorator can always be redeclared and confused with existing ones. I had Python projects where @public, @protected and @private were solely meant for visual reminders and for generating the docs with Sphinx. With the good Sphinx extension, you can decide how to render those functions and improve your documentation.
As a Sphinx maintainer
Whether we have a keyword or a decorator, it doesn’t really matter because we can still analyze the code and hope that reassigning public or private won’t be common. Currently, Sphinx’s autodoc exposes :meta public: and :meta private: for categorizing documented objects into public/private interfaces.
I’ve seen multiple OSS projects using that in their docs and this is roughly equivalent to convening the intent of something private/public in a more systematic form, rather than just writing it in prose. For Sphinx, one sometimes needs to have more “meta” attributes and I consider @public/@private to be specific cases of a larger feature where you can put “attributes” (simliar to __attribute__((...)) in C/C++) on different objects (whether functions, structures, or data). It’s not really a type annotation, but it’s an annotation in the broader sense (there is no way to annotate a function definition itself except maybe as follows:
def foo(x: int) -> str: ...
foo: Annotated[Callable[[int], str], dict(public=True)]
So I would also like a built-in that allows to specify arbitrary “attributes” on functions/data/classes such that consumers can decide what to do with them. In the case of @public, the interpreter itself is the consumer as it determines what to do for __all__, while for @private, Sphinx could be have its own consuming logic. This would be complementary to Annotated which is specifically for type purposes.
As a CPython core developer
There are some arguments I would like to challenge / confirm in the PEP:
Syntax is permanent: a soft keyword constrains the grammar forever, must be taught to every future Python programmer, and is unavailable to any module supporting an older interpreter.
We introduced async def in 3.5, match in 3.10, type A = ... in 3.12, lazy import in 3.15, yield from in async in 3.16, so libraries desiring to support “older interpreter” are already stuck with not using some features. If they want to do the transition, they can use the atpublic package (AFAIU, we will just bring atpublic to the stdlib right?). As for teaching it to every programmer, I consider we already have lots of features that need to be taught and one more isn’t really the bottleneck.
In general I don’t find the argument of “it must be taught to everyone” compelling enough, especially for features that are oriented towards library authors rather than everyday users.
Nonethless, I can understand why not introducing a keyword is preferrable as a transition. This reminds me of @coroutine vs async def (although the two were slightly different I believe but essentially used interchangeably) where we eventually deprecated the @coroutine decorator in favor of a new syntax. If the decorator form of your proposal is widely used, we can also consider introducing keywords (which I believe are more elegant in this case).
If
__all__does exist it must be a list, orValueErroris raised, and the decorated object’s name is removed from it if present.
globals, creates
__all__as an empty list if needed, and appends one string.
I know people that specify __all__ at the end of the module. How would this work? When __all__ is empty, I use a tuple, not a list. __all__ only needs to be a sequence as specified in the language reference
The public names defined by a module are determined by checking the module’s namespace for a variable named
__all__; if defined, it must be a sequence of strings which are names defined or imported by that module.
Only PEP-8 is stating that we should use the empty list. However, this is not necessary.
@barry What happens if I write public(a=1) followed by private(a=1)? or vice-versa (and maybe del a in between). Since private() doesn’t alter __all__, this could be in conflict with previous (or future) calls to public. If I do del a in between, I would expect no runtime issues, but the fact that "a" is still in __all__ would make the __all__ desynchronized again (and I see public as a way to synchronize __all__).
@barry The following is mentioned:
The strongest objection to this proposal concerns the function call form, and it is worth stating explicitly. Given
public(SEVEN=7),SEVENis bound in the module’s globals by a function that reaches into its caller’s frame. Nothing about that binding is visible in the syntax tree.
However, I should mention that we already have a precedent with enum.global_enum:
import enum
assert "EVEN" not in globals()
assert "ODD" not in globals()
@enum.global_enum
class Parity(enum.Enum):
EVEN = enum.auto()
ODD = enum.auto()
assert "EVEN" in globals()
assert "ODD" in globals()
In addition, there is something visible in the syntax tree. The fact that SEVEN is used as keyword parameter for public.
As a library author
I remove my CPython core developer hat and solely retain my hat as a Python developer and library author. Those PEPs seem to focus on solving the following problems:
- Spelling the public / private interface more explicitly than just through docs.
- Reduce the amount of work for writing
__all__.
As a library author, I agree that repeating the names in __all__ is burdensome. However, since nothing is enforced at runtime (that is, a user could still do from mod import _private_function or from mod import dep where dep was imported by mod solely for its own purpose and wasn’t meant to be re-exported), I don’t find those proposals good enough. I’m not saying they have no value, but the reason why as a library author I would use @public or __all__ is entirely because I don’t want users to think they can import something I considered private. But this proposal still leaves me with no way to easily enforce that. However, as a first step, I also wouldn’t be against it as it still solves the synchronization problem with __all__.
In conclusion, while this helps linters to decide what’s public/private as well as library authors for __all__, this doesn’t help users that don’t use linters. Instead, what I would like to see is for library authors to be able to enforce what they consider private, public or protected at runtime, and anything that violates this contract would get an exception. But this is likely an other PEP for which the semantics and applicability are still unclear even to me (informally, I wished to have access modifiers as in C++ and Java with possible runtime and static checks, but only on demand, yet I’m afraid we would hit the same issues as we did with -X lazy_imports=none).
7 Likes
Now that PEP 842 is dead, I’d like to express my support for PEP 844. Some suggestions:
- I think @dimaqq makes a good point that users will probably use this in classes. So, why not just support methods? We could specify that classes can have
__all__variables too. Otherwise, these should raise exceptions when used in classes. - It’d be nice if
public/privatehid private names fromdir(). - Runtime enforcement seems to be a splitting issue in these PEPs. I suggest making it opt-in here.
@privatein particular feels like it should add some sort of runtime enforcement. - I would really appreciate it if there were a reference implementation in a linter. It’d make it much more convincing that this will actually help the documentation aspect in practice.
1 Like
Ah, that was what PEP 842 (now sadly withdrawn) was all about.
2 Likes
Maybe you should make another PEP that suggests that dir just go with __all__ if it’s available? It seems like that came up a few times during the 842 discussion?
And it is! But you can do something a little more explicit, at the expense repeating yourself, but which linters today can recognize:
PI = public(PI=3.141592)
Yeah, but there’s been no proposal to do so, and that’s definitely out of scope for this PEP. I’m not sure how that would even work since decorators are “just” syntactic sugar. E.g.
@public
def foo():
pass
is equivalent to:
def foo():
pass
foo = public(foo)
2 Likes
The atpublic library has a couple of ways to do this. You can infer __all__ using the public.populate_all() function. You can also just turn the list into a tuple at the bottom of the module:
__all__ = tuple(__all__)
private() does in fact alter __all__; it removes the name from __all__ if present[1].
If you del a you can’t private(a) and you’ll get a NameError as you should expect. This PEP changes none of that.
What will be added in atpublic 8.0 is that you can do private('a') (note the string argument) and it will remove a from __all__ if it exists. PEP 844 proposes no semantic changes to Python, so this all works as you’d expect it to.
I guess that’s a fair call out.
While I think PEP 844 is out of scope for this, as mentioned earlier, I think it would be feasible to add a helper function you could call in your module __getattribute__() to do the enforcement. As I see it though, that’s not something I want to propose building into the language specification.
there’s a bug that will be fixed in
atpublic8.0 where@privatecreates a__all__if one doesn’t exist, but it shouldn’t do that. Instead it should only remove the name but if there’s no__all__it should just keep Python’s default missing-dunder-all semantics. ↩︎
1 Like
This brought to mind the possibility of public def something_public(): ... and private def something_private() ...
Has this syntax been discussed previously?
I think suitable directory structure can help us solve the problems we are discussing. I’m particularly thinking of the user experience, for people who want to use a mythical our_project. Here’s my suggestion.
USER EXPERIENCE
I suggest the top level directory has exactly two subdirectories, like so:
our_project
our_project/api
our_project/_
The beginning user is told that it’s OK to use:
from our_project.api import something
And also that it’s not OK to write:
from our_project._ import something
MIGRATION
Suppose my_project already exists, and has a different directory structure. So how to migrate to the above structure?
Well, that’s easy.
$ mv -iv our_project temp_name
$ mkdir our_project
$ mv -iv temp_name our_project/_
$ mkdir our_project/api
And now you have an empty folder in which you write the API. And if you want you could place in that folder cleverness and magic that issues deprecation warnings. Or the API could be versioned.
CONCLUSION
My main point is that directory structure and and I should be used to help solve this problem. I’m not much bothered at this time whether or not we have the same preferences regarding directory structure.
A really important point is that our_project/api can have one directory structure, and our_project/_ can have a completely different structure.
Some projects might prefer one top level subdirectory for each current version of the API, like so:
our_project
our_project/api_v1
our_project/api_v2
our_project/api_v3
One final point. Technical programming skills are required to write our_project/_. Skill in usability and user experience and so forth is required for our_project/api. For example, consider default values for function arguments.
It's not a linter but here's how atpublic could be supported in Griffe in case you find that useful:
import ast
import griffe
class AtPublicExtension(griffe.Extension):
"""An extension to support `atpublic` decorators."""
# -------------------------------------
# Functions and classes
# @public
# def function(): ...
# @public
# class Class: ...
# -------------------------------------
@staticmethod
def _apply(symbol: griffe.Function | griffe.Class) -> None:
for decorator in symbol.decorators:
if decorator.callable_path == "public.public":
symbol.public = True
# We need to update exports in case other modules
# wildcard-import from this one, to be able to compute
# names available within them.
symbol.parent.exports.add(symbol.name)
# This assumes we support `__all__` in classes now.
# If not, we'd return early for non-module-level symbols.
# Example (before entering the loop):
# if not symbol.parent.is_module: return
else:
# Here we consider the symbol is private by default
# when the extension is merely enabled, but we could also
# only set `symbol.public` to false
# when `@private` is detected.
symbol.public = False
symbol.parent.exports.discard(symbol.name)
def on_function(self, *, func: griffe.Function, **kwargs) -> None:
self._apply(func)
def on_class(self, *, cls: griffe.Class, **kwargs) -> None:
self._apply(cls)
# -------------------------------------
# Attributes (regular assignments)
# SEVEN = public(SEVEN=7)
# -------------------------------------
def on_attribute(self, *, attr: griffe.Attribute, **kwargs) -> None:
if (
isinstance(attr.value, griffe.ExprCall)
and attr.value.function.canonical_path == "public.public"
):
attr.public = True
attr.parent.exports.add(attr.name)
attr.value = attr.value.function.arguments[0].value
else:
attr.public = False
attr.parent.exports.discard(attr.name)
# If `private` was used, we'd also update the value:
# attr.value = attr.value.function.arguments[0].value
# -------------------------------------
# Attributes ("call assignments")
# public(SEVEN=7)
# -------------------------------------
# For public attributes like `public(SEVEN=7)`,
# we need to inspect the AST ourselves,
# because the main AST walker doesn't do anything
# with function calls.
def on_module_instance(
self,
*,
node: ast.AST | griffe.ObjectNode,
mod: griffe.Module,
agent: griffe.Visitor | griffe.Inspector,
) -> None:
if isinstance(node, ast.AST):
self.mod = mod # hack to access it in `visit_expr()`
# This will pass through `visit_expr()` below.
self.generic_visit(node)
elif isinstance(node, griffe.ObjectNode):
# Here we explicitly don't do anything when
# Griffe uses introspection: the attribute is assigned
# and naturally appears in the module members,
# and `__all__` should be similarly populated.
pass
def visit_expr(self, node: ast.expr, agent: griffe.Visitor) -> None:
if (
isinstance(node.value, ast.Call)
and node.value.func.id == "public"
and self.mod.resolve("public") == "public.public"
# We would also have to support `public.public(SEVEN=7)`.
):
attr = griffe.Attribute(
name=node.value.keywords[0].arg,
value=griffe.safe_get_expression(
node.value.func.keywords[0].value,
parent=self.mod,
),
annotation=None, # no annotation with this syntax
docstring=None, # not supported yet, needs changes
public=True, # or False if `public.private` was used
)
self.mod.set_member(attr.name, attr)
It’s incomplete (see comments) and I might have forgotten some cases, but that gives a good idea of how this would be supported in Griffe (either through an extension such as this one, or natively).
3 Likes
Well, that’s essentially PEP 842 except that private is the default (and so is omitted) and public is called export.
2 Likes
I would be concerned by the performance impact such a pattern would have. It would be called by every access to a module name. If runtime checks are something we are even only considering as an option, we need to think about how current design choices will affect this.
In this context, I’d like to reference my idea to manage the public/internal state via two namespaces, which can naturally handle distinction at runtime.
Isn’t my version of your idea simpler? It accomplishes the same thing wihtout havinig to create or access a new dunder, and it can be converted to a hub at any time without changing any user code.
I’d say both have their merits and it’s a trade off.
Your proposal does not introduce new dunders, which makes it more lightweight. But OTOH dunders are there to formalize certain concepts, and codifying them explicitly can be helpful.
Not having that, your proposal has to compensate with conventions and background magic. I.e. code is implicitly moved to the _src namespace: What does this mean for the fully qualified name of moved names; do tbey contain _src now? This also means, there’s a difference in what is statically written in the file and what is available at runtime. This is a significant semantic change and may warrant stronger anchoring in the language. Linters still need to be aware of this to correctly parse files.
That would be hard to support in runtime/static analysis tools indeed 
PyO3 people like to lie about a name’s provenance (they don’t do it on purpose, it’s mostly accidental), to make it appear as it comes from the public hub rather than the private compiled module, and I have to tell them “please do not lie” (set the right __module__ value), as it makes it impossible to know whether a name was declared or imported in a module. See “make your compiled objects tell their true location” in Griffe’s docs.
Pure Python code that messes with symbol location is in the same vein, or worse, and I don’t want to deal with that 
1 Like
Yes exactly. But qualified names don’t usually affect control flow. It’s almost always for information purposes.
(I could have suggested that the locations don’t move, but I would rather make it equivalent to adding a hub so that if a hub is added later, nothing will change.)
What do you do for modern code that uses hubs? E.g., practically anything in numpy?
Ideally, you should be reporting the exported path rather than where the symbol is defined.
I’d say my proposal is significanly less “magic” since it relies on no new features. Everything it does could be accomplished by rearranging code. That’s the whole benefit: it’s exactly what creating a hub would do, so if you eventually end up creating a hub, nothing changes.
Yes, any change that hides symbols has to affect linters. That’s not really distinguishing any solution.
The names in the hub are extracted as aliases to the names in the submodules. Knowing the true location of a name, and knowing where it is imported is important for various reasons. Griffe knows both the exported and defined paths
To be fair, I just jumped at the “changes qualified name” aspect (sorry), but I think supporting your proposal wouldn’t be bad at all:
- static analysis: found name, name appears in
__all__and__restricted_export__is true → create fake_srcsubmodule and add name to it instead of current module, then add alias to this name in the current module - runtime analysis: found name, check whether declared here or imported, hopefully qualified name tells it comes from
_srcsubmodule, so create alias. Later, when inspecting_src, handle name as usual.
EDIT: ah, but dynamically created submodules are not easily discoverable. inspect.getmembers() does not return those 
So reaching the _src submodule during runtime analysis is not easy. I think.
No worries!
Pretty sure we would ensure inspect.getmembers returns the fake module since it should behave exactly as if we had created it.
1 Like