The second proposal of the Public API Trilogy has been publish, albeit out of order. You’ll have to go back and read the second novel when it comes out. Here then is PEP 844 proposing to add public and private builtins. The proposal is based on the atpublic library and its decade-long experience.
6 Likes
Thanks for the write-up!
As the author of PEP 842, I’m partially playing devil’s advocate here, but I think it would be good to respond to the arguments of “Library consumers use runtime introspection for documentation”. The documentation aspect of __all__ is nice, but it’s not clear to me how users are discouraged from grabbing things that look public (i.e., missing a leading underscore).
I can see @public (or @export) as a totally viable alternative for export, if there’s so much feedback against export being a (soft) keyword that it jeopardizes the PEP.
However I don’t really see the point of also having @private. It doesn’t have semantics, it’s just documentation. So why not use a comment? After all, private is the default. I don’t see a reason to make this explicit – it’s just going to cause ultra-purists to add noise to their code by using @private for everything that’s private (and they’re going to ask for that functional form for sure).
Even though this PEP makes a big deal of not introducing new syntax, I don’t see that as its key differentiator. This PEP doesn’t propose enforcement at all. I personally think that would be a mistake – most packages that implement access control do implement enforcement, since they really truly don’t want users to access private attributes. So I’m still behind PEP 842.
3 Likes
public()additionally has a function call form for names that cannot be decorated, such as constants.
I don’t see that working for type _ = ... aliases; any thoughts on that?
Right now, I don’t think any of the three proposals do what I would want as a library author. All of them deal with module level private/public, with the other two being more focused on the re-export problem.
My interest in any of the proposals lies in being able to better indicate the public API without having to do things like import foo as _foo and then use foo as _foo everywhere internally, but to do that for module level and not classes as well seems horribly incomplete for purpose, and not a significant enough gain as proposed by any of the proposals.
Similarly, I don’t think repurposing __all__ works anymore. (It it did, I don’t think this would be something many library authors would have a need for any change here at all) There are various details I may consider public and stable, but not something that should come along for the ride with import * from foo, especially when taking into account things like typing symbols that often have short names that other projects may also use the same name due to obviousness when locally used. (an example of this is a typing name like Callback, where it describes a specific requirement of Callbacks my library accepts.)
2 Likes
I suspect it’s for anyone who wants something like a linter rule saying, “everything must be explicitly marked public or private” so you don’t miss something. But if the assumption is that marking anything public in a file implicitly means everything unmarked is private then it is overkill as I would assume you would just fix the thing you accidentally didn’t make public. The only other use is the inverse where something is entirely private, and so you don’t have anything marked public but you want a way to state that fact.
1 Like
Playing the devil’s advocate here as well: It’s often proposed that features should first be implemented as third-party library and if they get serious traction, one may consider including in the stdlib/core language.
Keeping __all__ up-to-date is cumbersome, still only few projects have adopted atpublic to manage all. We should try to find out why. Hypotheses could be
- they just didn’t know
atpublicexists - they didn’t want to add a dependency for this
- being able to see the complete public interface centrally at the top also has merits - maybe that’s considered more important than the syncronization effort.
- maybe keeping
__all__up-to-date isn’t too hard after all? At least, linters can flag undefined entires (ruffF822: undefined name in__all__). And in the reverse case of forgetting to make an element public, it’s not clear whether a missing entry in an explicit__all__list or a missing@publicdecorator is easier to spot.
Putting @public into builtins would only help with the first two.
Also, have you considered putting this into the stdlib instead of builtins, e.g. into a new api module? This may be a smaller change, and something like
import api
@api.public
class MyClass:
seems decent enough.
The title may be a bit confusing for those who don’t know the context (haven’t read the previous threads), so I will fence the keywords with backticks.
I like the general idea of public() but also worried a bit about the relation of __all__ and the different levels one can have of being public.
I think a nice convention may be to combine using public to mark something as exportable with the underscore convention (or private) for marking something as truly private, since that leaves undecorated names not starting with an underscore as possibilities for an intermediate state of (somewhat) “public and stable”, i.e., items one could, e.g., rely on in different parts of a large package (large enough that one doesn’t want to just cross-important everything), but which are not meant for direct use outside of the package (in the sense of “if the API changes in the next minor version, you adjust, you don’t complain”). I think within the astropy package, which has many subpackages dealing with different (astronomical) context, this might work reasonably well.
-- Marten
My personal take on it, and thus the focus of 844 is that the combination of an @public decorator, underscore-prefixing non-public names, introspection of __all__, and good documentation is enough to prevent accidental usage of non-private names[1]. I don’t think we should make it harder to deliberately use private names, and the combination of above should also make it clear what is intended as private vs public.
There’s always the possibility that someone will use a private name because it’s useful to them and that’s the way Python works. I think that’s a Good Thing! Even in CPython there have been cases where something starts out private, becomes useful, and gets turned into a public name.
at least to the Pareto principle trade off ↩︎
It’s not entirely obvious, but it does have mild semantics in atpublic; it will remove a name from __all__ . It’s mostly there for symmetry and documentation, but I wouldn’t have a problem if it’s not added to builtins.
Fair enough, though I feel like we’ve tried various enforcement and restriction techniques in the past[1] and they never could be made reliable or Pythonic. Maybe restricted execution is a different enough use case, but it definitely left a bad taste in my mouth. I don’t want Python to be annoying 
.
e.g.
Bastion’s,rexec, etc. ↩︎
True enough. An undocumented workaround[1] would be:
type Alias = int | str
public(Alias)
The longer, repetitious-but-currently-supported spelling is:
type Alias = int | str
public(Alias=Alias)
and I would have to think about whether this is a valid use case that should be supported ↩︎
Again, because my preferences are against strict enforcement, _foo naming for non-public class attributes and methods gives me everything I want without annoying restrictions.
Another thing @private could be used for is accidental duplication:
@public
class Thing:
pass
# 8000 more lines of code
@private
class Thing:
pass
In this case, "Thing" would not appear in __all__.
I can’t claim any kind of scientific study, but I do think that the first two (lack of exposure and extra dependency avoidance) are the two primary reasons.
Modern linters do help somewhat with __all__ syncing, i.e. F822 as you point out, but not the reverse case. Missing @public decorators are always possible, but I think that’s a mistake much less common than trying to keep very distant __all__'s in sync.
I have, but have so far rejected that because a) it seems weird to add a new top-level module just for one new thing, and b) I’d still like to avoid the extra import, which again is some small amount of friction.
FWIW, atpublic 8.0 will have an extra that puts it in builtins via .pth/.start files, so you could pip install atpublic[install] and you won’t need the extra import (likely littering many modules in your library).
While the proposed built-ins look good in the decorator form, as the PEP correctly points out both usages of the function call form come with their downsides.
Not clear to linters that a, b, c and d are names:
public(a=3, b=2, c=1)
public(d=9)
No longer DRY:
a, b, c = public(a=3, b=2, c=1)
d = public(d=9)
Usually I’d understand the rationale since adding built-ins is cheaper than making syntax changes, but in this case the PEP is pitched as a companion to PEP-843, which is already proposing syntax changes anyway, so I still think it’s worth making public and private soft keywords in the same go if it helps achieve the goal with a cleaner usage:
public a, b, c = 3, 2, 1
public d = 9
and then if public and private are made soft keywords, we might as well turn the decorator form into actual syntax too:
public def foo(): ...
public class Bar: ...
The PEP makes the argument that new syntax is costly and constraining:
New syntax is the most expensive thing Python can add: it must be taught, it cannot be back-ported, it constrains the grammar permanently, and it is unavailable to every module that must still run on an older interpreter. A builtin costs none of that, is trivially shimmed on old versions, and (as is the case here) has a decade of usage experience behind it.
As I read it, the PEP is scoped to solve actual problems without requiring new syntax and the baggage that comes with it.