I recently wrote a proposal to hide private attributes from modules. My motivation for this was to prevent people from relying on stdlib internals. Now that the PEP has been withdrawn, I still think we need to find a way to achieve this goal, because the status quo of prefixing with _ just isn’t working.
So – what should we do? From the discussion thread, the two ideas so far are:
Implement something similar to PEP 842, but only for the standard library.
Convert standard library modules to a “hub” model, where modules are separated into two parts: one that holds the implementation, and one that exports them. For example, random.py could become random/__init__.py and random/_impl.py.
Can describe the problem you are trying to solve? If people use _-prefixed names from the stdlib, the core team is not obligated to ensure compatibility, right?
Well, I don’t known much about the implementation details of the stdlib, but from a distance I can imagine that reorganizing module structure would be a lot easier than having to introduce complicated visibility management mechanics. So as a (probably naive) outsider, I think that something like @NeilGirdhar 's proposal from PEP 842: Module Exports (new revision) - #129 by NeilGirdhar indeed sounds like a good solution, viz. option 2.
I wrote a lot about this in the motivation section of my PEP. In short: we’re not obligated to ensure compatibility, but in practice, we choose to do so, because the alternative is to break a lot of downstream users.
It’s likely easier mechanically, but as I said on the original thread, I don’t think that’s a good long-term solution. We might as well make things nicer for ourselves if we can.
Is there a better way to place the public API facing users looking at a module? (The policy of _ prefixing isn’t uniform). From that perspective, I think the answer is a clear and resounding “yes”, we can create a dunder describing such.
Do we need enforcement if the intended public API is clearer, or is that enough to be willing to break people using things that fall outside of it in the future? If it’s not, then there’s an argument that the entire API is public by convention of unwillingness to break users.
Although only 3 of the examples are specifically using _-prefixed names from the standard library. One was used in an example in the stdlib’s documentation and another was insinuated[1] as the way to do something by its repr. So really only one, fairly small example. This really sounds like cherry picked space bar heaters to me.
Personally, if I could change anything, it would be for the stdlib to have never used _ prefixes for things people are actually supposed to use in the first place. Then no-one would have ever been given reason to doubt whether the underscore prefix convention applies to the stdlib. But I don’t have a time machine…
maybe insinuated is quite the right word but hopefully you know what I mean ↩︎
To me, there is no doubt that a underscore prefixed name is not meant to be API, be it a PyPI package or some standard lib module. That is, unless the docs state otherwise, and even in those cases, I wish there simply were a non-underscore alias (say logging.acquire_lock) introduced soon that I could use instead.
For both of you: For brevity’s sake, the PEP can’t be a complete list. I grabbed what I found or knew about and put it in the PEP. I don’t want to reopen the talk of whether prefixing is enough, because that was already discussed extensively in the other threads.
Is it not an option (call it option 3) to simply stop choosing to ensure compatibility?
Ultimately, if an API isn’t documented, it’s private. And if users are ignoring that fact, that’s their choice but also their risk. There are awkward cases where the documentation is incomplete (importlib.resources, I’m looking at you ) but we can make exceptions without abandoning the general principle. And we can fix the docs…
If we implement some sort of stdlib-only protection, some people will still work around it (both PEP 842 and the hub model can be worked around fairly easily if a user wants to). Do we avoid breaking those users? Don’t we just end up with a never-ending arms race of “no really, it’s private, we mean it, we will break you without warning unlike last time” assertions?
PEP 387 explicitly says “Note that if something is not documented at all, it is not automatically considered private”.
Again, PEP 842 (and this thread) is more about cases where users accidentally reach for private APIs or are otherwise too tempted to use something private. I think by providing some sort of friction, we’ll see both less private API usage and more requests to properly promote private things to public (rather than just grabbing them and hoping for the best).
I’m not sure how relevant this quote is, in this context or in general, but I am not familiar with the deliberations that led to PEP 387, so anyone who has more insight please set me straight.
This does not mean that something that is “not automatically considered private” by virtue of it being undocumented is therefore public. At best I think it is undefined, and as such is not covered by 387. In fact, 387 says it is and example of something that is “explicitly not part of the public API. They can change or be removed at any time in any way.” Something that is undocumented is not necessarily private, but it is unambiguously not public and is categorized by PEP 387, by the same line item you partially quote, as receiving the same level of support as things that are explicitly declared private.
As a response to @pf_moore’s comment that “if an API isn’t documented, it’s private.” is splitting hairs. It receives the same level of support as something that is actually declared private, which I believe is the salient point.
I think before asking “how do we prevent someone from using a private name?”, we should ask “why did they use a private name in the first place?” If they thought that the private name was part of the public API, then, what led them to believe that? If they used a private name on purpose, then why wasn’t the public API sufficient?
You could guess and say, “they didn’t know that underscore names are private.” And somebody else could say (seems more likely to me), “they used an underscore name because the functionality wasn’t available in the public API.” But, I feel like if, instead of just finding examples of using private names, we could find out the reasons why they did that, it would probably help guide us to the solution to preventing using those names.
For the case of accidental use, the stdlib is actually in a special situation already in that most of the names editors provide come from the typeshed stubs and not the stdlib itself. Perhaps we can look at not exporting private names there?
I know this is already the case for annotationlib[1], trying to use _Stringifier will cause every checker I’ve tried to say it does not exist even though it does and it doesn’t show up on autocomplete. On the other hand ctypes._CDataType may be given as an option in autocomplete even though it doesn’t actually exist at runtime.
its private names aren’t even defined in the stub file ↩︎
Even the hub model isn’t super great. As the PEP pointed out, linters don’t like it (it’s unused exports and it might not be immediately clear that these imports are meant to be used by other packages; you essentially have to hard code the name __init__.py to avoid these problems). For me, even a minimal version of PEP 842 with just
from .module export symbol
would be useful.
And yes, there are workarounds and they work okay, but the language would be a bit nicer to use (for me at least) if this syntax existed.
Yeah, I believe that’s the interpretation. I wasn’t there when it was written, but when I’ve seen that line referenced, it’s been used to prevent core devs from writing off issues where breakage was caused by something that looked public, but was supposed to be private.
I’m not entirely sure, some of the names just leak but are used for typevars like _T. Otherwise I’d be guessing. I’m also really not familiar with ctypes as a module, looking at the stub file I’m not completely sure what’s ‘real’ and what isn’t.
This is only for IDEs, the REPL still gets the real details. We can override __dir__ if we need to hide things there. Looking at help() (which I rarely do for modules) I’m more struck by how it shows every inherited dunder method[1]. I kind of feel like if someone actually manages to find some internals through help() then that’s an achievement in and of itself!
There’s about 4 pages of dunder methods inherited from Enum for annotationlib.Format. ↩︎
If the focus is on the standard library, would it be enough to do the following?
Define a new dunder that, when present, specifies the public api. By “Public api,” this means the api as intended for use by consumers of a library[1], not those developing the library. This also means that a user looking at what’s in a module doesn’t need to cross-reference the rendered documentation to determine if something was considered to be documented for public use.
Change dir and help to only list things in the above dunder when present by default, but add a kwarg to them for showing private details.
Don’t add any new syntax or enforcement mechanism, commit to a “willingness to break users of private names”. Make sure this is mentioned in the release notes for the first cpython version with this dunder as a means of communicating it, and encourage users in any such notices to reach out for public api promotion. That’s not the same as requiring those users be broken, but it frees up maintainers to do it when appropriate for the overall maintenance of the public api.
If pressed for a policy on point 3 there, I’d say that in cases where a private name became public due to there being a good reason for users to have reached for it anyway, having an alias to the old name for transitionary use for the duration of the lifetime of the python versions prior to the name becoming public avoids breaking. aliases are relatively easy to mechanically maintain and phase out.
This definition allows the dunder to also be used by third-party libraries, while allowing linters to have a consistently applicable definition they can refer back to for “should we warn on this use”, which overcomes one of the reasons why a warning for using a name not in __all__ would be too noisy. While the focus here is the standard library, I think having something consistently applicable ecosystem-wide may help the standard library see tools point out issues here to users. ↩︎
) but we can make exceptions without abandoning the general principle. And we can fix the docs…