Hi,
I’ve written PEP 842, which specifies an __export__ variable that controls variable visibility in modules.
The full text is available here:
7 Likes
I trust you on mechanics, but I’m highly skeptical of the motivation.
This needs a very practical and impactful motivation to justify changing one of the defining characteristics of the language. Right now, it doesn’t have it.
“Imagine that a developer wants to define a private class in their module” is a great reason to choose a different language, but also it’s not something developers want to do, it’s something developers decide they have to do in order to achieve something else (such as “my users need help figuring out which class/function they should use” or “my colleagues can’t be trusted”).
A proposal that directly disrupts one of Python’s most unique and idiomatic features starts me at -1000. A strong motivation has some chance of moving that (implementation has no chance), and currently the motivation in the PEP adds -100 for me, which is the wrong direction.
What problem is this solving, other than the “I miss this feature” problem?
16 Likes
Thanks for the feedback, Steve.
My primary rationale for this was imports, especially in the standard library. For example:
import json
json shows up in the __dir__. PEP 387 tells users not to rely on this, but I thought that was basically a workaround, and we needed a better solution.
I tried to be very clear about this in the motivation; what should I change to try to make it clearer?
Ouch. I’m not sure I follow how this disrupts “one of Python’s most unique and idiomatic features”. Could you elaborate on that?
As I noted here, __export__ is not magic. You can get around it with __dict__ or one of a bunch of different hacks. The point is to be more expressive with what a Python module declares as public. (I get that it’s a big change in the sense that Python has never really had “access modifiers”, but try to frame this more as “we want to make backward compatibility easier for module authors”, not “we want this feature from C++”.)
1 Like
I agree with Steve. This feels very like trying to change Python into something that it explicitly isn’t. While I understand and accept the “if you don’t want to use it you don’t have to” argument, at the same time a language is the sum of the features it provides, and this feature sends a message that is contrary to the established policy of openness that Python has had from the start.
I’d like to see concrete use cases. And in particular, use cases that assume that the users of a module expect to use it according to its documented interface, and only access notionally private symbols with an understanding that they do so at their own risk.
I don’t think it’s a workaround at all. It’s normal practice to import what you need freely, just as it’s normal practice for users to stick to the documented API of a module.
2 Likes
I tried to be very clear that this isn’t the purpose of the PEP. I wrote a paragraph about it in the rationale:
Python does not include access modifiers as a language feature for a reason. To quote Eric Smith: “Access to internals of other classes is a feature when you need it”. This PEP does not intend to change this convention, nor should it be interpreted as an indication that Python is tending toward the direction of true access modifiers.
I’m really not trying to add private/public to Python, and I think interpreting this PEP as such completely changes the intention behind the API.
My own use case (and from the others I have spoken to) is with help() and similar functions at runtime. I find it much quicker to call dir() or help() on a module to find out what it exposes. Here’s an example I just thought of:
- Imagine I’m interacting with a module to find the digits of pi. Let’s call it
pi. pihappens to import thenumbersmodule, without any leading underscore.- I call
dir(pi), and seepi.numbers. As a user, if I’m looking for digits of pi,pi.numberssounds like exactly what I want. - I try to use it and get a weird exception at runtime. Yes, I should have looked through the docs for
pi, but that’s not the point! A Python module comes with tons of runtime features to make this kind of introspection possible. If you want to argue that “users should just read the docs”, then I’m open to hearing it, but keep in mind that you’re also arguing against the existence ofhelp().
5 Likes
Could you at least add a section to the PEP explains how this differs from __all__?
After all that’s what people use today.
15 Likes
This might feel a bit abstract and wishy-washy, but let me try and capture it (Raymond Hettinger is better at putting this into words than I am, but I’ll do my best). I’ll almost certainly describe things that aren’t some peoples’ experiences, even some people who have been highly involved in this community for a long time[1], but I’ve also spoken to a lot of developers with a lot of language experience and think I’m talking about something that’s at least more common with Python, if not representative of it.
Writing Python should be close to a “zen” experience. At its best, when you get into the flow state, the thoughts you are thinking about the thing you are trying to build are shaped like Python code as you go through the process of writing it down. It’s often said that Python “fits in my head”, sometimes merely referring to the number of rules, but usually it seems to mean that the way a Python program is structured aligns with how the developer’s memory retains the information, and so they can fit a bigger program in because it’s already the right shape.
Bear in mind that Python pre-dates code editors with automatic completions. So nobody coded by pressing a dot and scrolling through a list of every name that the computer can find - to be efficient here, you wrote code that flowed fluently, so you didn’t have to look it up every time.
This flow is the critical thing. Python is popular as a prototyping language. Why? Because it’s great for one person to flow through the design of their program very quickly and fluently, and (here’s where this starts becoming directly relevant) part of that flow comes from the lack of boilerplate decision making.
Try and flow the same way in Java, C#, Rust, C++, or virtually any other significantly structured language and you will find yourself interrupted every other line with a decision to make. Should this be public? Private? Protected? Friend? Static? Global? RAII? GC? Which type? Will it convert? Will it cast? etc. You simply can’t flow.
Python is such a beautiful exception here, because you don’t have to make these decisions at every point. That’s one of the things that many of us love about it, and I’ve come to find it’s the primary reason for me.
I can take the same base concept in a number of different directions as well - exceptions work great for similar reasons, the standard library works great for similar reasons, functions over methods work great for similar reasons. They all lead to a really nice flow that other languages don’t even attempt to replicate.
And so proposals to add things that are explicitly about decisions that really shouldn’t need to be made have a much higher bar to cross before I would consider them beneficial. As others have said, __all__ or del <name> already exist, if you find there is some need to do this. I almost never have needed them.[2] I almost never have seen a scenario that needed them. And I really really want to see cases where other people have needed them before endorsing the need.
7 Likes
You could also have done help(pi.numbers). Or just help(pi). After all, if the module help doesn’t very clearly explain how to get the digits of pi, that’s a fairly obvious failing in the docs.
As a counterpoint, I regularly use dir to find details of the internals of a module, with the explicit intent of either understanding details about the implementation, or using those internals[1]. This PEP would encourage[2] module authors to prevent me from doing just that.
So either way, some group of users is inconvenienced. And in that situation, status quo wins even if no other argument against the PEP is compelling.
4 Likes
I like the proposal. It helps prevent accidentally relying on implementation details. As a library author, I have a use for this. It’s not as extreme as the jax example from the other thread where lacking this, they have a really long import name spelling out you shouldn’t be touching the things inside of it. My use case here is being able to more freely change my own code without breaking users. I have a documented breaking change policy, including what counts as one, how they will be announced, etc. But at the end of the day, any exposed detail may end up relied upon by someone.
I don’t view this as “against python” either, I personally actively encourage people to just vendor it or ask me to support something if they need it supported. This is just letting the language help express the social boundary without someone having to go find and read through whatever project’s relevant policies might be. All of the normal “Scripting/prototyping” language arguments still have access, just indirectly, and I’m fine with that.
7 Likes
quoted here for those who didnt read the prior thread: some libraries already protect their internals because it helps with managing supporting users. Is the above really a “zen state”, or something people do because of the absence of something better?
4 Likes
I wholeheartedly agree with you that this is one of the many things that makes Python awesome. I just think we don’t see eye-to-eye yet on how this will be used.
See my previous comment for my own personal motivation. Based on the original discussion thread, there are definitely libraries that want this; JAX has been doing some nasty things to try to prevent people from touching internals. My problem with the existing solutions are:
__all__doesn’t have any intrinsic relation to__dir__. “Unexported” (as in, not in__all__) names are still listed. And anyway, if you access a variable not in__all__, there’s no way to realize that you actually did that. I often use__all__to remind myself what things are for use in my module, and I can’t count how many times I’ve realized later that something meant to be exported wasn’t listed in__all__. Since its conception,__all__has been a setting to control namespace pollution with wildcard imports, and it just doesn’t fit the role of__export__very well in practice.delis painful and separates what should be public/exported from a module. From a maintainer POV, you have to search through a list of everything that isn’t exported, rather than a list of what is, and I think that’s much less useful.
Anyways, with __export__, you do about as much thinking as you would with __all__ (in fact, I expect many users to quite literally do __export__ = __all__). I’m not completely sure how to describe this, but I would argue that you’re not really making a decision every time you add a name. You’re just writing code, and if you decide that another module needs something, you can export it (or not care about exporting at all, if you’re just prototyping). This proposal is also trying to decrease the amount of thinking, since you don’t have to think about “should I prefix this with an underscore”?
In fact, I considered just making this PEP into a flag that made __all__ strict; see this section. If you interpret this as an extension to __all__ and not “access modifiers in Python”, does that make the proposal easier to swallow conceptually? If not, please offer some guidance on where you would like to see this proposal go. I get that we’re basically in the realm of philosophical discussion about Python, but I hope you understand that this proposal does offer real benefit to users (and us as core devs, since it improves stdlib maintenance), and we should investigate ways to achieve that without completely breaking Python’s conventions.
Hm, okay, I hadn’t considered this use case. Though I don’t find this convincing for not having this feature at all – it just seems like another base to cover when designing. I suspect that the number of people who want to prevent users from relying on private details far outweighs the number of people who want to understand internals with dir. We have to make tradeoffs, and there’s basically always going to be some group of people inconvenienced by adding a new feature. For your case, you’d be a del module.__exports__ away from continuing your workflow. Another option is to add an unexported argument to dir.
1 Like
This is where the importance of hearing lots of opinions and finding some kind of average makes sense. Because I hear what you’re saying, and I’ve heard it from other people, but not from most people.
It seems a more common thought process is to think about the code you’re currently writing and not the code that is one day going to use it. Which means decisions about exports are an interruption to those people. I think it’s more noticeable in languages like Powershell and Typescript that require explicit export, and you notice the rate at which people forget to do it because it wasn’t in their mind as they were writing the code.
So my primary point is I don’t want to read your comment, I want to read it in the PEP 
That’s the main thing I’m asking - put the motivation in the PEP, not in the discussion. The PEP text needs to be motivating enough on its own, it shouldn’t force people to go and read all the discussion around it (that’s optional for when the historians need to find the root cause of the decision).
I see Michael’s comment on JAX’s naming as well, and my response would be that this falls under the “wanting to do something else” that I had in my first reply. The motivation is not “I feel like this should be private” - the motivation is “my users are confused about what they should be doing”.
Apparently they are anti-leading-underscore, which is a shame because that’s a very long-standing convention, and using one would likely make their lives easier and bring relief to their keyboards by shortening the names. Presumably they haven’t offered alternative ways to achieve the same thing, if users are intentionally using clearly marked private interfaces, or their developers have had previous poor experiences with users insisting on applying arbitrary conventions (“It didn’t have an underscore so I’m allowed to use it and you have to support me”) and decided to avoid that kind of argument[1].
But also, I don’t think “block internal access” is a useful or desirable language feature. The code is in your own process - it’s effectively your code once it starts running, regardless of who wrote it. Unless the JAX team are going to send an engineer to debug and fix your specific error on your device, they aren’t responsible for whether you touch internals or not. Trying to prevent it really isn’t any more effective than casually hinting with an underscore that this isn’t the API you want.
Only because you asked me to back up my “idiomatic” comment
Why not just follow the conventions? Use underscore prefixed names when we don’t intend for users to use them, and non-underscore names when we do?
And yes, I saw the link to the brief PR discussion from the PEP that mentioned the “transitive imports are not automatically public API in the stdlib” rule. This is where proper PEP wording (and occasional history research into discussions) becomes important - that rule was added far more recently than any of the actual conventions, and it was to formally approve the consensus that we don’t need to treat removing internal imports as breaking changes. It’s not guidance for users on how to interpret a module’s API, and I’m pretty sure we discussed changing all transitive imports to the as _mod pattern and ruled it out because we don’t like to do big mechanical changes like that on the codebase.[2] But the implication of that rule should be that anywhere we touch an internal import, it should be given a leading underscore, because it’s not part of the public API, and the way to communicate that in Python is to add the leading underscore.
(Aside: I’m going to be AFK for a couple weeks from about now, so when I don’t respond further, that’s why. Good luck everyone!)
Which I am incredibly sympathetic to, and have absolutely used passive aggressive naming before to achieve the same thing. ↩︎
I also think it’s a silly thing to demonstrate, since users and library developers shouldn’t be expected to do the same in their code, but the practical argument was stronger than the style example argument so it didn’t need to be brought up. ↩︎
4 Likes
It can be very difficult to anticipate the questions people are going to ask. This is the whole reason we do discussion 
.
No worries. I won’t bother writing some long response this time – I’ll just update the PEP to try to address your comments, and you can re-read it whenever you get back. I appreciate the quick feedback! (Seriously, this has been very constructive so far, which is often not the case for PEP discussions, so thank you.)
1 Like
I’m going to reiterate this point. The PEP doesn’t really have much more justification than “Prefixed names are less maintainable”, which honestly feels like it’s almost entirely subjective. I personally find underscores perfectly maintainable, and the fact that many libraries have been using them for years suggests this isn’t an uncommon position. You don’t even have to prefix everything. Pip uses a private namespace pip._internal to clearly identify private names[1]. And if using underscores for private imports is awkward (a POV I sympathise with) we could easily promote a convention “imports are always considered private unless explicitly included in __all__” (i.e., the convention the stdlib currently uses).
One thing you don’t discuss explicitly (at least, I didn’t spot it if you did) is that this adds overhead to every module lookup. In order to deliver the semantics described in the PEP, every lookup has to now include a lookup of the module __export__ variable, and a scan of that list. That’s not a huge overhead, but it is a cost, and it could affect performance-sensitive code. Furthermore, the __export__ lookup almost certainly can’t be optimised away, because of the usage you described yourself in response to my use case - modifying mod.__export__ from outside the module, at runtime.
So ultimately, this won’t stop a determined user from bypassing the privacy requirements, it adds cost to module variable lookups, and it provides no benefit to code that already respects existing privacy conventions.
I’m sorry, but it still feels to me like this PEP comes from an adversarial mindset where library authors view their users as opponents who cannot be trusted to act in good faith. Maybe that’s the reality you have to deal with, but it’s not how I see my users, nor is it how I want Python users in general to view each other. I don’t find the jax example convincing for exactly that reason. What sort of users do the jax maintainers have to deal with? Why the heck don’t they simply close bug reports from such users, and move on?
Is this an attempt to solve a social problem with a technical mechanism?
Pip has no public API, so everything lies within
_internal, but that’s a separate point
↩︎
8 Likes
Really well put, Steve. I think you captured how I’ve always felt about Python even 30+ years on, and I’m not sure Raymond could have done better 
6 Likes
There are plenty of ways to optimize it. For example, we could look at __export__ once and cache it, but add a dict watcher that forces another lookup if it changes. I just haven’t bothered to implement it yet.
It’s supposed to be the exact opposite; I wrote this with the mindset that library authors view their users as teammates (to contrast with “opponents”) whom they are trying to help use their library. A user shouldn’t accidentally reach for an internal attribute, and users that do want to access internals should be encouraged to vendor it or ask to make it public, not just grab it and hope for the best. The whole reason this PEP makes __export__ easy to bypass is that it is recognized that accessing internals is sometimes useful – it just shouldn’t happen on accident, and shouldn’t be taken lightly when intentional.
I also don’t think _ is always perfectly clear. It definitely depends on the context. ctypes, for example, is chock full of APIs prefixed with _ that are part of the public API (or private APIs that are so widely used that we couldn’t possibly remove them). Granted, ctypes is a bit of an extreme example, but my point is that the underscore prefix is not a universal golden rule.
Because that’s the “opponent” view that you’ve described. Having dealt with this in CPython issues many times, simply saying “you’re not supposed to do that” is often unhelpful and frustrating to users. We want to help users understand what’s private and public, not shrug them aside every time someone gets it wrong. If they decide to access internals anyway, del module.__export__ should make it immediately clear that they’re opting in to breakage.
10 Likes
For what it’s worth, there is a problem here that I think is worth solving.
My IDE regularly recommends indirect imports (such as in this example, where you’re importing json in this module, the intention isn’t to re-export it, so if another file has from foo import json I consider that an indirect import) and if there are good tools to resolve indirect imports I’m not aware of them. (Unlike unused imports and unsorted imports for example.)
TBH I can’t really see myself maintaining an __all__ even if it would resolve this problem. That just seems like too much work to be worth it, even when it’s all happening within apps and libraries I control.
4 Likes
I’m sure they do know about leading underscores, so I’m just guessing, but I imagine the reason they went so far is because the Jax library is used by internal Google users. If those users start relying on library internals, the Jax team can’t just say “well, that was clearly marked with leading underscores, so tough luck we aren’t responsible”. Everyone has a manager and team goals and they aren’t going to let the library change just sink their project because “technically the library isn’t responsible”.
My guess is that the crazy name acts as a giant neon sign to any reviewer of client code to reject such uses. Just a guess.
2 Likes
This feature addresses the common library desire to have code laid out as a tree S, but to expose a tree T. I think it might really help to understand why libraries go through all the trouble to do that.
Yes, sure, if you’re just zennily writing a small library, you maybe don’t care. S=T for you.
As libraries get bigger, the constraints placed on how code is laid out can easily become disconnected from how that library should be presented to the user. Then, it becomes a necessity that S ≠ T.
Now, how should libraries make that happen?
I don’t know if this is the right proposal for that. But it would be nice to have a clean, easy way to specify T. Here’s what I ended up doing in my code.
It is annoying because it means that whenever I add something to the code, I have to keep T in sync.
Just to be clear, I don’t think that underscores are a reasonable solution because in a large library underscores are often used to indicate objects that are private to a module.
Using __del__ is significantly more confusing and annoying because you have to remember to delete all kinds of things. Even from .x.y import z pulls in both z and x.
Using __all__ is fine, but it means that S and T overlap in the source, which confusing for the library-writer.
I’m not sure what the best solution is. I think the solution I copied from Jax is the nicest. But maybe in a more ideal world, you would have something more compact.
3 Likes
For me I get almost all the gains this PEP describes by using my atpublic package, and I think with a few tweaks, it could for you too. At least, you should consider it in contrast to PEP 842, and potentially where – if 842 is eventually accepted – it could help with better UX.
TL;DR: atpublic gives you two decorators @public and @private. The former is used way more often than the latter. You can do something like:
# mymodule.py
from public import public
@public
class MyPublicClass:
pass
@public
def my_public_function():
pass
There are a few key benefits:
@publicis clear documentation about the intent of that name’s visibility.- The visibility intent lives with the definition, not in a distant part of the file.
__all__never gets out of sync, and I’m sure we’ve all experienced that.
The module provides other APIs and benefits, but y’all can read the docs for that, and I don’t think it’s particularly relevant for this discussion.
As @steve.dower points out so eloquently, Python’s flow state means you often[1] don’t actually care. But I think sometimes you do need to be more explicit about intent, and I’ve found @public is useful for that. The project came out of the GNU Mailman work, where you had a big pile of Python, many APIs both public and private, and __all__ often got out of sync. But I still contend that the documentation aspect of @public is really the most important piece, so the enforcement side of PEP 842 is less interesting to me.
I once semi-proposed bringing @public into builtins, but there wasn’t much appetite for it at the time. There should be little worry about the performance of a decorator that tweaks a module global and then returns the object unchanged. I once implemented it in C and it’s overhead at module import time was not measurable. Maybe lazy imports would make it more appealing too.
Aside: you’ll soon be able to install public and private in builtins through an extra: pip install atpublic[install]. The optional extra adds those functions to builtins through PEP 829 .pth/.start files.
The one thing that PEP 842 proposes that atpublic doesn’t provide is attribute access blocking. It probably could without a ton of work, although I’d have to look at how 842’s reference implementation works. You’d still get the public names in __all__, but it also wouldn’t be much work to support a __export__, which if PEP 842 is ever accepted, I’d likely add anyway.
usually? ↩︎
2 Likes