And, to be clear, there’s nothing wrong with that, right?
3 Likes
Absolutely not. I love extension modules. I’ve written and published a couple myself, and I think Python is better because of them.
What I’m trying to argue is that controlling what you expose is already something thousands of Python packages can do because they’re implemented as extension modules, and I think that’s a great thing. So why shouldn’t pure Python packages have the same capability? 
I don’t think anyone would argue that extension modules are somehow “unpythonic” just because they don’t let you poke around in all of their internals.
7 Likes
Could you explain how this is an “unreliable runtime warning”? What makes it unreliable?
Sure there is – you just wrap the call in with warnings.catch_warnings, or use the __dict__ if you really want. I’m having a really difficult time grasping why you’re a fan of the exception but so opposed to a warning.
Neat, but keep in mind that this solution is somewhat insane. It’s on the same level as writing a custom codec to implement new syntax. I don’t find this example convincing for the idea that this PEP “isn’t offering anything new”.
Try to think about this in context a little more.
A user has spent a few months developing a small app, and in it, they’ve used the internals of your library. They weren’t necessarily trying to; they typed something.name_they_wanted and their editor found something._name_they_wanted. They’re already using sys._current_exceptions (which is a documented, public API) in their codebase, so a leading underscore doesn’t mean anything special to them. Their code works!
Then, say a few months later, they upgrade your library. All of a sudden, their code is broken. They track it down to a change in something._name_they_wanted, and file a report upstream. Instead of being told that their code will be fixed, they’re instead told that their code was never right in the first place (citing the leading underscore in _name_they_wanted, which they already find confusing due to the leading underscore in plenty of public APIs) and the only fix is to switch to a different public API that requires a lot of refactoring across their codebase.
Had they known _name_they_wanted was private (or seemingly didn’t exist, since the editor doesn’t suggest it) when they first added it, they could have made a quick search and found the public API when the code was much smaller. But now, they have to either stay on an older version of your library, or put in a lot of effort to migrate their code to the public API.
I hope you realize that this isn’t hypothetical. There’s a reason CPython has so many public APIs that were once private – people found them, used them, and we decided that the best course of action for users was to simply maintain them as public instead of breaking user code.
This isn’t supposed to be secure. People like to use internals for all sorts of monkeypatching. We still want to support that.
The point is to make it explicit that something is private, because I don’t think underscored names do that job perfectly.
Sure, and this is close to JAX’s internal_do_not_use naming, but that’s not nice to the maintainer at all.
And this is still possible. This PEP doesn’t prevent that. Again, this is not “access modifiers in Python”. You’re still free to explore a module’s internals if you want; you’re just not prone to making the mistake of reaching for an internal name without realizing that it’s internal.
This proposal isn’t supposed to remove the underscored names convention. It’s supposed to clear up the cases where users either aren’t completely acknowledging them or their meaning it ambiguous. I wrote a paragraph about this in the PEP:
even if this PEP is accepted, it’s expected that “underscored” names (names prefixed with a leading
_) will remain a staple of Python for years to come. The purpose of this PEP is not to eliminate the need for_in module-level names, but instead to clear up corner cases where a private name is ambiguous or tempting. In other words, this PEP is intended to improve expressiveness and clarity with private APIs, not to add brand new functionality.
I also wrote a section about this. I strongly believe that assuming every developer reads the complete reference documentation for something before using it goes against Python’s “flow” that others have described in this thread.
Another problem with the “read the docs” argument is that private things usually aren’t documented. The user has to just know. There’s nothing telling you not to use the standard _colorize module – you just have to guess because it’s prefixed with an underscore. You might say that Python should clearly document that modules prefixed with _ are private, but oh wait, _thread is public!
Agreed, I was kind of thinking out loud. The more general issue here is whether we want private names to be local to a module. For example, if I have a _utils module, I only want the contents of it to be accessible from inside my package. That’s not solvable with the current PEP, and I’m trying to come up with solutions.
2 Likes
That’s on the user side, not on the warning side. Wrapping with the context manager has other issues, as it can catch things in code that you didn’t intend on catching (code that you’re calling calls other code that has a warning), and is one of the ways users wouldn’t even see the warning. The correct way is a warning filter narrowly tailored to the exact case, which is difficult for users to get correct, I mentioned as much before.
The ideological problem I have with with the warnings is that users shouldn’t get warnings for things that aren’t actually issues. It’s not an issue for someone to use an internal, it’s just something they are supposed to be intentionally doing and aware of such. I explained in more detail in an earlier post about this.
OK, thanks. That example helps me understand your reasoning. I still can’t say that I’m in favour of the proposal, but at least I understand why we differ on what constitutes an “adversarial” approach. I shouldn’t have used that term at all.
Let’s focus on the technical aspects of the proposal - there’s plenty to deal with around how to mark symbols as “private”, around what constitutes acceptable use of private symbols (versus what should be flagged), how to “flag” unacceptable use in any case (error, warning, linter, something else?) and how to bypass the restriction and say “I know what I’m doing, so just let me go ahead” (if indeed we want to allow that at all).
But it does put the burden on the user to add what is essentially boilerplate that adds no functionality, but simply says “yes, I know what I’m doing is naughty, and I accept the risk”. In some senses, it feels very like the sort of click-through terms and conditions that no-one reads. Hmm, wait…
And yes, I know the analogy isn’t perfect, but realistically, I’m pretty sure that it won’t be long before LLMs learn to just add requests.__exports__ = [] at the top of their code, and StackOverflow answers appear (and get copied blindly) saying the same.
4 Likes
But you think users should get exceptions for things that “aren’t actually issues”?
I see your point, but I think it’s also very difficult to predict the future here. I would like to think that people won’t just put the escape hatch everywhere on the basis that it’s a code smell, but practicality does beat purity. Do you know of any history here? (Not necessarily with private attributes, but any case where a language added friction to prevent something and everyone just worked around it.)
2 Likes
I’m “okay” with that, because it is an issue with the exception version. That version it isn’t code that will work at runtime, so the user needs to make a decision on if they want to accept working with an internal and all that entails, and the decision is reliably made, not neccessarily supressed accidentally (well, other than if someone else modifies another module’s __export__, something I don’t think should be allowed). It doesn’t ever fly under the radar in an update, it doesn’t break some users but not others based on their CI settings or what other libraries have touched with warning filters, or catching too broadly by using the context manager, it’s consistent.
I’m also fine with just export as documentation of public api without a warning or exception. I think it’s totally reasonable to say __all__ isn’t __export__, but we’re fine leaving any enforcement to static analysis/linters and other tools. (I think a better name in that case might be __public_api__)
1 Like
Nothing directly comparable. But there’s plenty of code with # noqa and # type: ignore annotations all over the place. Why didn’t they address the problem rather than working around it? And deprecation warnings have a terrible history of trying to add friction to using constructs we want to remove, but simply ending up either annoying people or forcing them to suppress the warnings.
The problem here is similar - we’re adding friction to prevent people doing something (use private attributes) but we aren’t offering a better alternative. So what are people supposed to do other than work around the issue?
Often people use pip’s internals because the supported alternative (run pip in a subprocess) is too slow, too clumsy, or otherwise more difficult than using an internal API. I’m not sure that making use of the internals just as slow, clumsy and difficult is really the right answer here…
1 Like
If we’re going to go the soft keyword route, then something like this is more succinct:
from X export Y
6 Likes
I meant, the best we can do in current Python.
1 Like
I think the user experience of warnings is terrible. Python users are already very familiar with exceptions. try/except is basically Python 101, and it’s easy to apply at whatever granularity you need, from a single line to an entire application. The same isn’t true for warnings.
I think most users will eventually end up with a warnings configuration that silences more than they intended. I don’t think we should be designing a feature where the recommended workaround is “go configure your warnings filters.” Warnings are easy to ignore, easy to accidentally suppress, and many users don’t really know how they work.
2 Likes
I’m also going to point out again that I think people will generally fall into two camps:
- People who want to enforce their public API, and are willing to adopt
__export__because they want an error. - People who think the whole idea is unpythonic, and will just continue using underscores and
__all__without ever bothering with__export__.
The runtime warning option doesn’t really satisfy either group. At best it serves a small minority, and at worst it serves no one.
1 Like
This example seems very artificial to me:
On the one hand, this hypothetical developer is experienced enough to write a non-trivial app that includes threaded exception handling with sys._current_exceptions()—on the other hand, they have never heard of the very common convention[1] that leading underscores typically indicate internal code?[2]
On the one hand, this hypothetical developer knows my library well enough to use it so extensively that moving away from the internal APIs would require a lot of refactoring—on the other hand, they never even bothered to look at any public docs about it but only ever blindly guessed at function/variable names?
I’m willing to belief that I’m seriously underestimating the lengths that people will go to in order to avoid reading docs. Maybe there are actually a significant number of these hypothetical developers out there. If there are, though, and they are so determined to ignore all existing hints, I’m not convinced that the additional hints in this PEP will make much of a difference.
Personally, in the small packages I’m maintaining, I’ve found a mix of
- the leading underscores naming convention,
- not including certain functions/classes/submodules in the public API documentation and
- in rare cases, explicitly stating in a docstring that a certain API is considered unstable and may change at any point
to be sufficiently clear that I’ve not encountered serious problems with users relying on internal APIs. Accordingly, I don’t see any use for __export__ in my projects and would likely not use this, e.g., due to the DRY overhead already mentioned by others. (I can see how it might be useful for other projects who need to be more explicit about distinguishing their public/private APIs, though.)
However, I see a potential social problem:
If something like __export__ is adopted, I worry that some fraction of users would see this as the only mechanism for documenting which APIs are non-public; and that they then treat all my underscore-prefixed, undocumented APIs as public and overwhelm me with bug reports if I ever change them, thus effectively forcing me to adopt __export__.
(And notably, it wouldn’t be enough for me to adopt __export__ after I get a wave of these bug reports—by that time it’s too late and people have already started relying on internal APIs. I would be forced to adopt that as a precautionary measure, as soon as the PEP is accepted.)
which is even documented, to a degree ↩︎
Even the
sys._current_exceptions()docs state that this function “should be used for internal and specialized purposes only”. ↩︎
5 Likes
It’s really not. Several examples have already been shared in this thread of both breakages as a result of relying on internal APIs and how libraries are going to reinvent __export__ regardless.
I don’t think this PEP introduces a fundamentally new capability, and that’s a good thing. Libraries could already hide their internals, raise errors if you access their internals, and block you from messing with their internals. They just do it in an ad hoc manner because they have a need for it.
4 Likes
- I think you’re significantly overestimating how often people ignore warnings.
- I’m not really worried about accidental suppression, especially if we add a new
ExportWarningcategory. If we really wanted to, we could add awarnings.allow_private_namescontext manager or something like that.
I can’t speak for others, but from my personal experience, I typically use these when Python’s type system can’t express what I’m trying to do, so I’m not sure it’s a super comparable example. This is a good discussion point though.
To be straight: warning or not, if it’s clear that there’s a large number of people who won’t adopt the idea because they consider it “unpythonic”, then I’m going to withdraw the PEP. One of the great things about Python is that its feature usage is generally consistent – people don’t just disallow certain language features in their codebase because they don’t like them. I’m not at all comfortable with introducing that.
sys._current_exceptions was a random example I pulled out of the documentation. My point is that there are plenty of public, stable APIs that are prefixed with a leading underscore that are encountered without being an “expert” in Python.
Yes, which is why I’m going to change the PEP to use decorators and an export keyword.
I think we’ve come full circle on whether __export__ (on its own) is a good idea, so I’m going to stop replying to comments solely about it. I need time to write the new revision.
1 Like
What do we need decorators for here?
I think decorators look nicer for functions/classes.
@export
def hello():
...
vs
export def hello():
...
I think the fewer keywords we can have on a single line, the better. I’m open to convincing, though.
My thinking is the exact opposite. Also, the export keyword involves much less fuss: you don’t have to think about decorator ordering or resolving the name. And you still need the soft keyword anyways for re-exporting imports, so the decorator approach is just inconsistent.
1 Like
You want more keywords on the same line?
I find the consistency argument convincing, though.
Good idea. My anxiety about this thread has been steadily rising because of the amount of quick back-and-forth that feels like either talking past each other or polarization (witness the quoting levels).
We also don’t need new voices in the discussion that simply state in their own words which side they support.
I think coming up with the ideal syntax will be relatively easy.
Regardless of syntax, the most difficult open problems seem to be around the user experience when accidentally accessing an unexported name (warning? exception?) vs. intentionally (monkey patching, missing public API, tolerated use of internals, etc.); and how to support internal import of non-exported names. Also how to handle the transition with minimal breakage without weakening the feature.
3 Likes