Speaking as a user, if I develop a script in using standardlib internals[1] in Python 3.X and they’re broken in Python 3.X+1, that’s entirely fair game. We’re all consenting adults. You allow me to access those internals because I’m an adult[2], and I appreciate that. And if my code doesn’t work anymore in 3.X+1 because you decided to change the internals, I won’t complain about that because you’re adults[3] too.
The imports do complicate the matter. I still think import foo should make foo a “private” symbol insofar as that is possible. Don’t include it in dir for example.
The names like NamedTuple._asdict() also confuse the matter - I would consider these part of the public API even though they start with a _. Naively I would suggest these public API symbols that start with a _ deserve an alias that doesn’t start with a _.
If you gave me a time machine, that would be one of those cases to fix by not making it a method in the first place (instead, functions in the related module for accessing things like this).
It’s a reasonable example for why _ prefixing can’t be a one-size-fits-all answer though.
Just because you can does not mean that you should. How often are you coming across things that are truly only available in the internal API? (And if you are finding things often, why aren’t you telling us that and asking for public alternatives?)
Python is notoriously annoying to upgrade, and part of that is because we change the internals. We should aim to make upgrading easier, and this is a step in the right direction.
I agree with your general idea just from a “cleanliness” perspective.
One of the beautiful things about the Array API is how much shorter dir got with the Array API versus the old Numpy API. It’s really a breath of fresh air.
IMO, the standard library should expose exactly its public symbols and nothing else.
One way to design a new dunder __restricted_export__ = True would be so that its presence in a module:
automatically interprets all of the symbols in the module some_parent.some_module as if they were declared in some_parent.some_module._src
creates a module some_parent.some_module whose symbols consist of exactly those in __all__ and that points to the corresponding symbols in some_parent.some_module._src.
Essentially, this would be as if you had made the hub with none of the pain of making the hub. This:
Requires minimal effort for developers
Ensures that absolutely nothing is exported except the intended symbols
It’s also easy to phase out if this ends up being a bad idea since linters or LLMs can easily turn your “pseudo-hub” into a real hub with guaranteed success (assuming nothing is done conditionally).
It does not add things to __all__ for you though. If you need that, I suggest accepting @barry 's PEP 844 or simply using a package like it.
@ZeroIntensity What do you think? Would that solve your problem, or is there still something missing?
I’ve been thinking about this some more and have realised why the problem as stated seems so far from my reality, what is stated as “isn’t working” is in my experience very much working. 95% of my time is spent using modern tooling which have come a very long way in the 25 years I’ve been using Python.
Sitting in any environment which is using an up-to-date LSP / type checker provides a lot of red flags when non-public, by the standard rules, ie. _ prefix, not in __all__, are accessed, and given that baseline (most of the std lib modules) use of a non-public attribute really does feel like the person had to actively ignore what they were already being told.
Now that doesn’t address either the standard repl or run time enforcement.
If the standard repl can be made LSP/type checker aware (quick and dirty experiment with ipython below) then the repl gets those red flags. The repl has had some wonderful improvements lately around error messages which kind of fit a similar space.
Take the case of a non trivial private function we’ve determined we want to use as it’s not exactly available any other way (the actual function doesn’t matter, just that we need it for something)
In [1]: %load_ext ty_ipython
ty checking and Ruff wildcard-import checking enabled for Python cells
In [2]: from inspect import _shadowed_dict
<ty-ipython>:1:21: error[unresolved-import] Module `inspect` has no public member `_shadowed_dict`
In [3]: from inspect import _shadowed_dict # ty: ignore[unresolved-import]
In [4]: _shadowed_dict(str)
Out[4]: <object at 0x100cf8a80>
In this world dir does show _shadowed_dict but even if that was not shown inspect.__dict__ would show it, and source shows it, so there is some way to see it. Where is the line drawn? How many red flags are enough?
If no number of red flags is enough then in a world with runtime enforcement of “no you can’t have _shadowed_dict”, we are stuck duplicating non-trivial code which otherwise works perfectly well for our case, has gotten changes across python versions, and we’ve signed on to the risk, through ignore[unresolved-import]. This is the part of current discussions which gives me most concern.
We live (happily?) with an entire type checking system which has no impact at runtime and I’d suggest that typing errors are far more problematic than using a non-public import. Type errors are bugs in the program now, non-public imports are potential bugs which affect only when you upgrade and are so somewhat more controlled.
Do private imports, accidental or intentional rise above type checking?
I think we should just clearly document that these introspection features cannot be used for that, and that the way to determine what the public API is is to read the actual documentation.
I’m all for being nice to users, but the gist of what you’re suggesting seems to me a rather narrow of that. Lots of users are surprised that slices don’t include the endpoint; does that mean we make it start including the endpoint, or provide some kind of alternative slice that includes the endpoint? Lots of users are annoyed by having to type self in all their methods; does that mean we come up with some sugar to hide it? Lots of users think the logging module has a confusing and inconsistent API; does that mean we’re going to overhaul it?
Being nice to users does not simply mean accommodating what they happen to be doing. Niceness is as much in how things are said as in what is said. It is possible to nicely say that what a user was doing is unsupported and so unfortunately their code will break.
I think it’s perfectly possible and reasonable for “niceness to users” to take the form of clear documentation and consistent guidance nudging users to use that documentation. There seems to be a trend towards various means of converting documentation into some kind of machine-readable format that tools can use to do various things for the user without their having to read the documentation. Such things have their place, but I think more and more we are going a bit too far in that direction. No one should think they can successfully use any library without looking, with their human eyes, at documentation intended for human eyes. The extent to which they can succeed without doing so may vary from one library or task to another, but it just seems futile to me to go down this road.
This gets at another related issue for me. If we want to make Python less annoying to upgrade, great! Maybe we can do that. But we can’t do it with this proposal, because, as you say, only part of that annoyance is due to this particular detail of private APIs — and I would say it’s a pretty small part. There are plenty of backwards incompatibilities that are made “in the open” with knowledge that they are changing the API (which is why there’s such a thing as a backwards compatibility policy at all). There are, in my view, some that have been particularly egregious examples of failure to honor the existing policy, notably the JSON number limitation. I think a fair number of users will see changes like this proposal as meaning something like “We’ve decided to make it a little harder for upgrades to break in this one particular way. . . but it’s still going to be really easy for them to break because of all the other ways they break”, which is pretty cold comfort.
If the goal is “make upgrading Python smoother”, then I’d say let’s adopt that as a goal, and comprehensively consider how we can move toward that goal. That would likely require a range of efforts across many dimensions of Python (for instance, a more conservative backwards compatibility policy). But I think it’s a mistake to attempt piecemeal changes such as this without an ongoing, across-the-board commitment to prioritizing upgrade woes. Otherwise, whatever gains are made by this change may just be canceled out by some other decision that makes things harder later. The right goal (if we want to set one in this area) is a reduction in the overall net pain of upgrading, and we can only achieve that by giving it additional weight in every decision that is made about Python.
I can’t remember ever needing something that was only available via the standardlib internal API. I think once there was something that I did using the internal API because that was the first solution I found, for a throw-away script. It was code that can’t be expected to be maintainable, but I was nonetheless glad that Python allowed me the freedom to write it.
(I have needed something from the Pandas internal API somewhat recently, but I’m pretty sure that when they change the internal API I’m (ab)using, they’ll also ‘fix’ things so that the thing I need becomes public.)
+1 (and to the rest of Brendan’s post, but especially this bit)
It sounds like a __help__ module/class attribute used by the help() function to restrict the list of members shown is potentially the way forward here. Historically, that would’ve been in __doc__ as prose, but if we need a machine-readable format then another list of strings representing “these are the members you were looking for” intended as a convention for the tools under our control is a very easy and safe way forward.
It’s also possible that we could simply align on “having a docstring[1] means it is public” and use that to filter help() output.
Any change here with an impact on the stdlib is going to require a monumental review and enhancement effort to produce a result. If that effort means that all public APIs get docstrings and all non-public APIs with docstrings are explicitly marked in the docs, that just makes things better for everyone. An effort that does the bare minimum to “protect” ourselves isn’t actually that helpful to users.
That doesn’t explicitly say “internal” or “unstable”. ↩︎
This thread isn’t PEP 842. I’m not formally proposing anything yet. I’m trying to gather ideas on what we should do; there is nothing to object to yet, apart from the various mini-proposals that have been brought up in this thread. But if you are referring to one of those, I have no idea as to which.
What I will say is that “tell users to read the docs” is a cop-out argument. I have gone into immense depth on why that doesn’t work in practice, and I suggest reading the rest of the PEP 842 discussion before continuing to push that.
Sure, that’s part of the goal (in addition to reducing core dev burden). If you want to propose other ways to help with this, I’m all ears (though I don’t think a more conservative backward-compatibility policy is the way to do it – if anything, that will make the problem worse).
Sorry, calling that a cop-out argument is the cop-out argument.
We simply can’t assume that our code can be designed in a way that doesn’t require users to learn how to use it. As much as I’d love that as an outcome, it can’t be mandated, enforced, or achieved on the scale that the stdlib has to cover.
Reading the docs to learn about the APIs is essential. Needing to re-read them every time you use them would be a design flaw, but the first reading is not. We simply can’t trust that their use will be guessable, and more importantly, we can’t treat them as supported APIs under those circumstances.
I think the overall “user won’t read docs” is a failing argument, but I agree we shouldn’t be using this as the only way to convey the public api. a key point here is that users shouldn’t have to reference rendered documentation or the files and directives configuring a documentation generation tool while looking at the code, IDLE/REPL output, or IDE completions to determine if something is public, yet that is the current status quo.
People design APIs to be intuitive. If something is placed in the right spot and named well, users can infer what it does without jumping to the docs. (Are you really going to go read the docs for something called set_name?)
I think it’s quite common for users to take an example from the README, start using it as a “hello world”, and iterate using autocomplete/dir() (or in today’s age, LLMs) from there.
This is the point that I’ve been trying to convey over and over again. If you see something that looks like what you want, the underscore prefix is often just not enough, especially when people call dir() or things like that.
I realize you were AFK for most of PEP 842, but I’ve explained myself here many times already, so I apologize for my frustration with this.
Agreed. To me, a large portion of this discussion seems to come down to wanting very clear boundaries on what the core devs are allowed to change and what we’re not[1]. Enforcement is secondary - we only need enforcement if we can’t trust our users to follow the policy. And trusting our users should be fundamental here.
Having clear boundaries is IMO a good thing both for the core devs and for the users. It matters very little how we establish those boundaries, just that everyone should be clear on what they are.
Personally, I thought the existing rules were clear enough. But the examples @ZeroIntensity has presented suggests that probably isn’t true (at least, not all of the core devs interpreted the existing rules the same way that I did). So let’s improve them.
But let’s start by just defining clear rules, and not leap straight into trying to add enforcement (whether it’s by actual errors or warnings at runtime, or just “making it harder to access private details”).
After all, even if we do define some explicit mechanism, we need to know what to apply it to, and the rules are what we can use to determine that.
I wonder. Is an unstated part of the issue here that LLMs don’t read documentation, and will use anything that works? Or maybe less controversially, that IDEs don’t read documentation and offer private APIs on an equal footing with public ones? Even dir() counts as a (simple) IDE-style tool in this sense.
Because in that case, my view is that (1) the tools need to improve, and (2) users working with tools that have flaws like this should understand that it’s their responsibility to check what the tool is offering. Basically, “my tool told me this API was OK” is never a valid argument.
At least, not without following our deprecation policy ↩︎
I think the whole “can we trust users” (whether it’s to read docs, or to respect signals that an API is private) is becoming a little too heated. Can I suggest that we drop it for now, and give people time to cool off.
While that’s happening, maybe we can focus on how we decide what is a public API in the stdlib and what is private implementation detail? Because the evidence is that we don’t all currently agree on even that.
Been there Your frustration is barely showing, btw, you’ve got a lot further to go before you start being accused of being cranky
I’m also aware that I missed most of the discussion, so I’m intentionally trying not to re-litigate issues that I’m sure came up (while also preserving my own sanity by not actually reading that entire thread - if GitHub wasn’t down right now I’d probably even be doing useful work instead of catching up on dpo ).
Totally agree, which is why all my suggestions were things that would be accessible at runtime. Back when I worked on one of the earliest IDEs that was exposing this information to developers in real-time we had these kinds of discussions all the time.
The unavoidable premise is that the docs are the definitive source of truth on what’s “public” (aka. intended to be used), because that’s how it’s always been done. If a user doesn’t follow the docs, they’re out of support, and we aren’t obliged to treat their issue as a bug (because, e.g., they used something they weren’t told to and we broke it in an update).
The current problem is that a number of rules have grown up around that, which seem to keep being thrown around as the definitive rules, when they just aren’t. We’ve got docs and years of convention, which sometimes get codified into rules, but we also don’t like rules so the rules get exceptions (like “we should use a leading underscore for private, but because we haven’t always done that you shouldn’t assume the absence of a leading underscore means public, and we also don’t want a drive-by PR that renames everything to add an underscore so don’t do that either”).
So as Paul says, we need the clearer set of rules, and it’s likely more of a design manifesto than a simple easy-to-follow PEP. It’ll require a lot of research into discussions from the past to get a good, unbiased read on what was intended by the things that did get written down, and distilling that into something that is complete enough to be read and followed in isolation. The great thing is that virtually all of these discussions will be in mailing list archives, because we did so little in person until relatively recently. Endless amounts of Guido’s wisdom (from when he was the final decision maker on everything) is right there waiting to be collected
My experience has been the opposite - the LLMs do read documentation and docstrings and follow them. IDEs certainly don’t do a lot to filter, other than the leading underscore, though we were adding all sort of statistical methods to Visual Studio (and presumably VS Code, though I haven’t been as involved there) to prioritise the members that people actually use over the ones that aren’t, and those tended to be more than good enough. We invested quite a lot in trying to filter in ways that helped our users, whether the library developers had done it or not, I don’t know that we ever saw users blaming us for showing them members they weren’t meant to be using, but maybe that’s changed in the last decade.
We often say that the public API (of a Python package) is what we say it is (read, what we document and/or communicate). So, quick idea: why not reusing what’s already documented as public API → the standard lib’s objects.inv, generated by Sphinx? https://docs.python.org/3/objects.inv
Teach linters to load that, so they can warn when the user accesses a non-public symbol (from the py domain). That also forces the authors of the stdlib docs to make sure everything that should/must be considered public is documented, and that nothing that couldn’t/mustn’t, isn’t (at least not with an entry in the objects.inv file). The documentation truly becomes the source of truth. No runtime enforcement though, so this could be deemed insufficient still.
The nice snowball effect is that this works for other libraries too, and so gives incentive to provide an objects.inv, and therefore proper API documentation. The hard-part is knowing where to find the objects inventory given a package/distribution name, but with PyPI metadata that might be easier than thought.
).