I think this is a good solution for big libraries, but the downside of this is that it requires at least a directory for every module that wants to follow it. My primary motivation for this PEP was the standard library, and I don’t think it’s a good idea to turn every lib.py in the stdlib into lib/_internal.py and lib/__init__.py.
Either way, this has the same indirection problem as __all__ (and __export__), because you’re declaring that something is exported far away from the definition. I’m becoming more and more convinced that @export/export syntax is the way to go.
So, before I pursue that, what should the defaults be (as in, exported by default or unexported by default)? Should @export work in classes too? What about attributes? (I mean this as a discussion point for all readers – I’m not personally asking you to design the entire proposal )
I think we just need a simple way to make it raise an exception.
Does this need to be something that’s enforced by the runtime? It’s this enforcement that doesn’t feel like something that belongs in Python to me.
If the intent here is to indicate to developers that something is not supposed to be used then the most useful place to see that is when writing the code rather than running it.
If this changed to be something intended for static tools this could be applied using a special annotation, a decorator or something like the convention that’s applied to stub files for imports. This could also cover the difference between something being internal to a file, class or project.
I think adding __export__ to existing stdlib modules will probably just lead to unnecessary churn and breakage downstream.
I like the export syntax too . One argument for the soft keyword version:
For a lot of libraries, you want to declare that something is exported far away from the definition because you not only want to say that the thing is exported, but also to say where it’s exported. For example, numpy’s __init__.py both gives all the exports, and tells you where they’re being exported. Similarly, it’s numpy.typing.__init__.pygives the things exported there, etc.
When I proposed the export soft keyword much earlier, it was motivated by wanting to make it so that these __init__.py files don’t have to repeat the import with populating __all__.
Both the export decorator and soft-keyword help you in the case where you “export in place” from the definitions themselves.
But if you want re-export in __init__ by importing definitions, then the soft keyword would eliminate the population __all__, which is just nice.
SciPy once exposed NumPy modules at the top-level, such that scipy.random was an alias for numpy.random, and so on, before being deprecated and removed:
scikit-learn once vendored six and joblib and downstream packages imported them from sklearn.externals.six and sklearn.externals.joblib, which broke when removed in sklearn 0.23. Some discussion threads advised workarounds like pinning scikit-learn==0.20.3 or sys.modules['sklearn.externals.six'] = six to keep old code running.
I think the dir()/help() enforcement is useful to have at runtime. I could see arguments against a warning upon access, but I still believe it’s worth having because it seems possible that people would put # noqa without actually understanding why the linter is angry at them. I’m open to convincing, though.
How would users be broken? The PEP only specifies a warning. If they’re doing -W error, then they’re opting into breakage anyway because of potential deprecation warnings.
Currently, I’m leaning towards a new design for the PEP with the following things:
export syntax for variables, based on Hugo’s earlier suggestion. For example:
from foo import bar
export bar
export spam = 42
@export (module-level names as private by default)/@private (class names as public by default) for classes and functions. For example:
@export
class Public:
def public_method(self):
...
@private
def private_method(self):
...
__export__ as specified in the PEP, as a backwards compatibility shim.
What do others think of this design? Too much/too little? Better fit for the type system only?
Although I think most people will either use __export__ because they want it to be enforced, or not use it at all. I’d expect only a small minority of cases where someone wants an access modifier that’s merely a warning rather than an error.
I don’t know. I can’t speak for them - maybe it is. Or maybe they’ll just monkeypatch __exports__ and avoid changing all the rest of their code. The point is, I’m not sure that as a pip maintainer I want to impose that cost on them. After all, while it’s a pain that people ignore our statements that pip has no public API, it’s not that much of a problem. As I said earlier, anyone who raises an issue will get a polite explanation that what they are doing is unsupported, and that’s usually the end of it.
Pip is special because it has a high-profile dependency that uses its internals. And we don’t want them to, but we acknowledge that there’s not really any good alternative. And of course, the fact that they are happy to deal with the consequences of their choice and don’t pressure us is a big factor.
I think simply ignoring the proposal and continuing to work as we do right now is the realistic answer. So I guess the question is whether the fact that the proposal fails to work for us is enough to make you (or the SC) feels like a significant enough problem to reject the proposal…
We must maintain strict backward compatibility for the language feature (not necessarily for the stdlib when it comes to disallowing access to e.g. imports that were never meant for export).
I presume you already have some rule about what to do if __export__ doesn’t exists (and it’s probably the same as for __all__). Having at least one export in a file turns on the mechanism and hides everything not explicitly exported.
Aprivate keyword/decorator for methods in classes, if we want it, should probably use the opposite default – methods are public unless explicitly marked private, so the default is backward compatible. Subtleties could arise with inheritance – you might want to change the visibility of a method in a subclass (though maybe containment-instead-of-inheritance is the right answer there).
It very much is intended to be enforced at runtime – otherwise you might as well put the info in a comment.
And this whole argument (which others in this thread have also used) of hiding access being somehow “unpythonic” is wrong – Pieter wrote this PEP because when developing large-scale projects people need certain things to be hidden, and the current options all are suboptimal.
Python has come a long way from 1995 (when “open kimono” was a feature).
If a coding agent doesn’t understand __all__, why would it respect __export__? And if you can teach it the latter, you can teach it the former. The intent of both, after all, is the same when you use them as “documentation” – it’s only enforcement of __export__ that is more strict than __all__.
It’s more a concern that a project further downstream is running tests with warnings as errors or something similar. A linter error is targeted more directly at the developer using the internals, rather than at anyone that uses the package that uses the internals.
In the same way that when I use pip install ... it’s not particularly helpful to tell me that pip is using a deprecated feature, but it is useful for the developers of pip.
A linter error is obviously stronger than a comment, which is what I suggested. A comment would require someone actually look into the code. Presumably anyone developing a ‘large-scale’ project will be using some kind of static analysis.
Just to be clear, my comment was under the assumption that accessing something not in __export__ will be a runtime error, and an LLM can react to it on it’s own without needing you to teach it because it’ll see an error that will say “foobar is private, you cannot use it”
But that’s just terrible – like a newbie human using random stuff they don’t understand until they hit a runtime error. Humans (except in a very early stage of learning to code) don’t code that way any more, and neither should coding agents.
I don’t follow this reasoning. pip-tools is opting in to that cost by using pip._internal, and I think the maintenance cost of matching pip’s internal changes against their code will far outweigh a single unchanging del pip.__export__ line.
Maybe I’m misunderstanding the proposal. Why would it be a single del pip.__export__ line? Wouldn’t there need to be __export__ variables in all of pip’s modules and submodules (all of which would be empty, as we don’t want to have any public API, as I said before)?
But it’s quite possible I’m misunderstanding how this would work, as I still don’t know how pip’s own code would have access to private (i.e., all ) symbols from other pip modules, so I’m sort of guessing how things would work right now…
+1 to this. The “you’re not supposed to do that” response is a big part of why I’ve drifted away from participating here. When the answer to a misuse is a shrug instead of an explanation of why the boundary exists, it reads as adversarial and it teaches people not to ask. Clearly documenting what’s public vs. private prevents the misuse in the first place, so nobody has to be told they’re holding it wrong. PEP 842 seems like a chance to do that.
Not really, no. This is a more heavyweight option than just writing a module __getattr__ that errors for private access, and it’s harder for users to work around if they actually know they are using an internal intentionally. It isn’t offering anything new.
The only potential benefit here over DIY on that is community standardization/understanding of the intent, but it comes at the cost of runtime warnings for something I don’t want to issue warnings for. If a user is using it intentionally, they shouldn’t have to work around warning filters, this isn’t broken code or pending removal, it’s just better communicating the actual intended public API of the module in a standardized way.
)
. One argument for the soft keyword version:
) symbols from other pip modules, so I’m sort of guessing how things would work right now…