Sometimes, I need internal API that should be called from the library code only; outside calls should be prohibited to the library users.
Python has a long tradition of using underscore prefixes for such private things. It works well in many cases, but sometimes underscores are not applicable.
The most obvious case is calling the class constructor. Suppose I have a class that a factory should instantiate; direct instantiation is discouraged. task =asyncio.create_task(...) is a good example; user code should never use task = asyncio.Task(...) form. The stdlib has many examples of such objects; many third parties also use the same patterns.
Obviously, Task.__init__ should exist to create a task; a special method should be named precisely __init__ without additional underscores.
In addition to special methods, a library author could prefer non-underscored methods like Action.run() over Action._run() for aesthetic reasons but warn a library user to call these methods directly if these methods are intended for use from the outer code.
The proposal is:
define @typing.internal decorator that could be applied to functions and methods only, decorating classes is forbidden.
The class does nothing in runtime except mark a decorated function with a flag.
Static type checkers (mypy, etc.) pass if an internal function is called from a package where the function was declared; otherwise, the checker raises an error.
The proposal intentionally doesnât want to make any restrictions on runtime; a user can shoot their leg if he really wants it.
The decorator should help âgood citizensâ by using a library code the way the library author intended; the author is free to modify declared internal details without accidentally breaking âgood code.â
If the proposal accepts positive traction, I can prepare a more formal document with the communityâs help.
I would be willing to support something like this (I have my own decorator like that) since documentation is usually not sufficient (at least for linters).
However, this would always require importing the entire typing module and I donât know how slow this can be. Alternatively, we could make a smaller module which only owns some âfastâ typing primitives (e.g., TYPE_CHECKING, and typing decorators, no typing types) (I think something similar has already been discussed somewhere but I donât remember where).
A more general alternative is to add some visibillity attributes similar to C/C++ compiler attributes which would simply be no-ops at runtime (they could even be eliminated!) but visible by static tools. But this is another topic on its own.
Please letâs keep @internal proposal as simple and tiny as possible.
Reducing the import time could be an important target, but it is definitely out of the scope.
I donât really think itâs out of scope because if we want to add it to the standard library (and use it for us), then we should reconsider the optimizations for reducing import times in general we did. Maybe asyncio doesnât have this issue but there are other modules that are worried about it (e.g., shutil, socket, uuid (a lot of work has been done for this one)) so I think it should be addressed at the same time (cc @Jelle who added the C module for typing).
Currently all stdlib types are checked by typeshed IIRC, I donât have a plan to apply @intenal to any CPython module directly but decorate some stubs in typeshed only.
Third-party libraries also either provide .pyi files or have import typing virtually in every module, adding a new function to typing doesnât change the picture for them.
Oh then itâs fine! I thought you wanted to include them directly at the level of the standard library which could have been blocking it for our needs. Sorry for misunderstanding you. In this case, I donât have any objection left.
A project could have multiple packages, with some functions in one package intended to be used by another but not outside code. As proposed, @internal wouldnât be of much help.
Take Kotlin for example: It is not possible to allow this using internal, as only code within the same module (a couple of files compiled together) can use internal-marked APIs. Instead, the aforementioned problem is solved with ApiStatus:
Indicates that the annotated element (class, method, field, etc) must not be considered as a public API. Itâs made visible to allow usages in other packages of the declaring library, but it must not be used outside of that library. Such elements may be renamed, changed or removed in future versions.
It is internal for the package, including all subpackages. E.g., internal function a.func is available from a, a.b etc; but it is not available from b or b.c. a.b.func is available from a as well.
I donât want to propose cross-package internals. I can imagine @internal(a, b) decorator form, but I really prefer avoiding the complication.
Internals in any language have their subtle differences. Iâve mentioned known usages to get the idea, not to borrow the whole implementation from.
I donât want to declare internal attributes, partially to avoid bikeshedding on what is the canonical form: var: Internal[Final[ClassVar[int]] or ClassVar[Final[Internal[int]].
Underscore prefixes for attributes already exist and work well.
Iâm striving to simplify the proposal rather than covering all possible cases. The idea could be extended in the future, though.
The same is true for internal classes. What is an internal class? An outside code cannot subclass the class? Maybe it makes sense, but the outside code should also instantiate the subclass somehow. If the base class has an internal constructor, instantiation raises a linter error.
Final classes have a little different logic; a final class and final attrs are completely fine, I use them in my code.
Internals is another beast; Iâm unsure if we need to cover all theoretical possibilities (but Iâm open to proposals).