Type-hinter hinting

Ideas
1

I see a lot of Python people trying to tackle the C-esque problem of forward declaration for type-hinting purposes. There is much following of the typing module’s example, loading modules and names, ballooning globals(), just to be able to hint names from them, and lots of comment-hinting and other workarounds.

It feels especially wasteful in code that makes no use of runtime introspection.

Perhaps a Python solution to a Python problem might be better.

import def <non-wildcard import statement>;
from <path> import def <non-wildcard list>;

Other wordings might be viable, I chose import def because it is import related and def is a keyword

>>> import def
  File "<stdin>", line 1
    import def

and we’re only seeking to import the definition of things (forward definition, if you will).

This statement WOULD NOT impact on the globals() namespace, it WOULD NOT [immediately] attempt to load anything.

Any runtime side-effects WOULD BE deferred to the point where someone tries to perform runtime introspection.

Meanwhile, type-analyzers can use this to recognize where to obtain hinting information from.

Example:

# Current code
from typing import Union  # for emphasis, aware it is deprecated.

from . Adaptors import Base

# None of these are actually used in this module.
from . Adaptors.Mesh import Mesh
from . Adaptors.Shader import Shader
from . Adaptors.Texture import Texture

# None of these are actually used in this module.
from Packages.Converters.ConvertMesh import MeshConverter as MeshC
from Packages.Converters.ConvertShader import ShaderConverter as ShaderC
from Packages.Converters.ConvertTexture import TextureConverter as TextureC

def process(asset: Union[Mesh, Shader, Texture], converter: Union[MeshC, ShaderC, TextureC]):
    if Base.legal_conversion(asset, converter):
        converter.convert(asset)

print(globals())

The globals printed here would include process, Union, Mesh, Shader, Texture, MeshC, ShaderC and TextureC, despite being entirely unneccesary.

If this becomes the following, meaning none of the ‘import def’ symbols would be available in the module’s namespace.

from typing import def Union

from . Adaptors import Base  # actual import

from . Adaptors.Mesh import def Mesh
from . Adaptors.Shader import def Shader
from . Adaptors.Texture import def Texture

from Packages.Converters.ConvertMesh import def MeshConverter as MeshC
from Packages.Converters.ConvertShader import def ShaderConverter as ShaderC
from Packages.Converters.ConvertTexture import def TextureConverter as TextureC

def process(asset: Union[Mesh, Shader, Texture], converter: Union[MeshC, ShaderC, TextureC]):
    if Base.legal_conversion(asset, converter):
        converter.convert(asset)

print(globals())

then the only non-default global would be ‘process’ - none of the others would appear in the namespace because they are purely type hinting definitions.

Open Questions:

  • Can this be implemented to solve inter-module circular dependencies?
  • Can we create declaration imports?
    – Something similar to all in index? E.g. def?
    – ‘from Module import def *’ to allow creation of forward-declaration ‘headers’?
  • How do I define an alias without a real import?
    – How do you write Tree = Iterable[Tuple[Path, DirEntry]] without importing all of these?
1 Like
2

I thought forward references was mostly an issue of intra-module dependencies. For inter-module, I use

import typing
from __future__ import annotations

if typing.TYPE_CHECKING:
    import array

def foo(x: array.array) -> int:
    ...

Which works fine for me

Edit: added imports to example, which I had assumed were obvious

3

I’ve just tried your code snippet in Python 3.10 and the first error it
raises is

NameError: name 'typing' is not defined

So then I fix that error with the obvious import typing then I get:

NameError: name 'mymodule' is not defined

Running type checkers is all very well and good, but if the code won’t
actually run, it doesn’t “work fine”.

To actually make it work, you need to either quote the annotation

def foo(x: "mymodule.bar") -> int

or run from __future__ import annotations at the very beginning of
your module.

1 Like
4

Your suggestions:

  • “we’re only seeking to import the definition of things”

  • “This statement WOULD NOT impact on the globals() namespace”

  • “it WOULD NOT [immediately] attempt to load anything”

don’t really work with Python’s execution module. We could make it work,
of course, but only by building a whole new execution module into the
language that would allow the interpreter to statically scan a module
without actually running it (as mypy does), AND introducing a second
global namespace exactly the same as globals() but only for type
annotations.

That would probably work for, oh, 90 or 95% of classes, but would have
problems with any module that generated classes dynamically.

Remember that classes in Python are not declarations, they are actual
executable code that run at run-time. And annotations are
currently not just declarations, but valid Python expressions
which are also executed:

class C:
    def method(self, arg: print("Hello world!!!") or int) -> float:
        pass

is a silly thing to do, but it does actually work, when the class
statement runs, “Hello world!!!” is printed and the annotation evaluates
to int.

Right now, the Python Steering Council are considering two competing
PEPs which will make annotations easier to work with.

The first, PEP 563, has actually been approved, and starting from Python
3.7 you can activate it with

from __future__ import annotations

This will change annotations, all annotations, into strings, instead
of evaluating them as expressions.

The second, PEP 649, was originally rejected by the Steering Council,
but they are now reconsidering it:

PEP 649 will change annotations to defer their evaluation until runtime
introspection.

3 Likes
5

In order to do this, you had to add 4 lines of boiler plate instead of 1.

import typing
from __future__ import annotations

if typing.TYPE_CHECKING:
    import array

vs

import def array
6

Thanks for the detailed response. It seems that, with some tuning, this would be a useful embellishment to PEP 563 because type checkers are going to want to know where a type name comes from, and people will solve that either by blindly importing modules and creating needless spaghetti references, or they’re going to use the new conditional-boiler-plate idiom.

from __future__ import typing
if typing.TYPE_CHECKING:
  import mod1
  import mod2

Which is essentially what ‘import def’ would be replacing, but it could avoid the potential complications that still exist when you try to line up your imports at the top of your code.

7

Sure, but you are forgetting to count the 10,000 lines of code in the
interpreter to support a whole new execution model plus a second global
namespace to hold the things which are imported-but-not-executed plus
the extra runtime cost for annotations plus the extra tests that have to
be run to make sure it all works, plus the documentation to explain it
all. Inventing a piece of syntax is the easiest part of a proposal.

Also, that boilerplate (a mere four lines) works all the way back to
Python 3.7 (by memory), whereas the earliest import def can work is
Python 3.11 or more realistically 3.12, and if you are writing a
library, the earliest you can use it will be when you have dropped
support for 3.10 and older. Which might not happen for five or ten
years.

So there is a ton of work needed to support import def to get a result
that we can already get, right now, using string annotations, or either
of the PEPs 563 or 649. High cost, low benefit. That’s my opinion.

You might like to take this proposal to the Typing-Sig mailing list,
where there are more typing experts. They should be able to give you a
better idea of how plausible this import def would be, and how
desirable it is.

3 Likes