Correct type annotations for SourceFileLoader source_to_code

What is the correct type annotation for SourceFileLoaders source_to_code import hook
called from FileFinder.path_hook that I should use?

From what I can establish the actual annotation in the source is:

def source_to_code(
    data: Buffer | str | Module | Expression | Interactive,
    path: bytes | str | PathLike[str],
    fullname: str | None = None, *, _optimize: int = -1) -> CodeType:

If I keep that type annotation I get warnings further on in my code. Testing shows that type(data) is bytes and type(path) is str. However if I change to:

def source_to_code(
    data: bytes, 
    path: str, 
    fullname: str | None = None, *, _optimize: int = -1) -> CodeType:

I then get the warning that the type annotation doesn’t match the source method.

You get warnings if you change the type annotations since you violated the Liskov substitution principle. But what warnings did you originally get exactly?

1 Like

I then pass data and path into another function which has to have exactly the same annotation to avoid warnings:

def func2(
        data: Buffer | str | Module | Expression | Interactive,
        path: bytes | str | PathLike[str]):

which then causes:

BytesIO(data)

warning:
Expected type 'Buffer', got 'Buffer | str | Module | Expression | Interactive' instead

line:

“” + path[:]

warning:
Member 'PathLike[str]' of 'bytes | str | PathLike[str]' does not have attribute '__getitem__'

line:

Path(path).resolve()

warning:
Expected type 'str | PathLike[str]', got 'bytes | str | PathLike[str]' instead

line:

path.rfind(“\”)

warning:
Member 'PathLike[str]' of 'bytes | str | PathLike[str]' does not have attribute 'rfind'

The warnings are actually pretty straightforward. The first one is because you’re trying to make a BytesIO but the constructor only accepts objects implementing the buffer protocol, whereas you’re directly passing the argument through (maybe you should handle the cases where the argument takes those types; I suggest you start with reading the docs). If you convert the path to string using os.fsdecode and then normalize it using pathlib.Path.resolve, the subsequent errors should go away.

1 Like

That is, if you change the raw string path manipulation code to use the pathlib.Path methods.

1 Like

I’m confused. Did you declare the type of your data and path variables when you created them?

Because the only way the type checker would think that data is of type Buffer | str | Module | Expression | Interactive is if you failed to give it any more specific information. If you know data is bytes and path is str, then declare them as such. Otherwise, what’s the point in using a type checker?

If you don’t know the types of bytes and str, then the type checker is giving you a valid warning - not all values for data that can be passed to source_to_code are suitable for passing to your func2, so you need to fix that problem (possibly the way @jonathandung suggests).

1 Like

If I change my annotations of Source_to_Code to:

class PreProcessLoader(SourceFileLoader): 
    def source_to_code(self,
        data: Buffer,
        path: PathLike[str],
        fullname: str | None = None, *, _optimize: int = -1) -> CodeType:

I get the warning:

Signature of method ‘PreProcessLoader.source_to_code()’ does not match signature of the base method in class ‘SourceLoader’

OK, I think I get what you’re asking now. You’re implementing source_to_code and calling your func2 from within that implementation? My apologies, that wasn’t clear from your original post.

In that case, you have to be prepared for any type supported by the annotation. At the moment you aren’t, because you’re passing data to BytesIO, which can only accept objects of type Buffer (as the warning says). You need to decide what to do if you get passed a str, a Module, an Expression or an Interactive.

Maybe the answer is to do a runtime type check and raise an exception when passed something that isn’t a Buffer. Maybe you need to work out when source_to_code can be called with one of those other types, and work out how to handle those cases. Only you can decide what your code should do.

What you shouldn’t do is try to hack the type annotations so that they don’t report issues. The whole point of using a type checker is to detect problems such as these, so if you don’t want to deal with the issues, just don’t run the type checker (or suppress it with # type: ignore).

1 Like

@jonathandung A bit confused by the type annotation of data:

Python source type annotation is:

data: Buffer | str | ast.Module | ast.Expression | ast.Interactive

but

type(data)=<class ‘bytes’>

Edit:
How do I check for if type(data) == "Buffer" to avoid the annotation warning?

You can just use a normal isinstance() call. if not isinstance(data, Buffer): raise TypeError will “narrow” the type of data in the type checker’s eyes.

1 Like

@duncathan Now completely free of type annotation warnings, many thanks.

1 Like

Buffer is an abstract base class. It cannot be instantiated, so type(data) would never give it. See the docs: collections.abc — Abstract Base Classes for Containers — Python 3.15.2 documentation

1 Like

Dunc’s solution would only work if you’re sure the loader won’t be passed non-buffer data. Have you tried adding a print in the method to see what types you get?

1 Like

Yes in my testing print(type(data)) is always <class ‘bytes’>