How Do I Avoid/Override the dict[str, Any] Type When Loading From TOML?

Python Help
1

I’m writing a Python package that I have tried to type-hint throughout, but I keep running into typing mismatches because the result of tomllib.load(config) is typed as dict[str, Any]. I understand why tomllib would do this, but I don’t understand how to prevent the resulting mismatch with my fully typed Config class.

I’ve tried loading the data into a fully typed ConfigLoader class that subclasses typing.TypedDict first, then passing that to my actual Config class, but that does not help.

Surely there’s a way to avoid this issue, but I am completely stumped. Any help would be greatly appreciated - even just a link or two to something helpful. I’ve searched all over, but can’t find anything useful.

Thanks so much in advance!

2

This is kind of the classic pattern for parsing.
Your TOML basically is dict[str, Any]: your python program cannot guarantee anything about the types of a parsed string.
So you write a Config.from_dict(input: dict[str, Any]) factory method that’s full of assert and cast statements and sometimes some #type: ignore[...].

There’s no way around using assert isinstance(...) unless you fully 100% trust the config file, and in that case you have to use cast and #type: ignore[...] to communicate to the type-checker that you fully 100% trust the config files.

You can also use a library like cattrs or pydantic, but that’s just an indirect way of writing assert statements.

3

This is effectively the boundary between ‘known’ and ‘unknown’, or ‘safe’ and ‘unsafe’. Some code needs to evaluate the data to determine whether it can be labeled as ‘known’ and ‘safe’.

4

One caveat - Python’s optimizations remove assert statements. If you need the checks in prod, use Pydantic or full if isinstance blocks instead.

5

As @peterc says, it is a fancy way of avoiding writing your own assertions, but depending on how detailed your TypedDict is, it might be worth using a well-tested library that derives its validators from your types, rather than writing them yourself. In addition to cattrs, I’d look into the msgspec.toml API. Each should be a very small lift (if you’re willing to add the dependency) and quite fast.

6

On the other hand, this can be advantageous too - if you only need the checks statically, then an assert has slightly less overhead than a cast in prod.

7

Thanks so much for your help, @peterc . I spent the day adding all sorts of assert isinstance(...) statements to my factory function and that made all the difference with the type checker. Not only that, it got me to think through a lot of edge cases that I would have missed otherwise.

I’ve used pydantic in the past and really like it (for the most part). I really wanted to avoid it in this case, as 1) It’s a bit of a bazooka when all I needed was more akin to a fly swatter, and 2) The package I’m working on is designed to be used in a System Administration context, so I’m really trying to keep the number of 3rd party libraries to a bare minimum.

In any case, you got me over the hump here and I appreciate it!