Proposal to Improve Support for Other Python Platforms in the Typing Specification

Typing
1

This is my first post to the Python discuss, so please be gentle. I am a long-time MicroPython user and have been working on type stubs for MicroPython for the past few year, but I still enjoy learning from others each day.

The Python typing community has made significant strides in improving type-checking capabilities across various Python implementations. However, in my little corner of the Python-ecosystem the appetite for static type checking is held back by some limitations of the specification on how type-checkers can apply the available the platform characteristics, to type-stubs.
For MicroPython the challenge is to distinguishing MicroPython-specific code and differentiating type stubs for various ports. This is currently not possible, and as a result I currently maintain multiple stub packages - each for a separate port board where less than 20% of the stubs are actually different.
This proposal aims to introduce enhancements to type-checkers to better support MicroPython and its diverse platform values.
Current platform checks by type checkers are intentionally limited in the typing spec: Type checker directives — typing documentation

I would like to discus and brainstorm the following proposal with the community, and based on the feedback, I aim to create a PR with a proposal to update the typing spec.

Current Challenges

Currently, platform checks are underspecified in the typing spec: Type checker directives — typing documentation, which only mentions sys.platform and sys.version_info.

MicroPython utilizes over 12 additional sys.platform values, such as esp32, esp8266, mimxrt, nrf, renesas-ra, rp2, samd, stm32, unix, webassembly, windows, and zephyr. Currently, developers need to write long and repetitive boilerplate code to handle these platform checks, which is both cumbersome and error-prone.

For example:

if sys.platform == "esp32" or sys.platform ==  "esp8266" or sys.platform ==  "mimxrt" or sys.platform ==  "nrf" or sys.platform ==  "renesas-ra" or sys.platform ==  "rp2" or sys.platform ==  "samd" or sys.platform ==  "stm32" or sys.platform ==  "unix" or sys.platform ==  "webassembly" or sys.platform ==  "windows" or sys.platform ==  "zephyr" : 
     @overload
     def sleep_ms(delay: float) -> None: ...

While MicroPython defines sys.implementation.name as ‘micropython’, a conditional such as if sys.implementation.name == "micropython": does not work as expected with type-checkers as the use of this attribute is currently not part of the typing spec, and therefore not implemented. sys.implementation.name can be used to distinguish not only micropython, but circuitpython, pypy, cpython, ironpython and other implementations that may benefit for from this proposal.

sys.impementation.version is also not supported in the typing spec, and therefore not implemented. MicroPython has a versioning scheme that is different from CPython, and, same as with CPython, it is important to be able to distinguish in the type-stubs between different versions of MicroPython. For example, the current version of MicroPython v1.25.0 is sys.version_info = (3,4,0) and its sys.implementation.version = (1,25,0).
As the type_checkers will not run on micropython, the versions will need to be specified though configuration or commandline arguments.

Possible Solution
To address these challenges, I propose the following enhancements to the Python typing specification and type-checkers:

  1. Support a limited number of additional attributes, such as sys.implementation.name in type checkers:

    When type-checkers correctly process sys.implementation.name, this will allow type stub authors to use if sys.implementation.name == "micropython": or if sys.implementation.name == "pypy": to simplify authoring of type stubs for these platforms.
    my current thinking is that this should be limited to a small number of attributes, such as sys.implementation.name and sys.implementation.version, to avoid overcomplicating the typing spec.

  2. Allow for additional checks:
    Introduce support for more detailed sys.platform checks, allowing patterns such as if sys.platform in ('esp32', 'esp8266', 'mimxrt', 'nrf', 'renesas-ra', 'rp2', 'samd', 'stm32', 'unix', 'webassembly', 'windows', 'zephyr'):. This would significantly reduce boilerplate code and improve readability.

    • == and != are already supported
    • in and not in a set, list or tuple should be supported as well

Benefits

Implementing these enhancements will provide several benefits:

  • Reduced Boilerplate Code: Developers will no longer need to write long and repetitive platform checks, leading to cleaner and more maintainable code.
  • Improved Type-Checking Accuracy: Type-checkers will be able to accurately distinguish MicroPython, or PyPy specific code and differentiate type stubs for various ports, improving overall type-checking accuracy.

Related discussions :

11 Likes
2

I can’t say how good timed this is. I’ve been using MicroPy for quite some time, and I always try to annotate all my code, so in case you would need any help, I would love to give that. The lack of type-hints in MicroPy made using older code more complicated than it is, and it would be sick to see MicroPy of Python 3.6 (or above)!

Also regarding the core problem, couldn’t the developers reduce some boilerplate by defining shorter names at the beginning of code (e.g.:

esp32 = sys.platform == "esp32"

(just to improve some DRY)?!

3

Thanks for the post. I appreciate the clear articulation of the problem and the menu of possible solutions.

I think that both of your proposed solutions are possible, but we should understand the advantages and costs associated with each.

It’s important to understand that these conditional platform expressions need to be evaluated by a static type checker very early in the analysis process. In pyright (and presumably other type checkers), the evaluation of these expressions informs the construction of the code flow graph. The code flow graph is needed to evaluate the type of any expression (including type annotations and type qualifiers), and determine the types of imported symbols. That means these platform expressions need to be evaluated before any type evaluation is possible. It’s a classic chicken-and-egg problem. To make this work, type checkers need to work directly from the AST looking for specific syntactical forms. This is why the typing spec is so specific in prescribing which forms are supported. Even small deviations from the supported forms will prevent it from working. For example, if you change import sys; if sys.platform == "darwin": ... to from sys import platform; if platform == "darwin"; ... it will no longer work. These two variants are semantically equivalent but syntactically different, and the syntax matters here. Likewise, if you swap the operands (if "darwin" == sys.platform) or attempt to use a variable (as suggested above by @JoBe), it will not work. Any solution needs to respect the constraints faced by type checker implementations. For more details, refer to this pyright documentation.

Supporting sys.implementation.name is possible, but it has some (perhaps non-obvious) costs. It would require type checkers to add a new configuration mechanism for setting the desired implementation name. It would also require adding command-line switches, and it would require auto-detection of this value if it’s not specified in the configuration or command line. This wouldn’t be so bad if it were just type checkers, but many CI scripts, test harnesses, typing-related tools (e.g. mypy_primer), etc. also provide ways to configure the pythonVersion and pythonPlatform. This approach would compel all of these tools and scripts to plumb through another configuration setting. That’s doable, but it will take time and impose some not-insignificant cost across the ecosystem.

Supporting alternative syntax forms for sys.platform checks is possible and would be limited to just type checkers, linters, stub generators, stub testers, and similar tools. It wouldn’t require new configuration or auto-discovery mechanisms. Most other tooling and scripts would not need to be modified because they already support a pythonPlatform configuration setting and command-line switches. So this option is less costly than the first. If we were to adopt this option, I would recommend against supporting tuples, lists, and sets. I would pick one of them — preferably tuples — and mandate that everyone use that. This isn’t a situation where supporting lots of freedom, creativity, and redundant approaches is helpful. Picking one supported syntactic form is beneficial. Supporting both in and not in operators is straightforward.

Let me suggest a third option that may partly address the problem without requiring any new functionality. Both mypy and pyright allow developers to define “constants” in the config file. In pyright, this is done with the defineConstant setting, and both boolean and string constants are supported. Mypy supports always_true and always_false, which limits it to booleans. You could standardize a symbol such as MICROPYTHON (all caps to indicate that it’s a constant) that you use within your stubs. Consumers of the stubs could then modify their type checker configuration to indicate that MICROPYTHON should always be evaluated as True. If mypy were extended to support string-based constants (as pyright already does), you could standardize a symbol such as IMPLEMENTATION_NAME that takes on the value of micropython. Some variation of this idea may address part of the problem you’re facing.

I’ll note that none of the above solutions (including the two you proposed) address the “implementation version” requirement. Relative to platform comparisons, how important are version comparisons?

3 Likes
4

This requires the project to support only one IMPLEMENTATION_NAME value. But libraries often tend to support more than one implementation.

5

Simplicity and predictability

I fully agree on a tightly constrained set of conditions and evaluations.
But today the constrain is worded as “Don’t expect a checker to understand obfuscations” which, at least to me, is neither precise nor clear.

Thanks for the link to pyright’s implementation of this, which is definitely clearer, and seems a bit broader as well.

As a stub author I am faced with the unclarity of what may or may not be supported by other type checkers. so if we can bring the same level of clarity to the typing spec that would definitely be helpful.
I have adopted that notation in this response .

Tooling Implementation effort
Im very aware that there is effort, and likely cost, involved with any change, and that that should be balanced against its benefits.
There are also costs for not implementing this, we are replacing that effort on tooling by effort of some individuals to make it work. I know of several individual and community projects that have stopped maintaining, as it was too difficult/impossible to create one set of stubs that could be used by multiple platforms ( ports / boards) and versions. I have been at this for some 5 years now, and have been able to keep up, mostly though good support and clarity in the standards.

I cannot judge if its riskier to change an existing config options or add a new one,
but minimizing the change needed is definitely part of the equations that we need to discuss and solve.

Allow the use of externally defined constants as string literals

Before raising this I already tested if that would be possible. As my aim is to support multiply type checkers this would be limited to True False for MyPy.

Using defined constants as string literals would be a great addition to the spec, espcially if that would also allow for a in <tuple> check.

  • <DEFINED_CONSTANT> == <string literal>
  • <DEFINED_CONSTANT> in <tuple of string literals>

The teachability of this would be a bit more difficult, as we will need to educate creators and users of stubs to use the same constant name, which in the case if MicroPython will be derrived from sys.platform and sys.implementation attributes that seem to be defined for such a purpose.

my preference would be to be able use :

  • sys.implementation.name == <string literal>
  • sys.implementation.name in <tuple of string literals>

but indeed the defined constants could be a good alternative if added to the typing spec.

Allow the use sys.implementation.version

  • sys.implementation.version <comparison> <tuple>

I previously tested this to see if I could use a DEFINED constant for the MicroPython version but never got that to work properly there is only a <DEFINED_CONSTANT> == True check in mypy.

Even in pyright with a <DEFINED_CONSTANT> == <string literal> I would need to enumerate all the versions as literal strings, and update them as new versions come available. In the CI tests that I run to verify the stubs i create against mypy and pyright I currently have 42 check that are version specific, either to the introduction, removal of a new feature or the change of an API.

MicroPython is also used more and mode by professionals to create devices used by consumers, and in manufacturing and medical environments where the lifecycle of producs can be quite long. By allowing a type-stub to be valid for multiple versions of MicroPython this would allow code for devices to be maintained for a longer time, using the improvements in the recent stubs. This is no different than the extensive use of sys.version_info <comparison> <tuple> in the standard library to allow for backwards compatibility.

Without a way to check the version of the implementation I have to maintain each version of the stubs separately. I have created rather extensive automation for this but it would be great if I could use a single set of stubs for multiple versions of MicroPython. With 12 different ports ( not to mention the 180+ different boards) this is a lot of work to maintain.

Allowing for sys.implementation.version <comparison> <tuple> would be a great addition for MicroPython, but I cant assess this for other platforms such as pypy.

3 Likes