Make os.path a real module

Currently os.path is an alias for posixpath / ntpath:

# os.py
if 'posix' in _names:
    import posixpath as path
elif 'nt' in _names:
    import ntpath as path
else:
    raise ImportError('no os specific module found')

sys.modules['os.path'] = path

It would be easier for static type checkers to detect it if it was a real module (like in typeshed):

# os/__init__.py
from os import path
# os/path.py
import sys

_names = sys.builtin_module_names
if 'posix' in _names:
    from posixpath import *
    from posixpath import __all__ as __all__
elif 'nt' in _names:
    from ntpath import *
    from ntpath import __all__ as __all__
else:
    raise ImportError('no os specific module found')
1 Like

What problems are currently being caused by this?

On https://github.com/python/cpython/issues/116998 I asked that @Nineteendo open this discussion because I was concerned that this would break existing code in the wild.

1 Like

For what it’s worth, I always thought it was strange that importing os causes os.path to be “imported” (the platform-specific module becomes an attribute of os and is added to sys.modules with the aliased name, completing the illusion).

2 Likes

It’s been that way for ages.

1 Like

Trying to figure out what the advantage is here (or conversely: what the problem is that’s being solved). You say “it would be easier”, but is there actually a current issue?

1 Like

LIke the results of this Github search for "os.path is posixpath" OR "os.path is ntpath" OR "os.path == posixpath" OR "os.path == ntpath"?

I thought that “pattern” was even officially recommended somewhere.

3 Likes

I initially thought Pylance had trouble detecting the functions in os.path (that’s why initially created this issue). But that’s simply because it doesn’t detect re-exports in typeshed.

AFAIK, static type checkers never even look at os.py, that is what typeshed is there for after all.

I thought Pylance didn’t find os.path, because it’s not a real module. But typeshed simply doesn’t explicitly define the functions in os/path.py.

Maybe it’s indeed not a good idea as it would break some code, and it doesn’t even fix the original problem with Pylance and the re-exports.

It’s a fairly common pattern for packages to import sub-modules into the main package’s namespace to expose them as public APIs and also to do so based on certain criteria, e.g. type of OS, Python version or availability of alternative implementations (e.g. optimized versions written in C or Rust).

Rather than change the os module, I think it would be more useful to fix the type checkers or typesched to properly handle these situations, since they will see these kinds of mechanisms elsewhere as well.

Since type definitions are really API definitions, the type checkers should not really have to care about where an API is implemented, but rather just look at the API name and use the API definition for that name.

Things get a bit more complicated when the API definition depends on external factors, e.g. certain parameters are only available on some platforms, but handling those is out of scope for type checkers, IMO. The API definitions could just include all possible parameters and make them optional.

5 Likes

It’s a little hard for me to imagine why people would recommend doing that kind of check. Isn’t that what sys.platform is for?

Or more specifically, its what os.name is for, which maps more explicitly and directly to which path module is loaded.

1 Like