PEP 810: Explicit lazy imports

PEPs
103

I don’t know how to square this with what the PEP says:

If the global lazy imports flag is set to "disabled" , no potentially lazy import is ever imported lazily, the import filter is never called, and the behavior is equivalent to a regular import statement: the import is eager (as if the lazy keyword was not used).

If you follow the comments and replies back up you can see that the discussion about -X lazy_imports="disabled" starts from whether it would be necessary to allow a filter in this case.

This is not a hypothetical issue and will definitely arise in reality. People already do things like setting TYPE_CHECKING = True at runtime and it is already known that it breaks things. In the future such imports can be changed to use lazy import but then -X lazy_imports="disabled" becomes the new version of someone setting TYPE_CHECKING=True at runtime. There needs to be a clear understanding of what the expectations are here:

Is it reasonable for a user of a library to expect that the library functions under -X lazy_imports="disabled"?

If any one library in use in a given process does not support this then the whole process breaks entirely and immediately and there is no filter mechanism that can be used to make it work. I think that this is going to result in pressure on libraries to make this work but making it work removes a significant chunk of the benefit of using lazy imports since every use of lazy imports must be something that would have worked as an eager import.

2 Likes
104

Just for us to understand your point fully, is it fair to summarize your point as "we may need a version of the filter to also run when -X lazy_imports="disabled" is set?

1 Like
105

I saw this PEP on Hacker News and wanted to say this is one of the best-written PEPs I’ve read recently. Kudos to Pablo and the entire authoring team. Also I’m very happy to see that the community response here has been overwhelmingly supportive, which really shows how much this is needed

I’ve read through the rejected ideas section on keyword placement, and I understand the backwards compatibility concerns with from X lazy import Y. That said, I still find myself wishing there was a path forward for that syntax. The current lazy from X import Y reads awkwardly to me - it breaks the visual pattern of scanning for from ... import statements that I’ve internalized over years of Python.

I know the authors have heard this feedback multiple times and have solid reasons for their decision. And honestly, the feature is valuable enough that I’ll happily adapt to whatever syntax ships. But if the Steering Council were open to a deprecation path (like the suggested SyntaxWarning for from . module with extra whitespace), I think it would be worth the transition cost for better long-term ergonomics.

Regardless, this solves real problems I hit constantly with import overhead in production systems. Really looking forward to having a standard solution instead of fragile workarounds.

12 Likes
106

@pablogsal is there a way for community members to not “reopen” previously discussed discussion points, but also express support for positions that were brought forward more explicitly. An important part of the PEP process is to gather “community input” (see PEP 1). How can the community express support for some of the rejected ideas to hopefully make the Steering Council consider them or the alternatives, while also not continously derailing the conversation? As a community member putting a heart on 1 out of 103 (and counting) comments doesn’t feel like it might reach the council.

I will not name the topic/detail I’m personally interested in (to not derail the conversation) but currently it is part of the Rejected Ideas section even though there is significant community support. Can the community show support (without constantly reiterating the same arguments) in such a way that the authors could, for example, ask the Council to explicitly weigh in on that idea/topic? By having it in Reject Ideas and explicitly asking the community to not bring it up constantly we might not reach the consensus the community would have reached in a more open discussion.

I’d like to stress I’m not accusing of you of not having an open discussion. You have been doing so very well so far!

8 Likes
107

Thanks for your message! You raise a fair point, and I appreciate you bringing this up.

First, allow me to apologize if any of my comments made you or anyone else feel like they can’t express themselves or show support for ideas they care about. That was absolutely not my intention, and I’m sorry if it came across that way. Of course you should be able to express your views and support for alternatives!

Let me provide some context for where I’m coming from: managing thes discussions is always quite demanding as not only we need to read but also deliverate, sometimees code alternatives to explore ideas and incorporate feedback. Between the comments here, GitHub, other places, and private channels, we’ve been handling hundreds of comments and feedback messages (I’ve personally received over 30 private emails asking about different aspects of the PEP just the first day!). My goal in asking people to read through existing discussion before commenting and not please not reopen closed threads if possible is simply to make the conversation manageable and focused, not to discourage participation. We genuinely want to hear from the community and that’s why we’re here!

Regarding your specific question about showing support: I absolutely trust you and others in the community to strike a balance between helping us keep converging towards something and reopening discussions and expressing what you feel is important. If you believe a rejected idea deserves reconsideration and you want to add your voice to that, please do speak up! Just maybe reference the previous discussion and explain why you think it warrants another look. That helps us understand the depth of community feeling while keeping things organized.

The Steering Council will see all of this discussion, including the reactions and recurring themes. If there’s significant community support for a particular alternative, that will come through: both in the comments and in how we summarize the feedback in any PEP updates.

I really appreciate your understanding of how demanding shepherding these discussions can be, and I’m grateful for your thoughtful engagement. Please continue to share your views. We want this to be a collaborative process, and we want to ensure we reach the best possible outcome together.

Thank you for helping make this PEP better! :folded_hands:

8 Likes
108

Yes and no.

Yes in the sense that if there is -X lazy_imports="disabled" then there will need to be a way to set a filter or otherwise it will often be unusable in practice.

The other thing I would say though is that I am not really happy about having -X lazy_imports="disabled" at all. I can understand the idea of having -X lazy_imports="enabled" because it provides a way to get some of the benefits of the new mechanism in the PEP while still working with code that has not been updated to use things like __lazy_modules__ that are added by the PEP. Setting -X lazy_imports="disabled" though is the opposite: the code has been updated to use __lazy_modules__ or lazy import but there is still a flag to force it to behave differently from what the author intended.

I also want there to be a clear line drawn about whether it is reasonable to expect libraries to work under -X lazy_imports="disabled". The PEP is ambiguous about this but says for example “best practice is still to avoid circular imports in your code design” or above this was described this as a “code smell”. The implication here is that if the code was “better” then it would work with -X lazy_imports="disabled" so that should be a target or could be a reasonable expectation for users.

If the end result is that people do expect code, libraries etc to work with -X lazy_imports="disabled" then that constrains the use of lazy imports making some of the claims in the PEP untrue e.g.:

Lazy imports eliminate the common need for TYPE_CHECKING guards.

4 Likes
109

I want to bring this up again as I haven’t seen this concern addressed or discussed anywhere else so my apologies if I missed it. I can easily imagine that once the word spreads that lazy import is better in almost[1] every way over plain import it will become the de-facto norm, especially for newcomers to the language, causing churn, drive-by PRs sprinkling lazy on each import line, rewritten tutorials etc. Such disruption is often used by some core devs as an argument against an idea. I believe this calls for some clarification in the PEP, maybe as an answer to a question “So why shouldn’t I just use lazy import everywhere?” in the FAQ, a section on how having a way to lazily import is expected to affect the ecosystem, or a paragraph in the ‘How to teach’ section.

Btw personally I love the feature and all the work that went into it, just wanted to see this clarified somehow in the PEP.

  1. e.g. except when import-time side effects are required ↩︎

18 Likes
110

I’m glad to see this revisited, and I’m hoping the explicit version can overcome the prior issues in wanting to make this the default behavior of existing code that relied on imports having side-effects.

I don’t have much to add to this that hasn’t already been extremely well covered by the pep, but there are two things I’d like to chime in on here:

  1. There are some very specific parallel desires with typing that are currently handled (inadequately) by some people by not actually importing (if TYPE_CHECKING: ...), but breaking other introspection in the process. While I don’t think this rises to a level of necessity, and people should be able to use lazy imports in these cases instead, this may warrant a typing-specific extension of the idea that is slightly more inline with the current typing ecosystem if others find it insufficient.

  2. I don’t think the choice of keyword actually matters; lazy and defer (the latter appearing to be a popular alternative in discussion so far) both convey the idea properly enough. “Lazy” is likely the more accurate term, as imports are not deferred; certain aspects of them are lazily evaluated. While I have no strong preference one way or the other, I don’t agree with some prior statements about the word “lazy” being somehow less understood or less professional. The meaning of terms is context-dependent, and the idea of lazy evaluation is not even a Python-specific concept within computing.

6 Likes
111

I feel like you’re asking what libraries should be doing in response to this PEP? My plan is to do nothing.

As a library maintainer, I fully intend to treat support for running with global lazy imports as a feature request, not a bug report.

So, should you have any expectation that a library will work? No, none whatsoever. It might, but figuring that out is on you as a user.
Will I summarily close, in a hostile manner, any reports about it? Also no. I’ll weigh it like any feature request.

EDIT: sorry, I accidentally inverted “enabled” and “disabled” while reading @oscarbenjamin’s post, and my response isn’t fully applicable. I maintain a similar stance for the global disable flag, once I start using lazy imports in the future. Supporting either flag is a feature.

3 Likes
112

Kudos to the PEP 810 team for writing this PEP. As the Sponsor and an advocate for PEP 690, I think you did a really good job at addressing the major concerns leading to the rejection of 690. Let’s hope you’ve struck a winning balance this time.

Here’s my personal feedback on the PEP after having read it and at least trying to do a good faith review of this thread. I’ll just assume that anything you ignore has already been covered.

About the filter function

The following questions come to mind regarding the filter function.

  • Is it guaranteed to be called in a thread-safe manner? Being a process-global function (since it’s currently specified to be set/get by a sys function, but more on that below), are there any special considerations that are needed, either from the interpreter or from the author of the filter function? The section on thead-safety doesn’t address this directly, but I assume at least the intent is that yes, it is thread safe. That section could add an explicit guarantee.
  • The FAQ about performance does not address any overhead of calling the filter function, whether the performance overhead is different if a filter function is defined or not, and whether there are any optimizations when no filter function is defined, which I expect to be the norm. The section in the PEP should also be clearer when the filter function is called. I read it as saying it’s called at the point of the lazy import, not at reification time. ISTM calling a filter function has to impose some performance hit, let alone any logic inside the (user defined) filter function itself, so I’d like some discussion about that in the PEP either way.

Alternatives

  • I’m personally fine with proxy approach instead of the subclass-of-dict. I thought 690’s use of a modified dict was clever, but the proxy approach proposed in 810 is a better, more explicit solution, without the potentially global unintended side-effects. I don’t think a subclass gives any benefit.
  • I’m fine with lazy as the keyword, with defer being a decent second choice.
  • I’m also fine with the lazy from form. I actually think it’s a good thing that lazy imports of either stripe always start with the word lazy. I think that’s a better choice than mixing lazy import and from lazy import. (I also don’t think it’s worth any syntax deprecation to make from . lazy import work as you’d want it to.)

Other thoughts before the bikeshedding begins

  • In the migration FAQ, I think you should mention -X importtime as a way to help “identify slow-loading modules”, and perhaps give references to other profiling tools. Eventually, this could use a devguide or howto treatment.
  • In the rejection of from lazy section, you say “This is because from . lazy import bar is legal syntax (because whitespace does not matter)”. I suggest you make this more explicit: "this is because from . lazy import bar is legal syntax, and is equivalent to from .lazy import bar.
  • “Why you choose lazy as the keyword name” isn’t grammatically correct. Perhaps it’s missing a “did”?
  • I get that module.__dict__ will reify any lazy imports, but I’m somewhat concerned about the readability of seeing bare module.__dict__ in code. Would it not be better to add a .reify() method to module objects to make that explicit? It could no-op if it’s already reified.
  • The FAQ talks about the ability to remove if TYPE_CHECKING guards with lazy imports, which is great, but it does make it a little more difficult to see at a glance which names are imported just for the non-runtime type checker. Still, as I’ve been a fan of a TYPE_CHECKING built-in (to avoid the need to import it from typing), I think this is probably an overall win. What I am missing is a discussion about this PEP’s effect on static type checkers. Will they have to worry about every import rather than just the ones in a if TYPE_CHECKING block? It’s a naive question because I don’t actually know whether type checkers look for those blocks.
  • Along those lines, would it make sense to always treat imports in if TYPE_CHECKING blocks as lazy? This, along with TYPE_CHECKING as a built-in could make for a very easy migration to lazily importing all names that are imported just to satisfy static type checkers. I kind of also wish there were just lazy blocks, e.g.:
 as lazy:
    import foo
    from bar import baz

Bikeshedding FTW!

  • __lazy_modules__ → __lazy_imports__ to more closely mirror the lazy import syntax and -X flag.
  • Shouldn’t set_lazy_imports_filter() and get_lazy_imports_filter() live in importlib instead of sys?
  • -X lazy_imports=<modes> mode names: default → normal, enabled → all, disabled → none.

I think that’s it for now!

24 Likes
113

I didn’t mean to post this quote. But since it accidentally got posted. This is a great idea

1 Like
114

First thoughts after one pass reading the spec:

  1. new syntax for something that seems niche is too much
  2. there’s gotta be footguns here and there

Other thoughts:

The spec focuses on cli --help in argumentation. I wonder what other cases there may be. AWS Lambda and similar “severless” functions? Container startup for regular services when scaling out? Web pages? Bash completions?

I would like to see one large cli app updated with the new syntax to get some real, hard data. If I may suggest, charmcraft is one app where no major work was done to make cli fast. IIRC charmcraft --help takes >1s. Without the work done on a real app, I would personally consider the spec too abstract.

Case in point, consider smth like this:

# module foo
DEFAULT_PORT = 8080

# module bar
lazy from foo import DEFAULT_PORT

@app.command()
def serve(port: int = typer.Option(DEFAULT_PORT, "--port", "-p", help="Port to listen on")):
    with socketserver.TCPServer(("", port), http.server.SimpleHTTPRequestHandler) as httpd:
        ...

Without digging into the code, it’s easy to imagine that lazy imports solve slow startup, but here, the foo module needs to be loaded/run in order to resolve the constant.

2 Likes
115

The work has already been done as described in the motivation section of the PEP; the countless projects will simply migrate to the built-in mechanism.

3 Likes
116

I imagine linters will start checking for this pattern (a lazy import that gets used at module load time). It’s easy to do in cases where the lazy name is used in the module scope. There are harder cases too, like where a function is called at module load time.

2 Likes
117

Would this not work?

@app.command()
def serve(port: int = typer.Option(None, "--port", "-p", help="Port to listen on")):
    if port is None:
        port = DEFAULT_PORT
    with socketserver.TCPServer(("", port), http.server.SimpleHTTPRequestHandler) as httpd:
        ...
1 Like
118

In the case

import sys

lazy import mymodule
sys.path.append('./mypath_v1/')
mymodule.do_something()

lazy import mymodule
del sys.modules["mymodule"]
sys.path.remove('./mypath_v1/')
sys.path.append('./mypath_v2/')
mymodule.do_something()

I guess the first do_something() call will use the mymodule in the path ‘v1’ and the second in the path ‘v2’ (Assuming there is no ‘mymodule’ in the starting paths)… I am right ?
Shouldn’t the PEP be explicit about what is done in such a case ?

1 Like
119

I assume you meant to include del sys.modules["mymodule"], as otherwise the second import will use the cached version, just as it would without lazy.

I would expect the second call to use path v1, just as it would if lazy wasn’t present. I would expect the first to raise an ImportError, again just as it would without lazy, although this is more confusing as the import-time error gets deferred to time of use, because of the laziness.

I do agree with @ncoghlan though, this should be explicitly discussed in the PEP - is lazy import intended to be semantically equivalent to a normal import at the same point, or is it intended to be equivalent to “do the import at the last possible moment”? The former is easier to reason about in the face of code that manipulates the state of the import system, but the latter is arguably more intuitive in the abstract.

IMO, these are all good suggestions that improve clarity. Having said that, I’m not going to argue if the PEP authors choose to stick with the current names.

11 Likes
120

These are some not fleshed out idle rumination that might be interesting and I didn’t see them mentioned yet, but please feel free to ignore.

I wonder if you considered whether there is a more general mechanism which is orthogonal to importing here?

To quote the Python reference:

“The import statement combines two operations; it searches for the named module, then it binds the results of that search to a name in the local scope. The search operation of the import statement is defined as a call to the __import__() function, with the appropriate arguments. The return value of __import__() is used to perform the name binding operation of the import statement”

One of the differences of Python compared to compiled languages is that imports are resolved at runtime. Simplifying a lot, import foo is equivalent to foo = __import__("foo").

The PEP doesn’t change anything about the actual import mechanism (again, simplifying), only the foo = <something> part. The question then is, is it possible to implement a general feature such that <something> can be anything, not just __import__()? A possible syntax (not important for the discussion) can be lazy foo = __import__("foo"), with lazy import a syntax sugar. And it would work in any scope, not just module scope.

There are use cases for such a feature. I’ve seen lots of Python code over the years which mimics lazy evaluation, whether using explicit lambda thunks or proxy objects. Here is Django’s which is extensively used.

A complicating factor is that are two possible behaviors for such a feature, each with its own use cases. One is to cache the result after the first use, “call by need” (what lazy import wants), and one is to reevaluate on each use, “call by name” (what Django sometimes wants).

Another complicating factor is that in some scopes it might require locking.

BTW, I also wonder whether you consider lazy import to have semantic meaning or whether it is conformant to always treat lazy import as an eager import. If it’s the former, I wonder how it would affect Python compilers like mypyc if a lot of imports start becoming lazy. I don’t have internal knowledge of such tools, but wouldn’t lazy imports inherently prevent some important optimizations like cross-module inlining?

121

Ok, edited.

Thus the lazy import will browse the paths eagerly, and instanciate the objects lazily ???
If yes, is there any reason not to raise the ImportError eagerly ?

122

Not necessarily. It can just cache the value of __import__ (which includes the state of the import machinery in a closure), and do the path search lazily.

That’s just what it can do. What it actually will do is a question for the PEP authors, not me, to answer.

3 Likes