Makes sense. So, yeah, once again, deferreds are doing a bad job of their only possible use-cases. Seems to me I’ve heard this song before.
Could you elaborate a bit what you mean by this?
No. The nonlocal keyword is ONLY needed when you assign to a name. It is not required when you only refer to it. This is the same as with the global statement. It would confuse people GREATLY if suddenly there’s one place where that rule is violated.
Sure. Have a quick look back at my example with eval for context.
A function’s local variables are, by default, only valid while that function is executing. (I’m going to assume it’s not an async or generator function here; the rules are a bit more complicated for those, but the concepts are the same.) The function begins, then it ends, and every variable it had is gone. Any objects that were referenced in those variables and not anywhere else can now be disposed of. This is true even if the function has an inner function:
def outer(x):
def inner():
print("I am the inner function!")
return inner
Yes, the inner function might still exist, but it doesn’t refer to x, so that variable can be disposed of. If that’s a gigantic thing that takes up a crazy amount of memory, that can make a pretty big difference!
But what if the inner function DOES refer to that variable? Well, now every use of that name, whether in the outer or the inner function, has to be a slower “closure-friendly” lookup.
def outer1(x):
def inner():
print("Hello, world!")
x += 2
return inner, x
def outer2(x):
def inner():
print("Hello, world!", x)
x += 2
return inner, x
$ python3 -m timeit -s 'from deref import outer1' 'outer1(5)'
5000000 loops, best of 5: 47.3 nsec per loop
$ python3 -m timeit -s 'from deref import outer2' 'outer2(5)'
5000000 loops, best of 5: 65.8 nsec per loop
This isn’t TOO bad - closure references are also quite highly optimized - but it’s a price you don’t want to pay without needing to. And that’s how it is in CPython - you only pay this when there is actually a closure that makes reference to a particular name. If there were multiple variables and only some of them were referenced by the inner function, only those variables would become closure cells, so you don’t pay the price for the others.
Enter deferred expressions. Suddenly, ANYTHING could be a deferred. That print function? Well, maybe there’s a deferred expression injected into the module globals. We can’t know. With the semantics that I was responding to at the time (by the way, one of the REALLY annoying things about debating these kinds of half-baked proposals is that the semantics are like jelly nailed to a tree, in the hot sun, with a swarm of ants going for it - by the time you look at it, it’s already changed or vanished), the “undeferring” of that thing would potentially reference names in that scope. That means, just in case, Python has to store all variables in closure cells.
And that’s a price we should not have to pay, both in performance and in nightmarish action-at-a-distance.
def outer1(x):
def inner():
print(eval('x'))
x += 2
return inner
def outer2(x):
def inner():
nonlocal x
eval('print(x)')
x += 2
return inner
# outer1(5)() # NameError
outer2(5)() # 7
It can already be used in such manner. So the confusion is already there - although it is not confusing to me at all. If it fits the purpose, then it can be needed elsewhere too.
So I get that. But how is this different from the current situation with lambdas? Anything can be referenced from within lambda function too.
This wouldn’t introduce anything new, just re-use the mechanism of eval (default) & lambda via nonlocal mechanism.
Also, if nonlocal mechanism is not used, then it wouldn’t introduce any overhead - same as eval.
Furthermore, theoretically the overhead should be even lower, as scopes from which variables are taken can be destroyed as soon as lazy object is instantiated as it fixes value and does not track the reference.
You are right. Except that this proposal is more like 1/10th baked. I just like getting feedback as early as possible before going too far in the wrong direction.
Current description of lazy:
- Lazy fixes values at instantiation
- Variable lookups:
a) by default emulateseval’s behaviour
b) one can choose lambda behaviour with: nonlocal prefix - It can not be modified after instantiation
- It evaluates implicitly on referencing
- Once evaluated it caches the result and does not re-evaluate
def outer_lazy(x):
inner_lazy = lazy: x
x += 2
return inner_lazy
print(outer_lazy(5)) # 5
def outer_lazy(x):
def inner_lazy():
return lazy: x
x += 2
return inner_lazy
print(outer_lazy(5)) # NameError
def outer_lazy(x):
def inner_lazy():
return lazy: nonlocal x
x += 2
return inner_lazy
print(outer_lazy(5)) # 7
1 Like
Go back and reread what I said. You don’t have to put everything into closure cells just because a lambda MIGHT exist. They are compiled properly like every other function.
SIGH. So basically, you’re saying that you are quite happy to inflict on the rest of us the hassle of discussing something that keeps changing, has no defined semantics, and which you’re happy to be dismissive of problems because hey, it’s sure to be solved down the track.
That is NOT a good thing. Just to be clear here. Figure out your proposal FIRST, and THEN get feedback on it.
For how long? Is it going to be slippery and vanish away the moment I try to read the proposal? Am I wasting my time discussing this with you? It really feels like it.
2 Likes
@dg-pb I implemented lazy imports as a library solution.
I think the approach is only useful for handling imports, but it might be interesting for you. Maybe it will give you some new ideas.
Let me know if you have any questions.
1 Like
Could you kindly spell it out to me? I might be missing something. What is so fundamentally different with lazy in comparison to lambda? Lazy would follow similar protocol to lambda. If lambda references a variable from outer scope - then that variable gets put in a cell. Same with lazy, why would you need to put all variables in a cell in the case of lazy?
You put it in very negative light. I would rephrase: "I am happy to hare my initial insights and those who like to participate in consideration of this pretty early idea are welcome. Those who do not, are responsible for their own time. I take my responsibility of transparency seriously so that nobody wastes time due to misinformation. And in my experience finding out potential best as early as possible does lead to better results most of the time - so yes, I am quite happy to get out of “why not” for the time being to see “what could be”.
That is up to you to decide. You can help out by pointing things out that I am not aware of. Or you can wait until its at least half baked.
2 Likes
Thanks to you, hooks will be helpful to implement nonlocal refactoring in prototype.
Btw, ast.fix_missing_locations did become bottleneck in my endeavours. It is fairly easy to manually set 2 attributes during creation of nodes and could lead to reasonable speed improvements (depends on the context of course).
1 Like
The point is that every time this has come up previously, the single most prevalent response is “provide a complete proposal that has fully-defined semantics which you will stand by, and which people can reason about without the rules changing as the discussion proceeds”.
Not only have you ignored this feedback, you’ve explicitly stated that all you are offering is yet another 10%-complete solution.
And while yes, everyone is responsible for their own time, many of the more experienced posters here (in which I include myself and Chris) feel a responsibility to try to guide newcomers in how to make a successful proposal. Your statement “Those who do not, are responsible for their own time” feels to me like a pretty ungrateful way of responding to the time we’ve[1] put into trying to help, just because you don’t like[2] the advice you’re being given.
For any proposal to be anything more than a waste of everyone’s time, it will need the support of a core developer. At this point, I think you’ve lost the interest of every core developer who’s previously participated in these discussions except me - and I’m only still here in the hope that you’ll take the advice that’s been given (a hope that feels increasingly naïve to me, I’ll be honest 
)
Anyway, it’s up to you to decide whether you want to continue posting your ideas - but if you want to be respectful of other people’s time, I suggest you follow some of the advice you have been given.
3 Likes
I wasn’t present in any of the previous attempts. Thus, it didn’t feel like I am ignoring the feedback, which I am not privy to. I am aware that lack of concrete proposal was an issue, just did not realise that it was a mutual consensus of “complete proposal or nothing”.
I am more than ok with any helpful advice, the manner in which the advice is delivered can sometimes be an issue. If there is lack of basic respect I don’t think I can offer it back. Especially with the people from which I had an opportunity to receive personal insults.
So what is the situation? Should I delete this whole thread? Or is it ok to allow input in the meanwhile if anyone wishes to.
Comments so far have been helpful. The stage is early. I will work towards a complete prototype in python and will try to learn a bit more about cpython.
1 Like
I would suggest:
- You do go back and read the previous threads. Otherwise, all you’ll be doing is forcing other people to restate the objections that have been made previously.
- You accept the point that having a complete proposal is the only practical way forward, and write one. It doesn’t have to be perfect, it doesn’t have to be the final answer, but it does have to have complete semantics and you do need to be prepared to defend those semantics (i.e., “oh, well I guess we could have it do this in that situation”, where you change the proposal every time a flaw is pointed out, isn’t acceptable).
Remember - this forum is for developing ideas for new features for Python. Such features have to be implementable, they have to satisfy a need, and they have to be built on what’s already present. And there’s a lot of history, both here and in the python-ideas mailing list before it. Ideas that have been round for years (like this one) have been discussed many times, and no-one has yet found a solution. You’re not likely to just by chance come up with something no-one has tried before. Rather, you need to find out what went wrong with previous proposals, and make sure that your proposal addresses those issues. And you need to do that research, otherwise all you’re doing is expecting someone else to do it for you. We’ll help by pointing out things you’ve missed, but we’re not going to take you through all the old arguments if you don’t show at least some evidence that you’ve done the background research.
Your time would probably be better spent writing a specification. While it’s nice to have an executable prototype, and an implementation will be necessary if this idea is ever to go anywhere, for now what people want is the spec. So we can reason about what will happen if we use this feature in real code. I don’t need to run the code to know what a, b, c = range(10) will do - the language specification tells me[1]. Same here - write the docs, not the implementation for now.
It even answers questions like “yes, but what if I assigned a new value to the name
range?” ↩︎
7 Likes
Ok, cool.
Regarding things that have been done before. I welcome any resources. I have only read what I found at the time, which is the short document of last attempt and most of accompanying discussion (In addition to few PEPs that are related, but not the same thing).
Yes, I did. Your proposal is not in a fit state to discuss, as evidenced by the way it shifts and morphs when people try to discuss it. I put that in a negative light because it is a major downside to your proposal. Or rather, it’s a major downside to any discussion of it; you might see it as a great way to get feedback, but you’re wasting the time of everyone who participates. Why should we discuss something when you aren’t sure what your proposal even is?
Please can you stop asking other people to do your research for you? This forum. And from there, you’ll find links to the PEPs repository and the python-ideas mailing list archive.
Is the search feature too hard to use?
You always use “we” instead of “I”. You are talking as if you represent the whole community, which if true, would be very disappointing.
For what it’s worth, Chris represents me as well when he says that your expectation that others will do the research for you isn’t acceptable. And while neither of us might represent the whole community, we are both trying to give you a sense of what the community consensus is (based on the fact that we’ve both been through a lot of these discussions, and similar ones on other topics).
Your unwillingness to listen to feedback intended to help you, given in good faith, with at worst a certain level of terseness as a result of us being fed up repeating the same message over and over again, is at least as disappointing to us as the responses you are getting to your comments is to you.
1 Like
I am not asking to do a research for me.
If one includes a statement in the question it does not make it automatically true. Although, he made it seem that way and now you are on board with him on the same false statement.
Funny thing is that I agree with that statement as well - “one shouldn’t ask others to do his research”. The point is that the statement isn’t applicable to the situation at hand. And this is a continuous story with Chris A. - he distorts the truth about the situation by making “true statements” without question whether the opposite to those true statements is what is happening at the moment. And others buy into them. This is called a demagogy.
What I meant is if someone has a reference to material, which is 2 strokes away and could be easily missed, that would be of help. To me, this is a part of definition of teamwork - if I can save someones time by a multiple - I will, the other will do the same next time - in the end we both are better off.
Although, there are other structures of cooperation, I am not a big fan of those.
Also, I took liberty to to say this, as I did do research, but am still being suggested to do more, thus thinking maybe there is something, which I have missed and others can clearly see what is being repeated.
Also, I never ask for what I am not willing to offer myself.
Nobody said the information was “2 strokes away and could be easily missed”. The information is available in multiple discussions, in multiple threads, on this forum and the python-ideas mailing list (something I certainly thought I’d already said, even if you thought I meant something different…) But it’s hard to find, not only for you but also for me. However, and this is the important point, it’s not me who has a need to find that information, it’s you. So I am not willing to go and look up links for you, even though I could, because that would involve me investing my free time in doing work which:
- You should be capable of doing for yourself, if you are going to be able to write a viable PEP for this proposal.
- You would gain from doing, both in understanding the scope of the discussion, as well as getting a better feel for what sort of issues people want to see addressed in any proposal.
- Means that you will get to see the information in its pure form, unfiltered by the views and prejudices of someone else.
If you want, you can consider it as a “homework assignment”. I don’t like that term, as it implies that I think of myself as teaching and you as learning, and in reality I have no idea what our relative levels of experience are in language design, Python, or programming in general. However, it does capture the important point, which is that I think that doing that research yourself will benefit you far more than simply being “fed” information by others.
1 Like
I didn’t say that somebody said that. I just said that I think it could be the case. If it is not, then there is nothing further to say.
I completely agree with the rest of what you said, I know what I need to do and this is what I was about to do regardless if anyone had some good references to offer or not.
Well, every question Chris has asked is something that he’s asked before on other threads (I know because I remember the questions coming up). I can’t remember where, and I imagine Chris can’t either, so we can’t offer links, but yes, you have missed things.
What I would do in your situation is:
- Read every thread and PEP that you can find with a title that talks about “better syntax” for lambdas, or “deferred evaluation” or “lazy expressions”, or any other term that seems relevant. If you don’t find at least 4 threads over the past 5 years or so, you’ve missed some.
- Find Chris’ PEP about argument defaults. I did that research for you - it’s PEP 671, which I found by searching the PEP index for “parameter”, and when that didn’t get it, “default”. My next step would have been to search the PEPs authored by Chris. Find the discussions about that (they should be linked in the PEP itself) and read them to see where they talked about, or spawned threads about, deferred evaluation. They definitely did, because that’s why Chris gets so frustrated when deferred evaluation keeps getting proposed
That should give you multiple threads with 50+ messages in them to review. Take notes, particularly about corner cases that people describe where the proposals don’t have a good answer. Make sure your proposal answers every one of those questions.
That’s a lot of work. But it was a lot of work for every participant in every one of those threads to contribute their views, and you should respect the time they spent and make sure you’re not asking them to repeat their points here, just because you didn’t find their original comments. Remember, it’s not their responsibility to monitor this category and make sure that their points don’t get ignored just because time has passed.
Also, remember that all the time I’ve spent trying to give you pointers here (and it’s a few hours worth of work by now) has been time that I could have spent reviewing your proposal, and offering new feedback[1]. So none of this has been productive for either of us - and your proposal isn’t any closer yet to being viable as a result.
or, for that matter, just doing something else! ↩︎
6 Likes