An ASCII art picture to support the discussion:
,----------------- index -3
| ,-------------- index -2
| | ,----------- index -1
| | | ,----- ?? index "-0" for appending/extending
v v v v
[A, B, C ... X, Y, Z] _
^ ^ ^
| | `---- index 2
| `------- index 1
`--------- index 0
What are the options so far?
a) Stick with a[-n or sys.maxsize:] (what if len(a) > sys.maxsize?)
b) Custom sentinel
c) Re-using some other existing object for sentinel
Not sure whether there are any viable options for (c)⌠It seems that taking any existing object would inevitably break backwards compatibilityâŚ
I imagine (b/c) could be implemented fairly easily such that:
@lambda x: x()
class a:
def __len__(self):
return 10
def __getitem__(self, idx):
print(idx)
a[END:] # slice(10,None,None)
a[:END] # slice(None,10,None)
So that [] call modifies slice internally.
(It seems that this is only needed for slice as single unit access for this does not make much sense)
It wonât be. That would mean 2**31 or 2**63 objects in a collection (depending on whether itâs a 32-bit or 64-bit build).
Ah, ok. I see that __len__ can not even return a higher value.
However:
@lambda x: x()
class a:
def __getitem__(self, n):
return n
items_from_end = 0
a[-items_from_end or sys.maxsize:]
# slice(9223372036854775807, None, None)
# But:
items_from_end = -1
a[-items_from_end or sys.maxsize:]
# slice(1, None, None)
So what I am getting at is that there is a slight gap in programatic use, where the only viable option is:
a[len(a) - items_from_end:]
And there is no equivalent shorthand to replace it.
I am fine with the syntax above most of the time.
However, from my experience, len does become a slight bottleneck when writing optimized code.
It is not that silly from my POV.
Generally, I am fine with a[len(a) - n] and performance improvement would be of most value to me.
Not sure if it would be as beneficial to others as it would be to me, but if so, there are 2 options:
a) a[END - n:] would provide best performance as both __len__ and __getitem__ would be done at the same time.
b) Alternative could be to optimize len in similar manner as a[...] is optimized as opposed to a.__getitem__(...). Not sure if this would even be possible within sensible effort. If it is, this wouldnât be as performant as (a), but would have effect outside of a[...] (while (a) wouldnât).
So my take is that if it was no-effort-no-cost extension, I would have both (a) and (b) if possible.
However, it is not and I doubt that such micro-optimizations matter to many peopleâŚ
-1 on existence of âsyntax issueâ
+1 on optimization possibilities
Then how a[End - n] would be faster than a[-n:] if n else ()? All __getitem__-s would need to check for a sentinel, how can this be faster?
That solution doesnât have the same functionality: it doesnât allow for assignment. Not something I find myself needing, but it was a use-case mentioned in the OP.
This check is pretty much free in background of __getitem__.
Btw, I am uncertain about whether it would be possible to retain performance benefits after properly designing such. I.e.:
# What would this do?
a[END - n:]
# END - n ???
class End:
offset = 0
def __init__(self, offset=0):
self.offset = offset
def __sub__(self, other):
return End(self.offset - other)
END = End(0)
And this would certainly kill the performance.
So the only benefiting case would be when exact END is required: e.g. a[END:].
If it is the most common use case, then it be fine, but I prefer solutions that are âone best wayâ for all (or at least vast majority of) casesâŚ
Thus, from performance perspective, I instead would probably try:
a) further improving performance of function calls
b) looking into possibility to specialise len
Regarding (b), I am not sure how realistic is that, but if it is, then it could be applied to other builtins as well.
For example, while obj[item] can be used programatically, obj.attr needs to be used differently:
%timeit Ellipsis.__class__ # 14 ns
%timeit getattr(Ellipsis, '__class__') # 46 ns
I would be very satisfied if performance of the latter would catch up with the former.
Could maybe be something along the lines of:
if len is builtins.len:
# Do not do explicit call, but run it more efficiently in C
Given 99% of the time builtin names are not overridden, this might be a good idea.
Such could be applied to certain portion of builtins, which might result in non-negligible performance boost.
And I think this is the wrong thread for thisâŚ
Replacing *a[len(a):] with *tmp, followed by a.extend(tmp), is currently faster (with subtracted len(a) time). There is no benefit in this case
Yup, although it would improve performance of OPs example *tokens[len(tokens):], remaining = remaining.partition(' '), but it would not be the most performant solution.
However, improving len performance would have impact for other cases, where len is inevitable.
I had cases where this would have made an impact.
When there is a thread for it, I can dig those up. One pattern that comes to mind:
seen = {}
seen_add = seen.add
for item in lst:
...
seen_add(item)
if len(seen) == 10:
break
Youâre raising an important compatibility point â ideally custom __getitem__ methods should not have to handle any new sentinel value, just numbers.
Which means [] syntax (BINARY_SLICE & BINARY_SUBSCR opcodes) and operator.getitem() have to get new responsibility of translating it to a number? (with some performance cost?)
Additionally, itâd be great to support a[END-n:END-m], with proper handling of 0. So the sentinel would want to overload __sub__ (and maybe __add__ too) â and at that point has to produce result without knowing yet the length of the collection.
Itâd be a shame to introduce some new âsymbolicâ representation of things like â3 before endâ without exposing it to __getitem__ protocol 
. Luckily, we can have END-n == -n for positive integers, and only END-0 == END as symbolic representation that [] / operator.getitem further converts to len(a).
c) Re-using some other existing object for sentinel
What about len builtin itself?
__sub__ overload.a[len-n:] would be easy to remember and read.foo as well as foo(...) in same context.len-3 == -3 but len-0 is len would be visible to any user wondering how it works and just trying what it returns. That may only increase confusion.len itself gets processed before __getitem__ sees it would be another unexpected wart.Yup, this is why I am personally not very positive about this. I have no big issues with syntax:
a[len(a)-n:len(a)-m]
and
a[END-n:END-m]
would not be that much more convenient to me given performance penalties and complexities added in such a niche manner at the core of the language.
But I am not against it to the degree to advocate against it if many people want this and can build a proper case for it. If it existed, I might use it occasionally in cases where this does not incur performance penalty (probably only one really - a[END:]).
But even then, if len(a) was optimised to the degree that say it was 3x faster than it is now, I would probably choose a[len(a):] instead as it would be nearly as fast and consistent across all needs.
Also, from performance perspective, in many cases len(a) can be called once and indices adjusted iteratively in parallel. And there are fairly few cases where len needs to called iteratively. A lot of the time performant code looks more like:
import random
_rng110 = lambda: random.randint(1, 10)
a = [_rng110() for _ in range(_rng110())]
print(a) # [7, 6, 5, 7, 5, 10, 3, 9]
# ---
n = len(a)
s, e = n - 2, n - 1
while s > 0:
del a[s:e]
e, s = s, s - 1
print(a) # [7, 9]
So personally I would rather put effort into global performance improvements to builtin calls (len included) as opposed to adding specialised complexity with performance penalties in most use cases.