# List providing in-place operations on subsets using a (new) operator based selection notation

Howdy Folks,

currently I am pondering about how to implement an enhanced list, which provides an easy, intuitive and short notation for selecting subsets to be taken into account by specific list methods (it must be Python 2.7.18 compatible). The minimal code snippet belonging to the best idea I came up so far, does the following (where the method ‘elute’ just is an example):

``````   print ( enhList1 )
print ( enhList1.elute( (Elem > 5) & (Elem < 11) ) )            #new notation
print ( enhList1 )
``````

``````[1, 3, 5, 7, 9, 11, 13, 15]
[7, 9]
[1, 3, 5, 11, 13, 15]
``````

and

``````   print ( enhList2 )
print ( enhList2.elute( (Elem.a > 5) & (Elem.a < 11) ) )       #new notation
print ( enhList2 )
``````

``````[a = 1, a = 3, a = 5, a = 7, a = 9, a = 11, a = 13, a = 15]
[a = 7, a = 9]
[a = 1, a = 3, a = 5, a = 11, a = 13, a = 15]
``````

and

``````   print ( enhList3 )
print ( enhList3.elute( (Elem[1] > 5) & (Elem[1] < 11) ) )     #new notation
print ( enhList3 )
``````

``````[[1, 1, 1], [3, 3, 3], [5, 5, 5], [7, 7, 7], [9, 9, 9], [11, 11, 11], [13, 13, 13], [15, 15, 15]]
[[7, 7, 7], [9, 9, 9]]
[[1, 1, 1], [3, 3, 3], [5, 5, 5], [11, 11, 11], [13, 13, 13], [15, 15, 15]]
``````

and

``````   print ( enhList4 )
print ( enhList4.elute( (Elem['a'] > 5) & (Elem['a'] < 11) ) ) #new notation
print ( enhList4 )
``````

``````[{'a': 1}, {'a': 3}, {'a': 5}, {'a': 7}, {'a': 9}, {'a': 11}, {'a': 13}, {'a': 15}]
[{'a': 7}, {'a': 9}]
[{'a': 1}, {'a': 3}, {'a': 5}, {'a': 11}, {'a': 13}, {'a': 15}]
``````

So, my question now is: any ideas to do that better? Any ideas to improve that? Any ideas for a more elegant solution? And last but not least: any ideas how to change the code to also be Python 3.* compatible (yet it leads to: “TypeError: ‘>’ not supported between instances of ‘type’ and ‘int’”)?

The belonging code is the following:

``````# -*- coding: utf-8 -*-

#import common libraries

#helper class - element selection string type for enhanced list operations
class EnhList_SelStr(str):
"""
An instance of EnhList_SelStr is resp. can be given as the parameter
when calling the 'elute' method of an 'EnhList' instance.

It is a string type, which can be evaluated to determine,
whether an element of the enhanced list has to be taken into account
by a belonging list operation or not.

It can be extended using comparison operators and
the bitwise and/or operators as.

The current element to be taken into account (or not) is denoted by
'elem'; if the EnhList_SelStr e.g. is:

'elem > 5'

the current list element is taken into account, if the evaluation is true,
which in this example means: if the element is bigger than 5.
"""

#initialisation
def __init__(self, *params):
""" Ensure the standard str behaviour. """

str.__init__(self, *params)

#generate comparison methods __lt__ ... __ne__,
#called if "<" ... "!=" are used on self
for operatorT in ( ('__lt__', '<'), ('__le__', '<='), ('__eq__', '=='), \
('__gt__', '>'), ('__ge__', '>='), ('__ne__', '!=')  ):
exec( """def {0}(self, ohs):
return EnhList_SelStr("(%s {1} %s)" % (self, ohs))
""".format(*operatorT))

#called if the 'bitwise and' operator '&' is used on self
def __and__(self, ohs):
""" As there is no hook for the 'logical and' operator,
the 'bitwise and' is 'abused' instead.               """

return EnhList_SelStr("(%s and %s)" % (self, ohs))

#called if the 'bitwise or' operator '|' is used on self
def __or__(self, ohs):
""" As there is no hook for the 'logical or' operator,
the 'bitwise or' is 'abused' instead.                """

return EnhList_SelStr("(%s or %s)" % (self, ohs))

#helper class - meta class (just) for the Elem class
class EnhList_ElemMeta(type):
"""
This metaclass (just) is used to hook into the get attribute resp.
get item mechanism of the CLASS Elem (not of it's instances!).
"""

#method called by "Elem.nameS"
@classmethod
def __getattribute__(cls, nameS):
""" """

return EnhList_SelStr("elem.%s" % nameS)

#method called by "Elem[keyO]"
@classmethod
def __getitem__(cls, keyO):
""" """

#distinguish between a str key and an object (name) key
if    isinstance(keyO, str):
return EnhList_SelStr("elem['%s']" % keyO)
else:
return EnhList_SelStr("elem[%s]" % keyO)

#generate comparison methods __lt__ ... __ne__,
#called if "<" ... "!=" are used on self
for operatorT in ( ('__lt__', '<'), ('__le__', '<='), ('__eq__', '=='), \
('__gt__', '>'), ('__ge__', '>='), ('__ne__', '!=')  ):
exec( """@classmethod
def {0}(cls, ohs):
return EnhList_SelStr("(elem {1} %s)" % ohs)
""".format(*operatorT))

#called if the 'bitwise and' operator '&' is used on self
@classmethod
def __and__(cls, ohs):
""" As there is no hook for the 'logical and' operator,
the 'bitwise and' is 'abused' instead.               """

return EnhList_SelStr("(elem and %s)" % ohs)

#called if the 'bitwise or' operator '|' is used on self
def __or__(cls, ohs):
""" As there is no hook for the 'logical or' operator,
the 'bitwise or' is 'abused' instead.                """

return EnhList_SelStr("(elem or %s)" % ohs)

#placeholder class
class Elem(object):
"""
Can be used to create an EnhList_SelStr by using a new notation
comprising attributes, indices, keys, comparison and bitwise and/or
operators, for example:

Elem.a < 5

returns

EnhList_SelStr('elem.a < 5')

- which can be evaluated using eval().
"""

__metaclass__ = EnhList_ElemMeta

#enhanced list
class EnhList(list):
"""
List with enhanced capabilities.
"""

#initialisation
def __init__(self, *params):
""" """

#handle parameters
if    len(params) < 2:
#if none or one, keep the behaviour of the underlying
#built-in list type
list.__init__(self, *params)

else:
#if more than one, use them as initial list elements
list.__init__(self,  params)

#init lock
self._lock = Lock()

#method called by [key] operation
def elute(self, selS):
"""
Pop all elements from self for which the evaluation of selS
is True ('in place' operation).

'elem' can be used in selS to address each element.

The 'selS' parameter must be a string, which can be eval(uat)ed -
have a look at the description of the helper class EnhList_SelStr.
"""

#ensure, that the parameter is of type str resp. EnhList_SelStr
assert isinstance(selS, str), \
"The parameter 'selS' of the 'elute' method must be a str!"

#lock - (thread) locking mechanism just indicated here / yet
self._lock.acquire(True)

elutedL = EnhList()
indexI  = len(self) - 1
while indexI >= 0:

#'elem' is used in selS, see EnhList_ElemMeta.__getattribute__ resp.
#EnhList_ElemMeta.__getitem__ and so on
elem = self[indexI]

#just take elements into account, which fit to the
#evaluation of selS (for which selS is true)
if eval(selS):
elutedL.insert(0, self.pop(indexI))

#next element
indexI -= 1

#unlock
self._lock.release()

#return list with eluted elements
return elutedL

#########################################
### the following just is for testing ###
#########################################
class O(object):
""" """

def __init__(self, val):
""" """

self.a = val

def __repr__(self):
return "a = %s" % self.a

class L(list):
""" """

def __init__(self, val):
""" """

list.__init__(self, (val, val, val))

class D(dict):
""" """

def __init__(self, val):
""" """

dict.__init__(self, a=val)

#main
if __name__ == "__main__":

print ()
#list of objects with attributes
enhList1 = EnhList( range(1,17,2) )
print ( enhList1 )
print ( enhList1.elute( (Elem > 5) & (Elem < 11) ) )            #new notation
print ( enhList1 )
print ()

#list of objects with attributes
enhList2 = EnhList( map(O, range(1,17,2)) )
print ( enhList2 )
print ( enhList2.elute( (Elem.a > 5) & (Elem.a < 11) ) )       #new notation
print ( enhList2 )
print ()

#list of lists
enhList3 = EnhList( map(L, range(1,17,2)) )
print ( enhList3 )
print ( enhList3.elute( (Elem[1] > 5) & (Elem[1] < 11) ) )     #new notation
print ( enhList3 )
print ()

#list of dictionaries
enhList4 = EnhList( map(D, range(1,17,2)) )
print ( enhList4 )
print ( enhList4.elute( (Elem['a'] > 5) & (Elem['a'] < 11) ) ) #new notation
print ( enhList4 )
print ()
``````

Thanks and Cheers, Dominik

1 Like

You could look at placeholder · PyPI for a package that lets you write `_.r > 5` to mean the same as `lambda x: x.r > 5`.

I tend to like returning new stuff from functions rather than modifying stuff in place, so I might write something like this:

``````def partition(iterable, predicate):
mapping = [[], []]
for x in iterable:
mapping[predicate(x)].append(x)
return mapping

false_stuff, true_stuff = partition("Pyth0n", str.isdigit)
print(false_stuff) # ['P', 'y', 't', 'h', 'n']
print(true_stuff) # ['0']
``````

You could easily then do this to operate in place:

``````arr[:], selected = partition(arr, predicate)
``````
2 Likes

Hi Dennis,

thanks for your ideas! I did not know the placeholder library - nice

I indeed first started using a function as the return value of the overloaded operator methods too, but I did not came up with an easy solution to implement more complex, assembled condition terms alike:

((_ .r > 5) and (_ .r < 10))

which works fine with my “str-approach”:

((Elem.r > 5) & (Elem.r < 10))

Do you see a simple way to implement something alike that using the placeholder library? Overloading __ and __ and __ or __ seem not to work for function objects?

Cheers, Dominik

I did not use said ‘placeholder’ library in the end, but, when I had a look into it’s source code, I stumbled over ‘functools.partial’; using that seems to have done the trick…

So, thanks again!

Cheers, Dominik

Howdy Folks,

In the meanwhile said enhanced list already is working fine - together with said operator notation - in Python 2.7.18. Yet I am trying to make it also Python 3.* compatible. When I run the test functions in Python 3.3.0, I get the following error:

``````fct = Element < 5
TypeError: unorderable types: type() < int()
``````

whereby ‘Element’ is a class (inherited from ‘partial’) with a __ metaclass __ = ElementMeta and ElementMeta has an overloaded __ lt __ class method returning a ‘partial’. In Python 2.7 that works fine.

``````fct(4) ==> True
fct(6) ==> False
``````

What is the problem in Python 3.*? How can I solve it?

Thanks and Cheers, Dominik

Remember that `Element < 5` tries calling `type(Element).__lt__(Element, 5)`. So you need to make Element an instance of some class, rather than a class itself.

If that doesn’t work and Element really needs to be a class for some reason, you could use a metaclass to change `type(Element)` to be something else.

1 Like

Hi Dominik,

Metaclass issues can be very complicated to solve. You will probably
have more luck in getting an answer if you show us a minimal
reproducible example that works correctly in Python 2 but not in 3.

http://www.sscce.org/

This works for me in version 3:

``````>>> class Meta(type):
...     def __lt__(cls, other):
...             print(cls, other)
...             return True
...
>>> class A(metaclass=Meta):
...     pass
...
>>> A < 45
<class '__main__.A'> 45
True
``````

but of course I have no idea what your code is doing.

1 Like

Hi Dennis,

thanks a lot - your hint

got me back on track; after reading that, I asked myself, why I have chosen a class instead of an instance and came to the conclusion, that the belonging reason does not apply any longer.

Sometimes you cannot see the forest for the trees…

I changed that and now it seems to work under 2.7.* as well as under 3.*.

Cheers, Dominik

Hi Steven,

interesting - thanks for the example!

A minimal example of what I did, could be:

``````from functools import partial
from operator  import lt

class Meta(type):
@classmethod
def __lt__(cls, ohs):
return lambda x: lt(x, ohs)

class Element(partial):
__metaclass__ = Meta

fct = Element < 45
print( fct(44) )
print( fct(46) )
``````

which works fine under Python 2.7.18 and leads to

`TypeError: '<' not supported between instances of 'type' and 'int'`

under e.g. Python 3.6.13.

But if the following is used instead:

``````class Element(partial, metaclass=Meta):
pass
``````

it works fine under 3.6.13 (but not under 2.7.18 anymore).
Any idea why the “__ metaclass __” version leads to said error JUST under 3.*?

Cheers, Dominik

Howdy Folks,

the library now is available at:

enhaaancedLists · PyPI

Some examples for ADDITIONAL capabilities (standard list operations work too)
available when using said library are:

``````#convert a parameter list to an enhanced list
eL = EnhList(1,3,5,7)                                       #eL: [1,3,5,7]

#push single as well as multiple elements into the list
eL.push(9)                                  ==> None        #eL: [1,3,5,7,9]
eL.push(11,13,15)                           ==> None        #eL: [1,3,5,7,9,11,13,15]

#pop single as well as multiple elements from the list -
#note that push/pop implements a FIFO - in contrast to the standard list (LIFO)
eL.pop()                                    ==> 1           #eL: [3,5,7,9,11,13,15]
eL.pop( (elem > 3) & (elem < 11), single )  ==> 5           #eL: [3,7,9,11,13,15]
eL.pop( (elem > 3) & (elem < 11)         )  ==> [7,9]       #eL: [3,11,13,15]

#get items from list
eL[ elem >= 10         ]                    ==> [11,13,15]  #eL: unchanged
eL[ elem >= 10, single ]                    ==> 11          #eL: unchanged
eL[ elem <  3,  single ]                    ==> None        #eL: unchanged

#check whether list contains items
( elem <  3 ) in eL                         ==> False       #eL: unchanged
( elem >= 3 ) in eL                         ==> True        #eL: unchanged

#delete items from list
del eL[ elem < 12, single ]                 ==> ---         #eL: [11,13,15]
del eL[ elem > 12         ]                 ==> ---         #eL: [11]

eL = EnhList(1,3,5,7)                                       #eL: [1,3,5,7]
#check whether all element meet a condition
eL.areAll( elem % 2 == 1 )                  ==> True        #eL: unchanged
eL.areAll( elem     >= 3 )                  ==> False       #eL: unchanged

#map function on elements / work with items of elements
eL.mapIf( lambda x: dict(a=x) )
==> None        #eL: [{'a':1},{'a':3},{'a':5},{'a':7}]
eL.mapIf( lambda x: x['a'] + 1, elem['a'] > 3)
==> None        #eL: [{'a':1},{'a':3},6,8]

#work with attributes of elements
class Attr(object):
def __init__(self, value):
self.a = value
def __repr__(self):
return ".a=%s" % self.a
eL.mapIf( lambda x: Attr(x), lambda x: type(x) ==  int )
==> None        #eL: [{'a':1},{'a':3},.a=6,.a=8]
``````

Have Fun and Cheers,
Dominik

Hi Dominik,

“Any idea why the “__ metaclass __” version leads to said error JUST
under 3.*?”

`__metaclass__` has no special meaning in Python 3. The 2to3 fixer
changes it to the Python 3 version.

https://docs.python.org/3/library/2to3.html?highlight=metaclass#2to3fixer-metaclass

https://docs.python.org/3/reference/datamodel.html?highlight=metaclass#metaclasses

1 Like

Hi Steven,

I see - thanks!

Cheers, Dominik

Hello World,

version 0.75 is available:

A ‘SecList’ class has been added. It is a secured version of the aforementioned enhanced list class ‘EnhList’.

Access to its elements has been made ‘thread-safe’ by wrapping the belonging methods in a ‘with’ context automatically ‘acquiring’ / ‘releasing’ an internal ‘SemiBlockingMutex’ (a special multithreading / multiprocessing lock).

Example:

``````   #convert a parameter list into a secured list
sL = SecList(1,3,5,7,9,11,13)                                #sL: [1,3,5,7,9,11,13]

#if then a first thread e.g. would run the following statement:
poppedLtL = sL.pop( elem < 9 )

#and a second thread in parallel (!!) e.g. would run the following statement:
poppedGtL = sL.pop( elem > 7 )

#there would be no error and the result would be:
#poppedLtL <==> [1,3,5,7]
#poppedGtL <==> [9,11,13]
#sL        <==> []
``````

Have fun, cheers
Dominik