List providing in-place operations on subsets using a (new) operator based selection notation

Howdy Folks,

currently I am pondering about how to implement an enhanced list, which provides an easy, intuitive and short notation for selecting subsets to be taken into account by specific list methods (it must be Python 2.7.18 compatible). The minimal code snippet belonging to the best idea I came up so far, does the following (where the method ‘elute’ just is an example):

   print ( enhList1 )
   print ( enhList1.elute( (Elem > 5) & (Elem < 11) ) )            #new notation
   print ( enhList1 )

leads to ==>

[1, 3, 5, 7, 9, 11, 13, 15]
[7, 9]
[1, 3, 5, 11, 13, 15]

and

   print ( enhList2 )
   print ( enhList2.elute( (Elem.a > 5) & (Elem.a < 11) ) )       #new notation
   print ( enhList2 )

leads to ==>

[a = 1, a = 3, a = 5, a = 7, a = 9, a = 11, a = 13, a = 15]
[a = 7, a = 9]
[a = 1, a = 3, a = 5, a = 11, a = 13, a = 15]

and

   print ( enhList3 )
   print ( enhList3.elute( (Elem[1] > 5) & (Elem[1] < 11) ) )     #new notation
   print ( enhList3 )

leads to ==>

[[1, 1, 1], [3, 3, 3], [5, 5, 5], [7, 7, 7], [9, 9, 9], [11, 11, 11], [13, 13, 13], [15, 15, 15]]
[[7, 7, 7], [9, 9, 9]]
[[1, 1, 1], [3, 3, 3], [5, 5, 5], [11, 11, 11], [13, 13, 13], [15, 15, 15]]

and

   print ( enhList4 )
   print ( enhList4.elute( (Elem['a'] > 5) & (Elem['a'] < 11) ) ) #new notation
   print ( enhList4 )

leads to ==>

[{'a': 1}, {'a': 3}, {'a': 5}, {'a': 7}, {'a': 9}, {'a': 11}, {'a': 13}, {'a': 15}]
[{'a': 7}, {'a': 9}]
[{'a': 1}, {'a': 3}, {'a': 5}, {'a': 11}, {'a': 13}, {'a': 15}]

So, my question now is: any ideas to do that better? Any ideas to improve that? Any ideas for a more elegant solution? And last but not least: any ideas how to change the code to also be Python 3.* compatible (yet it leads to: “TypeError: ‘>’ not supported between instances of ‘type’ and ‘int’”)?

The belonging code is the following:

# -*- coding: utf-8 -*-


#import common libraries
from threading import Lock


#helper class - element selection string type for enhanced list operations
class EnhList_SelStr(str):
      """ 
          An instance of EnhList_SelStr is resp. can be given as the parameter 
          when calling the 'elute' method of an 'EnhList' instance.
          
          It is a string type, which can be evaluated to determine, 
          whether an element of the enhanced list has to be taken into account 
          by a belonging list operation or not.
          
          It can be extended using comparison operators and 
          the bitwise and/or operators as.
          
          The current element to be taken into account (or not) is denoted by
          'elem'; if the EnhList_SelStr e.g. is:
          
          'elem > 5'
          
          the current list element is taken into account, if the evaluation is true,
          which in this example means: if the element is bigger than 5.
      """
      
      
      #initialisation
      def __init__(self, *params):
          """ Ensure the standard str behaviour. """
          
          str.__init__(self, *params)    
          
          
      #generate comparison methods __lt__ ... __ne__, 
      #called if "<" ... "!=" are used on self
      for operatorT in ( ('__lt__', '<'), ('__le__', '<='), ('__eq__', '=='), \
                         ('__gt__', '>'), ('__ge__', '>='), ('__ne__', '!=')  ):
          exec( """def {0}(self, ohs):
                       return EnhList_SelStr("(%s {1} %s)" % (self, ohs))
                """.format(*operatorT))
          
          
      #called if the 'bitwise and' operator '&' is used on self
      def __and__(self, ohs):
          """ As there is no hook for the 'logical and' operator, 
              the 'bitwise and' is 'abused' instead.               """
      
          return EnhList_SelStr("(%s and %s)" % (self, ohs))     
          
          
      #called if the 'bitwise or' operator '|' is used on self
      def __or__(self, ohs):
          """ As there is no hook for the 'logical or' operator, 
              the 'bitwise or' is 'abused' instead.                """
       
          return EnhList_SelStr("(%s or %s)" % (self, ohs))            
          
        
            
#helper class - meta class (just) for the Elem class
class EnhList_ElemMeta(type):
      """
          This metaclass (just) is used to hook into the get attribute resp. 
          get item mechanism of the CLASS Elem (not of it's instances!).
      """
      
      
      #method called by "Elem.nameS"
      @classmethod
      def __getattribute__(cls, nameS):
          """ """
          
          return EnhList_SelStr("elem.%s" % nameS)
          
          
      #method called by "Elem[keyO]"
      @classmethod
      def __getitem__(cls, keyO): 
          """ """
          
          #distinguish between a str key and an object (name) key
          if    isinstance(keyO, str):
                return EnhList_SelStr("elem['%s']" % keyO)     
          else:
                return EnhList_SelStr("elem[%s]" % keyO) 
                
                
      #generate comparison methods __lt__ ... __ne__, 
      #called if "<" ... "!=" are used on self
      for operatorT in ( ('__lt__', '<'), ('__le__', '<='), ('__eq__', '=='), \
                         ('__gt__', '>'), ('__ge__', '>='), ('__ne__', '!=')  ):
          exec( """@classmethod
def {0}(cls, ohs):
    return EnhList_SelStr("(elem {1} %s)" % ohs)
                """.format(*operatorT))
          
          
      #called if the 'bitwise and' operator '&' is used on self
      @classmethod
      def __and__(cls, ohs):
          """ As there is no hook for the 'logical and' operator, 
              the 'bitwise and' is 'abused' instead.               """
      
          return EnhList_SelStr("(elem and %s)" % ohs)     
          
          
      #called if the 'bitwise or' operator '|' is used on self
      def __or__(cls, ohs):
          """ As there is no hook for the 'logical or' operator, 
              the 'bitwise or' is 'abused' instead.                """
       
          return EnhList_SelStr("(elem or %s)" % ohs)                 
                      


#placeholder class
class Elem(object):
      """ 
          Can be used to create an EnhList_SelStr by using a new notation
          comprising attributes, indices, keys, comparison and bitwise and/or 
          operators, for example:
          
          Elem.a < 5
          
          returns
          
          EnhList_SelStr('elem.a < 5')
          
          - which can be evaluated using eval().
      """

      __metaclass__ = EnhList_ElemMeta    
         


#enhanced list
class EnhList(list):
      """ 
          List with enhanced capabilities. 
      """
      
      
      #initialisation
      def __init__(self, *params):
          """ """

          #handle parameters
          if    len(params) < 2:
                #if none or one, keep the behaviour of the underlying 
                #built-in list type
                list.__init__(self, *params)
                
          else:
                #if more than one, use them as initial list elements
                list.__init__(self,  params)
                
          #init lock
          self._lock = Lock()
          
          
      #method called by [key] operation
      def elute(self, selS):
          """ 
              Pop all elements from self for which the evaluation of selS 
              is True ('in place' operation).
              
              'elem' can be used in selS to address each element.
              
              The 'selS' parameter must be a string, which can be eval(uat)ed - 
              have a look at the description of the helper class EnhList_SelStr. 
          """
          
          #ensure, that the parameter is of type str resp. EnhList_SelStr
          assert isinstance(selS, str), \
                 "The parameter 'selS' of the 'elute' method must be a str!"

          #lock - (thread) locking mechanism just indicated here / yet
          self._lock.acquire(True)       
          
          elutedL = EnhList()
          indexI  = len(self) - 1
          while indexI >= 0:
            
                #'elem' is used in selS, see EnhList_ElemMeta.__getattribute__ resp.
                #EnhList_ElemMeta.__getitem__ and so on
                elem = self[indexI]
                
                #just take elements into account, which fit to the 
                #evaluation of selS (for which selS is true)
                if eval(selS):
                   elutedL.insert(0, self.pop(indexI))
                   
                #next element
                indexI -= 1
                
          #unlock
          self._lock.release()
                
          #return list with eluted elements
          return elutedL
      
      
      
      
      
      
#########################################
### the following just is for testing ###
#########################################
class O(object):
      """ """
      
      def __init__(self, val):
          """ """
          
          self.a = val    

          
      def __repr__(self):
          return "a = %s" % self.a  
          
          
          
class L(list):
      """ """

      def __init__(self, val):
          """ """
          
          list.__init__(self, (val, val, val))

      
      
class D(dict):
      """ """
      
      def __init__(self, val):
          """ """
          
          dict.__init__(self, a=val)
         
               
      
#main 
if __name__ == "__main__":
  
   print ()
   #list of objects with attributes 
   enhList1 = EnhList( range(1,17,2) )
   print ( enhList1 )
   print ( enhList1.elute( (Elem > 5) & (Elem < 11) ) )            #new notation
   print ( enhList1 )    
   print ()  
  
   #list of objects with attributes 
   enhList2 = EnhList( map(O, range(1,17,2)) )
   print ( enhList2 )
   print ( enhList2.elute( (Elem.a > 5) & (Elem.a < 11) ) )       #new notation
   print ( enhList2 )    
   print ()
    
   #list of lists 
   enhList3 = EnhList( map(L, range(1,17,2)) )
   print ( enhList3 )
   print ( enhList3.elute( (Elem[1] > 5) & (Elem[1] < 11) ) )     #new notation
   print ( enhList3 )  
   print ()
   
   #list of dictionaries 
   enhList4 = EnhList( map(D, range(1,17,2)) )
   print ( enhList4 )
   print ( enhList4.elute( (Elem['a'] > 5) & (Elem['a'] < 11) ) ) #new notation
   print ( enhList4 )
   print ()

Thanks and Cheers, Dominik

1 Like

You could look at placeholder · PyPI for a package that lets you write _.r > 5 to mean the same as lambda x: x.r > 5.

I tend to like returning new stuff from functions rather than modifying stuff in place, so I might write something like this:

def partition(iterable, predicate):
    mapping = [[], []]
    for x in iterable:
        mapping[predicate(x)].append(x)
    return mapping

false_stuff, true_stuff = partition("Pyth0n", str.isdigit)
print(false_stuff) # ['P', 'y', 't', 'h', 'n']
print(true_stuff) # ['0']

You could easily then do this to operate in place:

arr[:], selected = partition(arr, predicate)
2 Likes

Hi Dennis,

thanks for your ideas! I did not know the placeholder library - nice :+1:

I indeed first started using a function as the return value of the overloaded operator methods too, but I did not came up with an easy solution to implement more complex, assembled condition terms alike:

((_ .r > 5) and (_ .r < 10))

which works fine with my “str-approach”:

((Elem.r > 5) & (Elem.r < 10))

Do you see a simple way to implement something alike that using the placeholder library? Overloading __ and __ and __ or __ seem not to work for function objects?

Cheers, Dominik

I did not use said ‘placeholder’ library in the end, but, when I had a look into it’s source code, I stumbled over ‘functools.partial’; using that seems to have done the trick… :stuck_out_tongue:

So, thanks again!

Cheers, Dominik

Howdy Folks,

In the meanwhile said enhanced list already is working fine - together with said operator notation - in Python 2.7.18. Yet I am trying to make it also Python 3.* compatible. When I run the test functions in Python 3.3.0, I get the following error:

fct = Element < 5
TypeError: unorderable types: type() < int()

whereby ‘Element’ is a class (inherited from ‘partial’) with a __ metaclass __ = ElementMeta and ElementMeta has an overloaded __ lt __ class method returning a ‘partial’. In Python 2.7 that works fine.

There it leads to

fct(4) ==> True
fct(6) ==> False

What is the problem in Python 3.*? How can I solve it?

Thanks and Cheers, Dominik

Remember that Element < 5 tries calling type(Element).__lt__(Element, 5). So you need to make Element an instance of some class, rather than a class itself.

If that doesn’t work and Element really needs to be a class for some reason, you could use a metaclass to change type(Element) to be something else.

1 Like

Hi Dominik,

Metaclass issues can be very complicated to solve. You will probably
have more luck in getting an answer if you show us a minimal
reproducible example that works correctly in Python 2 but not in 3.

http://www.sscce.org/

This works for me in version 3:

>>> class Meta(type):
...     def __lt__(cls, other):
...             print(cls, other)
...             return True
... 
>>> class A(metaclass=Meta):
...     pass
... 
>>> A < 45
<class '__main__.A'> 45
True

but of course I have no idea what your code is doing.

1 Like

Hi Dennis,

thanks a lot - your hint

got me back on track; after reading that, I asked myself, why I have chosen a class instead of an instance and came to the conclusion, that the belonging reason does not apply any longer.

Sometimes you cannot see the forest for the trees…

I changed that and now it seems to work under 2.7.* as well as under 3.*. :+1:

Cheers, Dominik

Hi Steven,

interesting - thanks for the example!

A minimal example of what I did, could be:

from functools import partial
from operator  import lt

class Meta(type):
      @classmethod
      def __lt__(cls, ohs):
          return lambda x: lt(x, ohs)

class Element(partial):
      __metaclass__ = Meta

fct = Element < 45
print( fct(44) )
print( fct(46) )

which works fine under Python 2.7.18 and leads to

TypeError: '<' not supported between instances of 'type' and 'int'

under e.g. Python 3.6.13.

But if the following is used instead:

class Element(partial, metaclass=Meta):
      pass

it works fine under 3.6.13 (but not under 2.7.18 anymore).
Any idea why the “__ metaclass __” version leads to said error JUST under 3.*?

Cheers, Dominik

Howdy Folks,

the library now is available at:

enhaaancedLists · PyPI

Some examples for ADDITIONAL capabilities (standard list operations work too)
available when using said library are:

#convert a parameter list to an enhanced list 
eL = EnhList(1,3,5,7)                                       #eL: [1,3,5,7]

#push single as well as multiple elements into the list
eL.push(9)                                  ==> None        #eL: [1,3,5,7,9]
eL.push(11,13,15)                           ==> None        #eL: [1,3,5,7,9,11,13,15]

#pop single as well as multiple elements from the list - 
#note that push/pop implements a FIFO - in contrast to the standard list (LIFO)
eL.pop()                                    ==> 1           #eL: [3,5,7,9,11,13,15]
eL.pop( (elem > 3) & (elem < 11), single )  ==> 5           #eL: [3,7,9,11,13,15]
eL.pop( (elem > 3) & (elem < 11)         )  ==> [7,9]       #eL: [3,11,13,15]      

#get items from list
eL[ elem >= 10         ]                    ==> [11,13,15]  #eL: unchanged
eL[ elem >= 10, single ]                    ==> 11          #eL: unchanged
eL[ elem <  3,  single ]                    ==> None        #eL: unchanged

#check whether list contains items
( elem <  3 ) in eL                         ==> False       #eL: unchanged
( elem >= 3 ) in eL                         ==> True        #eL: unchanged

#delete items from list
del eL[ elem < 12, single ]                 ==> ---         #eL: [11,13,15]
del eL[ elem > 12         ]                 ==> ---         #eL: [11]

eL = EnhList(1,3,5,7)                                       #eL: [1,3,5,7]
#check whether all element meet a condition
eL.areAll( elem % 2 == 1 )                  ==> True        #eL: unchanged
eL.areAll( elem     >= 3 )                  ==> False       #eL: unchanged

#map function on elements / work with items of elements
eL.mapIf( lambda x: dict(a=x) )                          
                    ==> None        #eL: [{'a':1},{'a':3},{'a':5},{'a':7}]
eL.mapIf( lambda x: x['a'] + 1, elem['a'] > 3)           
                    ==> None        #eL: [{'a':1},{'a':3},6,8]

#work with attributes of elements
class Attr(object):
    def __init__(self, value):
        self.a = value
    def __repr__(self):
        return ".a=%s" % self.a
eL.mapIf( lambda x: Attr(x), lambda x: type(x) ==  int ) 
                    ==> None        #eL: [{'a':1},{'a':3},.a=6,.a=8]

Have Fun and Cheers,
Dominik

Hi Dominik,

“Any idea why the “__ metaclass __” version leads to said error JUST
under 3.*?”

__metaclass__ has no special meaning in Python 3. The 2to3 fixer
changes it to the Python 3 version.

https://docs.python.org/3/library/2to3.html?highlight=metaclass#2to3fixer-metaclass

https://docs.python.org/3/reference/datamodel.html?highlight=metaclass#metaclasses

1 Like

Hi Steven,

I see - thanks!

Cheers, Dominik

Hello World,

version 0.75 is available:

A ‘SecList’ class has been added. It is a secured version of the aforementioned enhanced list class ‘EnhList’.

Access to its elements has been made ‘thread-safe’ by wrapping the belonging methods in a ‘with’ context automatically ‘acquiring’ / ‘releasing’ an internal ‘SemiBlockingMutex’ (a special multithreading / multiprocessing lock).

Example:

   #convert a parameter list into a secured list 
   sL = SecList(1,3,5,7,9,11,13)                                #sL: [1,3,5,7,9,11,13]
   
   #if then a first thread e.g. would run the following statement:
   poppedLtL = sL.pop( elem < 9 )
   
   #and a second thread in parallel (!!) e.g. would run the following statement:
   poppedGtL = sL.pop( elem > 7 )
   
   #there would be no error and the result would be:
   #poppedLtL <==> [1,3,5,7]
   #poppedGtL <==> [9,11,13]
   #sL        <==> [] 

Have fun, cheers
Dominik