How to add methods to immutable types such as str or int or function

Python Help
1

tl;dr What if Python had a mutable clone object which changes how the object works without changing its class level attributes.

Hello People!

(Backstory) Recently I have been working on creating a custom compiler for some custom markup / logic language I am working on. To do so in native Python, it gets a little tricky, so I decided to create a small library / package to be able to use it in several repositories easily (my project is distributed across 3-4 Python environments).

(Context) One of the major things my package is doing, is adding additional utilities classes, functions and sometimes on objects. These task of adding additional utilities happens during declaration and may / may not affect the actual usage of the things.

(Problem) As much as I am working on making the package great on itā€™s own, working with native immutable classes and objects make it lot harder. I canā€™t change the behaviour without creating a wrapper, but then I need to manually figure out all the properties and it gets hard and uncertain with all possible attributes.

The best I am able to do is working very cautiously and injecting the objects on mutable wrappers on original immutable entity. Although I am able to achieve what how I want it to behave (kind of), in a lot of cases I want to be able to create a mutation in the real object itself, like changing how a function primarily behave - like adding arg type validation, attached pre-call and post-call functions, and possibly more utilities.

I would suggest that having mutable types increase convenience, it would be a nice to have, but at the same time I understand that this makes code inconsistent and unsafe at times. For example, someone will be able to infect code, possibly with malicious content. This is a major plus for having immutable entities (at least primitives like int, str, function, etc).

Here is what Iā€™m proposing, we could possibly have a function inside object / external function that can convert the entity to a safe mutable copy. Accessing, updating or working with this copy and handling it is now very convenient, while the trade off being the security and proper usage is now expected in the hands of developer using the mutable copy. The original immutable entity will remain safe and preserve its natural behavour as it was intended to be.

Code-wise, something like this should give clarity:

hello_world: str = "Hello, World!"

hello_world.to_mutable_copy() # changes the variable to be a mutable copy now
hello_world.__call__ = lambda self: print(self) # gives error in normal case, but not here because it will now become a mutable copy.

hello_world() # prints "Hello, World!"

The example given is super simplified and I donā€™t encourage others to use the feature like this. At the same time I canā€™t really define how the Python developer community will use it, but I think having such thing might help developers working on projects and libraries.

For types wise, I think something like the following code will be feasible.

hello: str = "hello" # original type: str
hello_mutable: mutable<str> = "hello".to_mutable_copy() # internally converts to a type "mutable", with args (generics) of (str,).

My library primarily does work-arounds for function cases right now using decorators. It took me 3-4 iterations and will possibly take more to easily handle this task. I handle it by creating a wrapper function before using the actual injection to modify attributes and calling properties. The bounded functions are also handled in a similar fashion during class instantiation, where instead of directly using the function, bounded function is being handled.

Thatā€™s all what I wanted to say for now. Would love to know what the community thinks of this, how people would intend to use it if implemented, if people actually want this to be implemented or not, how people are working around this in their projects and packages, should this be converted to a PEP, and other aspects and views about this topic.

EDIT 1: I mentioned str for simplicity, my actual use case is for function types but thought it would be interesting and helpful to know for more generic.

EDIT 2: I started this thread to discuss the idea about having ā€œmutable clonesā€ of objects that change class level behaviours. In my codebase, I am not using this and have work-arounds which I donā€™t intend to change unless I find something solid.

EDIT 3: Added tl;dr + this update.

Thanks,
AttAditya

2

Youā€™re looking for wrapt: Proxies and Wrappers ā€” wrapt 2.2.1 documentation

Itā€™s also possible to subclass str directly, and add methods. See MarkupSafe for example: markupsafe/src/markupsafe/__init__.py at b2e4d9c7687be25695fffbe93a37622302b24fb1 Ā· pallets/markupsafe Ā· GitHub

3

Yes, libraries like wrapt and the one I created simplifies this process, but Iā€™m thinking with the perspective of what if I donā€™t need to import these features (like not having to pip install them), and a dynamic language like Python can give me such features directly?

That said, right now Iā€™m using custom package with custom handling as I wish to have.

Thanks for the reference as well, I believe it will help me improve my implementations.

4

You can mutate any python object by using ctypes. You can also crash python with it. (Doing so is a ā€˜consenting-adultā€™ activity; not a reportable bug;-).

You can encode strings to bytes and copy to mutable bytearrays. You can make a utf-8 encoded bytearray not be valid utf-8 and not convertible back to a string. Any resulting UnicodeDecodingError is a user bug.

I am dubious that we will hand users another generic footgun in the stdlib.

5

Crash? Donā€™t you mean ā€œRapid Unscheduled Discontinuationā€? :slight_smile:

Thereā€™s a bit of a problem here in that calling an object will invoke __call__ on the type, not the instance, but other than that, you can get extremely close with just core Python tools:

>>> class str(str): pass
... 
>>> hello_world = str("Hello, World!")
>>> type(hello_world).__call__ = lambda self: print(self)
>>> hello_world()
Hello, World!

Notably though, you need to explicitly CHOOSE to use your new subclass. Literals will never have functionality added to or changed. But this gives you similar behaviour to other options, without needing to pip install anything.

6

Iā€™ve tried to make stuff work with python, and from make work, I mean crash it several times while getting results that donā€™t even tell where the error is. Tried other approaches like function cloning and function attribute attacking by injecting a second co_code in the __code__ values, but I got weird errors about co_args not matching. I didnā€™t knew Python checked those details before encountering that error.

Btw fun fact: When you try to build __code__, if you try accessing the docs, it says:

print((lambda: None).__code__.__class__.__doc__)

Create a code object.  Not for the faint of heart.

So fully agreeing with:

7

What if we use __setattr__ in the wrapper class (assuming we create this wrapper class)? In that case setting is directly possible, although handling __setattr__ and __getattr__ both in a single class could be difficult, given it easily raises RecursionError. But I think if it is configured the right way, it gets things done right.

I like this work-around.

8

Do keep in mind that this is making a change for every instance of your str subclass, and if you did intend that, it would be better to just put that onto the class directly:

>>> class str(str):
...     def __call__(self): print(self)
...     
>>> hello_world = str("Hello, World!")
>>> hello_world()
Hello, World!
9

I think we can create a function that uses type to create a new subclass. That should probably resolve this global change problem. Although I guess this means there will be more memory overhead for each new class, not sure about how I can encounter that.

10

Iā€™m a bit confused ā€“ this wonā€™t work, because dunder methods are looked
up in the class only, so you canā€™t override them on a per-instance basis
like that.

It sounds like itā€™s not the object itself that you want to be mutable,
but its class.

Can you share a bit of your existing library code? That might give us a
better idea of what youā€™re actually doing.

11

Yes, in Python you have to change the change the class for such tasks, but I want to do it to the object itself without affecting the classes, however the object types are primitives like str, int, or in my case function.

For the code sharing part, unfortunately the code I wrote for that was unstable so I decided not to commit it, hence itā€™s no more with me. However, I do have similar piece of code:

def clone_function(f):
  FunctionType = type(f)
  res = FunctionType(
    f.__code__,
    f.__globals__,
    f.__name__,
    f.__defaults__,
    f.__closure__
  )

  res.__dict__.update(f.__dict__.copy())
  res.__kwdefaults__ = f.__kwdefaults__
  res.__doc__ = f.__doc__
  res.__module__ = f.__module__
  res.__annotations__ = f.__annotations__

  return res

This is a cloning function I created to clone functions using my library (kept it as a separate function in case I need to change some meta for the functions in future). This function is stable and it works fine.

The unstable one was similar, except instead of using clone, I tried to create a new clone and tried to switch the f.__code__ with some other __code__, then it becomes highly unstable and often gives error due to co_args, co_varnames and other attributes. Also doing that crashed 32 existing methods out of my 85 test cases, however surprisingly worked for a few tests.

And as mentioned earlier,

12

To turn this into an actionable proposal, youā€™ll need a lot more specification. And once you start to develop this idea further, it might stop looking like a good idea.

Anyway, you can implement this yourself in a library (and put that library on PyPI), if you make it a function that returns the copy, rather than a method:

hello_world = to_mutable_copy(hello_world)

Of course youā€™d need to implement mutable versions of str, int and function as custom classes ā€“ but thatā€™s work that someone will need to do for your proposal. Itā€™s possible to do this, though I donā€™t think itā€™s practical.