I was thinking about this idea. This is actually how BytesIO is implemented.
But what if the original bytearray is changed after creating a bytes? You need to check if there are external references to the internal bytes buffer (i.e. if the references count is larger than 1) before every mutable operation. You can do this for Python interface, this will add some overhead, but it may be tolerable. But in the C API there is PyByteArray_AS_STRING(). You cannot catch mutation of the memory array returned by PyByteArray_AS_STRING(). And it is not possible to make a copy in PyByteArray_AS_STRING(), because this C API never fails, so the users never check it for errors. The only way is to deprecate and finally remove PyByteArray_AS_STRING(), but this is too serios change.