Unexpected imports with readline.py and rlcompleter.py and others files (library injection) + pip module

Python Help
1

Spoiler: I’ve emailed security@python.org and they told me that it’s not a security issue so I feel free to create a public discussion here.

Vulnerable packages: PythonV3 (latest) and PythonV2 (latest)

Example1:

  1. mkdir /tmp/test
  2. cd /tmp/test
  3. echo ‘print(“hacked”)’ > readlines.py
  4. python (version 3, no arguments)
  5. you will see result of step №3.

Example2:

  1. mkdir /tmp/test
  2. cd /tmp/test
  3. echo ‘print(“hacked”)’ > rlcompleter.py
  4. python (version 3, no arguments)
  5. you will see result of step №3.

Example3:

  1. mkdir /tmp/test
  2. cd /tmp/test
  3. echo ‘print(“hacked”)’ > runpy.py
  4. python -m pip (version 3)
  5. you will see result of step №3.

Proof-of-concepts:
0day

(i am a new user, so i can’t add more than 1 embedded image and not more than 2 links), you can check another PoC here:

https://i.ibb.co/s3qvFvS/photo-2021-10-13-17-45-16.jpg

Description:

This allows hacker to execute arbitrary python code if another users
runs python command in certain directory.

So, for example, USER1 creates runpy.py with malicious code inside /tmp/ folder. USER2 switched current directory to /tmp/ and runs “python3 -m pip install test_package_name” and this command triggers malicious code of first user.

Full list of files:

# trying /tmp/py3/runpy.py
# trying /tmp/py3/readline.cpython-39-x86_64-linux-gnu.so
# trying /tmp/py3/readline.abi3.so
# trying /tmp/py3/readline.so
# trying /tmp/py3/readline.py
# trying /tmp/py3/readline.pyc
# trying /tmp/py3/rlcompleter.cpython-39-x86_64-linux-gnu.so
# trying /tmp/py3/rlcompleter.abi3.so
# trying /tmp/py3/rlcompleter.so
# trying /tmp/py3/rlcompleter.py

Security@python.org answered (short fragment):

The issue that you are talking about is described as a feature, not a
vulnerability.
. . .
No longer adding the current directory to sys.path has been discussed
multiple times, but so far, it remains the default behavior.

But I think that it’s not a problem with sys.path. It’s problem with python core which,
for some reason, started to input its’ core files from current directory, not python installed path.

Also about “feature, not bug”: there is no information in documentation about that python interpreter will run readline.py, rlcompleter.py and other files from current directory by default. So, if “feature” is not documented and can cause security problems - its a security issue. In this case, this feature leads to library injection vulnerability.

And last one my reply from email about isolation mode:

Yes, may be it can fixed by isolation mode, but without isolated
mode it looks like an unpredictable behavior which leads to a security
issue inside python core, which for some reason starts to load these
files from current directory.

Full email history:

https://drive.google.com/file/d/1i5YkOkxwpI2edxQ3yY_69K3F72iNWr2R/view

I will follow this topic, so feel to ask more information about this “feature” :smiley:

2

It is a sys.path issue from the perspective of that’s where the Python interpreter specifies the search order for modules, so it defines where this functionality occurs or not.

As initialized upon program startup, the first item of this list, path[0] , is the directory containing the script that was used to invoke the Python interpreter. If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first.

As for those specific modules, that’s because you’re launching the REPL. Launch your own module and what gets imported will be different (not only because you will trigger different imports but it even changes between Python releases).

As the security team said, this is all by design and known. If you don’t trust the files on your file system when executing Python files then that’s the first security issue to address as Python’s security model does not encompass untrusted files on your file system.

3

System utilities should also use isolated mode -I.

5

Yes, it solves problem. But it’s a “bad” bug fix, because solves problem only if script can be run in isolated mode. To fix this problem correctly, it’s needed to fix python core code.

P.S. One same example: there is a Remote Code Execution inside someone software - hacker can call os.system() function. And developers of this software advise users to turn off os.system function, although they can fix it in 2 lines of code : )

UPD: there are more information about bug inside next comment.

6

Let’s check file /usr/lib/python3.9/site.py:

. . .
    def register_readline():
        import atexit
        try:
            import readline
            import rlcompleter
        except ImportError:
            return
. . .

So, logically, if file /usr/lib/python3.9/site.py calls import rlcompleter, it must import file from current directory → /usr/lib/python3.9/rlcompleter.py.

But! With some code bugs it starts to import file in priority from current user directory, not /usr/lib/python3.9/, which must be in priority at sys.path for /usr/lib/python3.9/site.py.

P.S. I didn’t make a full source code analysis, but this looks weird for users, who are not expect that command python3 without arguments will start to run files from current directory (and same with python -m pip)

UPDATE:

I was wrong, there is information in documentation about sys.path: priority to the directory containing the input script, but anyway its not obvious for users that python3 will run these files from local directory by default, so think it’s better to fix (or write about it at documentation, but this is a bad way).

FIX
It can be fixed with adding full path of import script or temporary editing sys.path inside site.py file.

/usr/lib/python3.9/site.py:

. . .
def register_readline():
        import atexit # it is not affected because was imported earlier
        print(__file__)
        p_dir = os.path.dirname(__file__)
        sys.path = [p_dir] + sys.path
        try:
            import readline
            import rlcompleter
        except ImportError:
            del sys.path[0]
            return
        del sys.path[0]
. . .

P.S. will later add fix for pip

7

Please understand that changing these semantics will break people who rely on it. And these semantics have been around for decades, so there’s a reason why they have not changed even when people have brought it up previously.

1 Like
8

This question isn’t about semantics and how instruction import works. It’s about an unexpected behavior of Python Core while using python in interactive mode.

Noone fixed it before because, as i think, they didn’t recognize this bug (didn’t find any bugtrack thread/github issue/forum thread). For users, functionality of including rlcompleter from current directory is undocumented and hidden so 99.9% users even does not know about it.

Can you, please, give an example when my fix of file /usr/lib/python3.9/site.py will break anyone’s code? As i tested now - it influenced only at python interactive mode so you can still use it as a library.

P.S. I’m not asking to change whole python semantics. I’m asking about some fixes only in site.py file to make Python Core better with less bugs/security issues.

9

At least every core developer knows about this, so it’s definitely known.

You’re deleting sys.path[0] which is the current directory, so if I purposefully wanted to override a module in the stdlib you just prevented me from doing that.

I appreciate the concern, but as I am not about to push for the change I will be unsubscribing from this topic.

10

And noone of core developers created any single topic about this unexpected behavior during python interactive mode (about importing rlcompleter.py & readline.py from current directory)? If there are any topics, I didn’t find.

I don’t delete current directory path, only appending path /usr/lib/python3.9/ at the start of sys.path and deleting it after importing rlcompleter & readline, so current directory will still be at sys.path[0]. Please, recheck it again.

. . .
p_dir = os.path.dirname(__file__)
sys.path = [p_dir] + sys.path
. . .
del sys.path[0]
11

RedHat confirmed vulnerabilities and gave them CVEs: CVE-2021-23151, CVE-2021-44771

image
image1080×511 67 KB

12

Sigh … I’ll talk to my co-workers tomorrow and ask them to retract the CVEs.

PS: It’s "Red Hat, not “RedHat”.

1 Like
13

Okey, no problems :slight_smile: But don’t forget to tell them that:

1. This problem exists inside python3 console module, not inside whole python interpreter. So I do not ask to change whole Python import priority list/fix this bug, I ask only to agree that this is a security issue.

A similar example of the problem: there is a function inside programming languare named “system(cmd)” which can run system commands. The function itself is not a security issue, but if another application uses it incorrectly - the application will be vulnerable.

So, similarly, there is a construction of import priority in Python3 language, and its insecure usage inside python console interactive interpreter application (at my system it named “python3” executable application file inside /usr/lib/python3.9/ directory).

2. There are no any single topics about this “feature” → “bug” inside this module, which leads to security problems. But Brett Cannon answered, that “everyone of Core Developers” knew about it and did not provide any proof of his words.

3. There is at least one possible solution to the problem which I published in this topic before, but your colleague, Brett Cannon, didn’t correctly understand it and left thread. So the question “How can these three python lines of codes ruin anyone’s code?” remained unanswered.

4. Also if its an open community → it is better to discuss this bug publicly so everyone will be able to check arguments of two sides. The situation when one side privatly askes for his friends/colleagues to close CVEs is not correct at all. In my case, i had to write them about this bug only just because my last message inside this thread remained unanswered.

14

This is not a security issue. If an attacker can create readline.py in your working directory, you have already lost in any number of much, much worse ways.

15

Is this related?

16

Simple example: someone created a malicious readline.py file inside /tmp/ directory. Another user switched directory at /tmp/ too and wanted to run python3 command to check, for example, how much is 1+1. He thought that if run python3 without arguments - nothing more but legit python3 Core code will run. But, because of this “undocumented feature” malicious code will be run. So this is pretty a standard situation when hacker can create a file inside directory where python3 command can be run.

17

I think no, because methods which are described inside article are pretty documented :slight_smile: But the feature that python3 command without arguments will, by default, run some files(readline.py, rlcompleter.py) from current directory isn’t documented at all.

18

I think your messages are no longer answered because you are sure you are right, even when the kernel developers and documentation say otherwise. No one should document such things as the source scripts of a particular tool. They are in the public and everyone has the right to access them. The developers could also make an executable which drags nothing along. But why?

All functionality is described in the public and even in the official documentation: 6. Modules — Python 3.10.1 documentation. All packages which use dependencies one way or another have this “vulnerability” because of the language structure. It’s the same as saying that language and casts to machine code are vulnerable because someone doesn’t know how to use smart pointers (С/C++/Unsafe functions in other languages). Or to say that the officially documented postgresql functionality that allows COPY FROM is a vulnerability. To the latter, by the way, there is a CVE (CVE-2019-9193 : ** DISPUTED ** In PostgreSQL 9.3 through 11.2, the "COPY TO/FROM PROGRAM" function allows superusers and users) that many experts still don’t like. So I completely and totally agree that the CVE should be retracted.

The solution you discounted is not a solution. It is a “crutch”, which obviously will not be accepted by the language developers. It would be much better to do something like parsing .env, but Python doesn’t always set environment variables. Plus I’m not sure how it would work with Python for microprocessors, etc. All of this is a discussion to nowhere and about nothing.

UPD: Also, if “Red Hat” confirms “vulnerabilities” bypassing communication directly with developers - I’m sure that’s a disgusting practice and such CVEs should not be released to the public. Right now, it looks like you don’t understand the structure of the language and everything about it. The same goes for the Red Hat organization.

19

I think your messages are no longer answered because you are sure you are right, even when the kernel developers and documentation say otherwise.

You can check last comment of Brett Cannon which shows that he incorrectly understand my code fragment and left this thread.

No one should document such things as the source scripts of a particular tool. They are in the public and everyone has the right to access them.

So, its a bad documentation of python3 interactive console interpreter, because even if user read this documentation, he will not know that python3 loads these files from current directory. It’s not normal that to get this information they will be foreced to check all python3 application code.

The solution you discounted is not a solution. It is a “crutch”, which obviously will not be accepted by the language developers.

I completely agree with this, thought that the better sollution could be written by Core Developers. Also there are more than 10 libraries inside python interactive interpreter and only 2 of them are loaded insecure - so Core Developers somehow solved this problem.

All functionality is described in the public and even in the official documentation: 6. Modules — Python 3.10.1 documentation.

As I told before - functionality itself is not a security issue. Security issue is an insecure usage of this functionality. So, we need to check documentation about python3 interactive console interpreter, not Python3 language. Python3 interactive console is a separate application with its own wiki/man page inside Python Core.

Or to say that the officially documented postgresql functionality that allows COPY FROM is a vulnerability. To the latter, by the way, there is a CVE (CVE-2019-9193 : ** DISPUTED ** In PostgreSQL 9.3 through 11.2, the “COPY TO/FROM PROGRAM” function allows superusers and users) that many experts still don’t like. So I completely and totally agree that the CVE should be retracted.

I agree - COPY TO/FROM PROGRAM is a feature, not a bug (besides well-documented). But what will you say if, for example, there is a solution which gives everyone a seperated postgres sandbox and anyone can escape it using COPY TO/FROM PROGRAM? Yes, COPY TO/FROM PROGRAM is officially a feature, but even so, in my context this is a security vulnerability inside this solution which can be connected with CVE number (note that this CVE number must be connected with this vulnerable solution, not postgresql itself). The same with Python - this is a legit (and may be a feature) functionality, but in context of python3 executable - this is an unexpected non-documented (at python3 interactive interpreter wiki/man page) security issue.

Right now, it looks like you don’t understand the structure of the language and everything about it. The same goes for the Red Hat organization.

In fact, we don’t need to know real language/structure of python3 executable file. The real one we need to know is existance of vulnerability. So, even if this feature is described at programming language wiki, in the context of python3 interactive console application - its a security issue. The same as an example with “system(cmd)” command which i wrote some comments before.

1 Like