Issue with python3.11 connector on EL9

Hi,

I’m having an issue which seems to relate to python and I’ve been trying to find a solution but so far I’ve been failing miserably.
First time here on the forums and I’m sorry if the issue is misplaced, I tried to narrow it down the best I can. I’m not a developer, I can write some simple code to do some testing or do some automations but my coding skills are extremely limited.

On all the systems I’ll describe, these actions were performed on EL9 minimal installs, to which was then added the Development Tools packages (dnf groupinstall 'Development Tools')

I first encountered this issue on a RockyLinux9 system, where python3.11 was installed from source.
I’m deploying a frappe-bench application which uses an external mariadb database server.
After everything was installed the application failed to connect to the database server, returning the error 1045, "Access denied for user 'root'@'10.0.0.1' (using password: YES).

To make sure everything was OK with the credentials and connection, I made a simple shell script to connect to the database server and execute a command select user().
The shell script connects successfully and returns what expected.

+-----------------+
| user()          |
+-----------------+
| root@10.0.0.1   |
+-----------------+

Since the frappe framework uses pymysql, I made a similar script in python to connect to the database and do the same command but instead used used mariadb.

This is the python script:

#!/usr/bin/env python3.11
import mariadb

# MariaDB Server details
MARIADB_USER = 'root'
MARIADB_PSWD = 'root-password'
MARIADB_SERV = 'db-server.internal.net'

def execute_query(query):
    conn = mariadb.connect(
        user=MARIADB_USER,
        password=MARIADB_PSWD,
        host=MARIADB_SERV,
        port=3306
    )
    cursor = conn.cursor()
    cursor.execute(query)
    result = cursor.fetchall()
    cursor.close()
    conn.close()
    return result

# List user info
def display_user():
    query = "SELECT user()"
    result = execute_query(query)
    print(result)

display_user()

The python script returns the same error as bench. In full:

$ python3.11 db-test.py 
Traceback (most recent call last):
  File "/opt/frappe/db-test.py", line 28, in <module>
    display_user()
  File "/opt/frappe/db-test.py", line 25, in display_user
    result = execute_query(query)
             ^^^^^^^^^^^^^^^^^^^^
  File "/opt/frappe/db-test.py", line 9, in execute_query
    conn = mariadb.connect(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/mariadb/__init__.py", line 146, in connect
    connection = connectionclass(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/mariadb/connections.py", line 85, in __init__
    super().__init__(*args, **kwargs)
mariadb.OperationalError: Access denied for user 'root'@'10.0.0.1' (using password: YES)

So I did another VM with RL9 and instead of going for python from the sources, I went for the python3.11 on the appstream repo.
The mariadb-connector-c version from repo is 3.2.6 and I was going for mariadb on a lower version, instead of 1.1.10 I was looking to install version 1.0.10or 1.0.11.
Install results in:

$ pip3.11 install mariadb==1.0.11
Defaulting to user installation because normal site-packages is not writeable
Collecting mariadb==1.0.11
  Downloading mariadb-1.0.11.zip (85 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.9/85.9 kB 1.1 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Installing collected packages: mariadb
  DEPRECATION: mariadb is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for mariadb ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for mariadb did not run successfully.
  │ exit code: 1
  ╰─> [41 lines of output]
      10.8.8
      running install
      /usr/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/mariadb
      copying mariadb/__init__.py -> build/lib.linux-x86_64-cpython-311/mariadb
      creating build/lib.linux-x86_64-cpython-311/mariadb/constants
      copying mariadb/constants/__init__.py -> build/lib.linux-x86_64-cpython-311/mariadb/constants
      copying mariadb/constants/CLIENT.py -> build/lib.linux-x86_64-cpython-311/mariadb/constants
      copying mariadb/constants/INDICATOR.py -> build/lib.linux-x86_64-cpython-311/mariadb/constants
      copying mariadb/constants/CURSOR.py -> build/lib.linux-x86_64-cpython-311/mariadb/constants
      copying mariadb/constants/FIELD_TYPE.py -> build/lib.linux-x86_64-cpython-311/mariadb/constants
      running build_ext
      building 'mariadb._mariadb' extension
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/mariadb
      gcc -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DPY_MARIADB_MAJOR_VERSION=1 -DPY_MARIADB_MINOR_VERSION=0 -DPY_MARIADB_PATCH_VERSION=11 -I/usr/local/include/mariadb -I/usr/local/include/mariadb/mysql -I./include -I/usr/include/python3.11 -c mariadb/mariadb.c -o build/temp.linux-x86_64-cpython-311/mariadb/mariadb.o -DDEFAULT_PLUGINS_SUBDIR=\"/usr/local/lib/mariadb/plugin\"
      mariadb/mariadb.c: In function ‘PyInit__mariadb’:
      mariadb/mariadb.c:155:35: error: lvalue required as left operand of assignment
        155 |     Py_TYPE(&MrdbConnection_Type) = &PyType_Type;
            |                                   ^
      mariadb/mariadb.c:168:31: error: lvalue required as left operand of assignment
        168 |     Py_TYPE(&MrdbCursor_Type) = &PyType_Type;
            |                               ^
      mariadb/mariadb.c:174:29: error: lvalue required as left operand of assignment
        174 |     Py_TYPE(&MrdbPool_Type) = &PyType_Type;
            |                             ^
      mariadb/mariadb.c:180:34: error: lvalue required as left operand of assignment
        180 |     Py_TYPE(&MrdbIndicator_Type) = &PyType_Type;
            |                                  ^
      mariadb/mariadb.c:186:38: error: lvalue required as left operand of assignment
        186 |     Py_TYPE(&Mariadb_Fieldinfo_Type) = &PyType_Type;
            |                                      ^
      mariadb/mariadb.c:192:38: error: lvalue required as left operand of assignment
        192 |     Py_TYPE(&Mariadb_DBAPIType_Type) = &PyType_Type;
            |                                      ^
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> mariadb

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

So I went for the sources and got mariadb-connector-c-3.3.10, then installed mariadb==1.1.2 and got the same error.
I repeated the process on an AlmaLinux9 machine.
I removed any special characters from the password and used only uppercase, lowercase and numbers.
the exact same credentials work on the shell script, fail on python.
I made another simpler python script to test, but fails with the same error.
script:

#!/usr/bin/env python3.11
import mariadb

try:
    conn = mariadb.connect(
        host="server.internal.domain.net",
        user="root",
        password="password"
    )
    print("Connected!")
except mariadb.Error as e:
    print(f"Error connecting to MariaDB: {e}")

result:

$ python3.11 db-test-2.py 
Error connecting to MariaDB: Access denied for user 'root'@'10.0.0.1' (using password: YES)

Does anyone please have an idea? Thanks!

This is just a wild guess…

It seems odd to me that there are single quotes around the user and host, as you would see if repr() was used on a string.
Does the call to mariadb.connect() want the user and host in bytes and not unicode?

Hi Barry thank you for your reply.
I have tried both single and double quotes. Originally the first script had double quotes, I then changed to test and the version here came with the singles. But the second script I pasted here has double quotes and that one was never changed.
In the end, the result is exactly the same.

More information.
After @barry-scott’s answer I went double-checking on the quotes question. So I checked the MariaDB documentation and checked their example.
As it uses double quotes, I made sure my script was using double quotes (it is).

In the meanwhile I also gave a look to pymysql documentation and given their example I changed the script to just execute select user() after connecting, left the single quotes as per their example.
Result:

Traceback (most recent call last):
  File "/opt/frappe/db-test-4.py", line 5, in <module>
    connection = pymysql.connect(host='dev-dbs.de.subedinfo.net',
  File "/opt/frappe/.local/lib/python3.9/site-packages/pymysql/connections.py", line 361, in __init__
    self.connect()
  File "/opt/frappe/.local/lib/python3.9/site-packages/pymysql/connections.py", line 669, in connect
    self._request_authentication()
  File "/opt/frappe/.local/lib/python3.9/site-packages/pymysql/connections.py", line 957, in _request_authentication
    auth_packet = self._read_packet()
  File "/opt/frappe/.local/lib/python3.9/site-packages/pymysql/connections.py", line 775, in _read_packet
    packet.raise_for_error()
  File "/opt/frappe/.local/lib/python3.9/site-packages/pymysql/protocol.py", line 219, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/opt/frappe/.local/lib/python3.9/site-packages/pymysql/err.py", line 150, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1045, "Access denied for user 'root'@'10.0.0.1' (using password: YES)")

They mean the exact same thing in python, that will not change anything.

At this point I would suggest you check the logs from mariadb for clues. Does the server lig the access denied?
If so you will have to check the config, especially the password.

Hi @barry-scott thanks for coming back.
So about

They mean the exact same thing in python, that will not change anything.

by default I go for double quotes and as I said my strong suit isn’t programming but when it just keeps failing I’ll try anything I can think of. Some languages do differentiate so yeah.

At this point I would suggest you check the logs from mariadb for clues. Does the server lig the access denied?

Negative. I have enabled logging on the database server, writing to a file and I tail it while running the scripts.
The MariaDB Server only shows

...
2024-07-26 16:22:13 19 [Warning] Access denied for user 'root'@'10.0.0.1' (using password: YES)
2024-07-26 16:29:08 20 [Warning] Access denied for user 'root'@'10.0.0.1' (using password: YES)
2024-07-26 16:29:20 21 [Warning] Access denied for user 'root'@'10.0.0.1' (using password: YES)
2024-07-26 16:41:08 22 [Warning] Access denied for user 'root'@'10.0.0.1' (using password: YES)

About the password, I have double, triple, hundred checked the password. exactly the same as on the shell script.
And using frappe, which asks for the password instead of being set, I’ve pasted the password, typed character each at a time, always fails. I’m clueless.

By the way the server connection is TLS encrypted but I don’t think that should make any difference?

$ mariadb -h server.internal.net -uroot -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 26
Server version: 10.6.18-MariaDB-log MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> select user();
+-----------------+
| user()          |
+-----------------+
| root@10.0.0.1   |
+-----------------+
1 row in set (0.001 sec)

MariaDB [(none)]> SHOW SESSION STATUS LIKE 'ssl_cipher';
+---------------+------------------------+
| Variable_name | Value                  |
+---------------+------------------------+
| Ssl_cipher    | TLS_AES_256_GCM_SHA384 |
+---------------+------------------------+
1 row in set (0.001 sec)

MariaDB [(none)]> ^DBye

Using mariadb client I have no issue connecting.

You are doing all the debugging I can thing of.
One thought: try settings a simple password “asimplepassword” and see if things work. Maybe there is a character in the password that is not being handled the same way in all apps.

Hi @barry-scott
Since you’ve been kind enough to stick around, I’ve made a new test

Installed another EL9 (Alma this time around) but instead of adding any mariadb client or connector, I simply installed mysql client and libraries.

$ python3.11 db-test.py 
[('root@10.0.0.1',)]

straight out of the box…

So I think this is definitely an issue with the MariaDB Connector.