ModuleNotFoundError: No module named '_cffi_backend'

Hi,

when installing pyload-ng with pip (Synology DSM 7.1.1-42962 Update 5, Entware) I get the python error listed below. What could be the reason for this? The package cffi is installed, I have already tried to reinstall it - but that didn’t help.

...

Collecting zstandard (from Flask-Compress~=1.8->pyload-ng)
  Downloading zstandard-0.23.0.tar.gz (681 kB)
     ââââââââââââââââââââââââââââââââââââââââ 681.7/681.7 kB 1.9 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  Ã Getting requirements to build wheel did not run successfully.
  â exit code: 1
  â°â> [27 lines of output]
      <string>:41: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
      <string>:42: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
      Traceback (most recent call last):
        File "/opt/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/opt/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/opt/tmp/pip-build-env-rsu0d33l/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/tmp/pip-build-env-rsu0d33l/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "/opt/tmp/pip-build-env-rsu0d33l/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 507, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "/opt/tmp/pip-build-env-rsu0d33l/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 112, in <module>
        File "/opt/tmp/pip-install-xiuhz0xh/zstandard_1186808ca4a54cee8cc4277277fa137f/make_cffi.py", line 161, in <module>
          ffi = cffi.FFI()
                ^^^^^^^^^^
        File "/opt/tmp/pip-build-env-rsu0d33l/overlay/lib/python3.11/site-packages/cffi/api.py", line 48, in __init__
          import _cffi_backend as backend
      ModuleNotFoundError: No module named '_cffi_backend'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

à Getting requirements to build wheel did not run successfully.
â exit code: 1
â°â> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

To be on the safe side, I have now completely reinstalled all modules and it looks like the required module is there. Why is it not recognized?

root@Synology:/opt/lib/python3.11/site-packages# ll

-rwxr-xr-x 1 root root 649120 Aug 3 16:33 _cffi_backend.cpython-311.so

FYI its a backtick for the pre-formatted text quoting.
Like this:

```
Here is pre-formated text
```

Try adding the -v option to python command to see where it looks to load modules from. Here is what I see when I import cffi on Fedora ARM.

python3 -v
...
Python 3.12.4 (main, Jun  7 2024, 00:00:00) [GCC 14.1.1 20240607 (Red Hat 14.1.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cffi
# /usr/lib64/python3.12/site-packages/cffi/__pycache__/__init__.cpython-312.pyc matches /usr/lib64/python3.12/site-packages/cffi/__init__.py
# code object from '/usr/lib64/python3.12/site-packages/cffi/__pycache__/__init__.cpython-312.pyc'
# /usr/lib64/python3.12/site-packages/cffi/__pycache__/api.cpython-312.pyc matches /usr/lib64/python3.12/site-packages/cffi/api.py
# code object from '/usr/lib64/python3.12/site-packages/cffi/__pycache__/api.cpython-312.pyc'
# /usr/lib64/python3.12/site-packages/cffi/__pycache__/lock.cpython-312.pyc matches /usr/lib64/python3.12/site-packages/cffi/lock.py
# code object from '/usr/lib64/python3.12/site-packages/cffi/__pycache__/lock.cpython-312.pyc'
import 'cffi.lock' # <_frozen_importlib_external.SourceFileLoader object at 0xffff776514c0>
# /usr/lib64/python3.12/site-packages/cffi/__pycache__/error.cpython-312.pyc matches /usr/lib64/python3.12/site-packages/cffi/error.py
# code object from '/usr/lib64/python3.12/site-packages/cffi/__pycache__/error.cpython-312.pyc'
import 'cffi.error' # <_frozen_importlib_external.SourceFileLoader object at 0xffff77651610>
# /usr/lib64/python3.12/site-packages/cffi/__pycache__/model.cpython-312.pyc matches /usr/lib64/python3.12/site-packages/cffi/model.py
# code object from '/usr/lib64/python3.12/site-packages/cffi/__pycache__/model.cpython-312.pyc'
import 'cffi.model' # <_frozen_importlib_external.SourceFileLoader object at 0xffff77651880>
import 'cffi.api' # <_frozen_importlib_external.SourceFileLoader object at 0xffff7762f140>
import 'cffi' # <_frozen_importlib_external.SourceFileLoader object at 0xffff7762eb40>
>>>

Here is the output. How can I interpret this, is something missing here? In any case, I can’t see any difference - except for the different paths…

>>> import cffi
# /opt/lib/python3.11/site-packages/cffi/__pycache__/__init__.cpython-311.pyc matches /opt/lib/python3.11/site-packages/cffi/__init__.py
# code object from '/opt/lib/python3.11/site-packages/cffi/__pycache__/__init__.cpython-311.pyc'
# /opt/lib/python3.11/site-packages/cffi/__pycache__/api.cpython-311.pyc matches /opt/lib/python3.11/site-packages/cffi/api.py
# code object from '/opt/lib/python3.11/site-packages/cffi/__pycache__/api.cpython-311.pyc'
# /opt/lib/python3.11/site-packages/cffi/__pycache__/lock.cpython-311.pyc matches /opt/lib/python3.11/site-packages/cffi/lock.py
# code object from '/opt/lib/python3.11/site-packages/cffi/__pycache__/lock.cpython-311.pyc'
import 'cffi.lock' # <_frozen_importlib_external.SourceFileLoader object at 0x762e3090>
# /opt/lib/python3.11/site-packages/cffi/__pycache__/error.cpython-311.pyc matches /opt/lib/python3.11/site-packages/cffi/error.py
# code object from '/opt/lib/python3.11/site-packages/cffi/__pycache__/error.cpython-311.pyc'
import 'cffi.error' # <_frozen_importlib_external.SourceFileLoader object at 0x762e30f0>
# /opt/lib/python3.11/site-packages/cffi/__pycache__/model.cpython-311.pyc matches /opt/lib/python3.11/site-packages/cffi/model.py
# code object from '/opt/lib/python3.11/site-packages/cffi/__pycache__/model.cpython-311.pyc'
# code object from '/opt/lib/python3.11/weakref.pyc'
# code object from '/opt/lib/python3.11/_weakrefset.pyc'
import '_weakrefset' # <_frozen_importlib_external.SourcelessFileLoader object at 0x762fae10>
import 'weakref' # <_frozen_importlib_external.SourcelessFileLoader object at 0x762fa330>
import 'cffi.model' # <_frozen_importlib_external.SourceFileLoader object at 0x762e3290>
import 'cffi.api' # <_frozen_importlib_external.SourceFileLoader object at 0x76350b30>
import 'cffi' # <_frozen_importlib_external.SourceFileLoader object at 0x76350850>

I did not try a program that would bring in _cffi_backend just imported it directly and that works for me. Here is what I see that shows where _cffi_backend is stored on the disk:

:>>> import cffi
# /usr/lib64/python3.12/site-packages/cffi/__pycache__/__init__.cpython-312.pyc matches /usr/lib64/python3.12/site-packages/cffi/__init__.py
# code object from '/usr/lib64/python3.12/site-packages/cffi/__pycache__/__init__.cpython-312.pyc'
# /usr/lib64/python3.12/site-packages/cffi/__pycache__/api.cpython-312.pyc matches /usr/lib64/python3.12/site-packages/cffi/api.py
# code object from '/usr/lib64/python3.12/site-packages/cffi/__pycache__/api.cpython-312.pyc'
# /usr/lib64/python3.12/site-packages/cffi/__pycache__/lock.cpython-312.pyc matches /usr/lib64/python3.12/site-packages/cffi/lock.py
# code object from '/usr/lib64/python3.12/site-packages/cffi/__pycache__/lock.cpython-312.pyc'
import 'cffi.lock' # <_frozen_importlib_external.SourceFileLoader object at 0xffff877fcd10>
# /usr/lib64/python3.12/site-packages/cffi/__pycache__/error.cpython-312.pyc matches /usr/lib64/python3.12/site-packages/cffi/error.py
# code object from '/usr/lib64/python3.12/site-packages/cffi/__pycache__/error.cpython-312.pyc'
import 'cffi.error' # <_frozen_importlib_external.SourceFileLoader object at 0xffff877fce60>
# /usr/lib64/python3.12/site-packages/cffi/__pycache__/model.cpython-312.pyc matches /usr/lib64/python3.12/site-packages/cffi/model.py
# code object from '/usr/lib64/python3.12/site-packages/cffi/__pycache__/model.cpython-312.pyc'
import 'cffi.model' # <_frozen_importlib_external.SourceFileLoader object at 0xffff877fd0d0>
import 'cffi.api' # <_frozen_importlib_external.SourceFileLoader object at 0xffff877bb1d0>
import 'cffi' # <_frozen_importlib_external.SourceFileLoader object at 0xffff877bafc0>
:>>> import _cffi_backend
# extension module '_cffi_backend' loaded from '/usr/lib64/python3.12/site-packages/_cffi_backend.cpython-312-aarch64-linux-gnu.so'
# extension module '_cffi_backend' executed from '/usr/lib64/python3.12/site-packages/_cffi_backend.cpython-312-aarch64-linux-gnu.so'
import '_cffi_backend' # <_frozen_importlib_external.ExtensionFileLoader object at 0xffff877bae10>
:>>>

Apparently it also works directly for me, but not with pip:

>>> import _cffi_backend
# extension module '_cffi_backend' loaded from '/opt/lib/python3.11/site-packages/_cffi_backend.cpython-311-arm-linux-gnueabi.so'
# extension module '_cffi_backend' executed from '/opt/lib/python3.11/site-packages/_cffi_backend.cpython-311-arm-linux-gnueabi.so'
import '_cffi_backend' # <_frozen_importlib_external.ExtensionFileLoader object at 0x763d6850>

I have also tried “pip -vvv”, which gives me additional error messages, but I can’t really figure it out either:

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /opt/bin/python3 /opt/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py get_requires_for_build_wheel /opt/tmp/tmp5iz37end
  cwd: /opt/tmp/pip-install-1q44ozf1/zstandard_97ea1c0346b8400f8e037aefc13b23e7
  Getting requirements to build wheel ... error
Remote version of pip: 24.2
Local version of pip:  24.2
Was pip installed by pip? True
error: subprocess-exited-with-error

à Getting requirements to build wheel did not run successfully.
â exit code: 1
â°â> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Exception information:
Traceback (most recent call last):
  File "/opt/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 105, in _run_wrapper
    status = _inner_run()
             ^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 96, in _inner_run
    return self.run(options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 379, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 230, in _get_updated_criteria
    self._add_to_criteria(criteria, requirement, parent=candidate)
  File "/opt/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/opt/lib/python3.11/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
    return bool(self._sequence)
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 174, in __bool__
    return any(self)
           ^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 162, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 53, in _iter_built
    candidate = func()
                ^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 186, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 232, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 303, in __init__
    super().__init__(
  File "/opt/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 235, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 314, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 527, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 642, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 72, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/opt/lib/python3.11/site-packages/pip/_internal/distributions/sdist.py", line 56, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "/opt/lib/python3.11/site-packages/pip/_internal/distributions/sdist.py", line 126, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/distributions/sdist.py", line 103, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_internal/utils/misc.py", line 706, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 311, in _call_hook
    self._subprocess_runner(
  File "/opt/lib/python3.11/site-packages/pip/_internal/utils/subprocess.py", line 237, in runner
    call_subprocess(
  File "/opt/lib/python3.11/site-packages/pip/_internal/utils/subprocess.py", line 209, in call_subprocess
    raise error
pip._internal.exceptions.InstallationSubprocessError: Getting requirements to build wheel exited with 1
Removed zstandard from https://files.pythonhosted.org/packages/ed/f6/2ac0287b442160a89d726b17a9184a4c615bb5237db763791a7fd16d9df1/zstandard-0.23.0.tar.gz (from Flask-Compress~=1.8->pyload-ng) from build tracker '/opt/tmp/pip-build-tracker-vnoscfg4'
Removed build tracker: '/opt/tmp/pip-build-tracker-vnoscfg4'

How do you run pip?
Try this way python3 -m pip <command> where python3 is the python you used above to test the imports.

I have also tried this - but there is no difference, the same error occurs.

Its as if pip is not using the same python.

At this point I would be editing the installed pip code to add debug output to get insight into what the sys.executable and sys.path is leading up to the failure.
Are you happy to do that?

Sure, why not - if you tell me how to do it :grinning:

However, I wouldn’t really know which “other” python it could be. There is actually another python from DSM, for safety’s sake I have “hidden” it, but that didn’t help either, the error is still there.

root@Synology:/bin# ll python*
lrwxrwxrwx    1 root     root             7 Oct 20  2022 python.orig -> python3
-rwxr-xr-x    1 root     root          5544 Nov 15  2021 python3.8.orig
lrwxrwxrwx    1 root     root             9 Oct 20  2022 python3.orig -> python3.8

I finally deleted all python3 packages and reinstalled everything from the beginning - and now everything works, the error is gone.

@Barry Scott: Thank you very much!