Python 3.8 compatibility problem with Protos 5.27.0

I’m running a Python 3.8 app that is using some proto generated from a private repo. This integration was working until the generated code used the 5.27.0 version.

The error I’m getting is something like this:

Traceback (most recent call last):
  File "src/manage.py", line 54, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/runner/work/MY_PROJECT/MY_PROJECT/src/MY_PROJECT/models/__init__.py", line 8, in <module>
    from .user_profile import UserProfile
  File "/home/runner/work/MY_PROJECT/MY_PROJECT/src/MY_PROJECT/models/user_profile.py", line 11, in <module>
    from MY_PROJECT.lib.SOME_FILE import AdapultApiWrapper
  File "/home/runner/work/MY_PROJECT/MY_PROJECT/src/MY_PROJECT/lib/SOME_FILE.py", line 8, in <module>
    from MY_PROTOS.common.event.v1 import job_pb2
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/MY_PROTOS/common/event/v1/job_pb2.py", line 9, in <module>
    from google.protobuf import runtime_version as _runtime_version
ImportError: cannot import name 'runtime_version' from 'google.protobuf' (/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/google/protobuf/__init__.py)
Error: Process completed with exit code 1.

I already checked the files in the pod and all the generated code from the protos is there.

It seems the problem can be related to the “_” before “runtime_version”. I’m not sure how to fix this.

Here are the versions I’m using:

  • Python 3.8
  • pip 24.0
  • Buf 1.28.1 (to generate the Python code from .proto)
  • Protobuf 4.25.1 but I also tried 5.27.0
  • grpcio 1.60.0
  • grpcio-tools 1.60.0

Here is the buf.gen.yaml config I use to generate the final code:

version: v1
managed:
  enabled: true
  go_package_prefix:
    default: github.com/PRIVATE_REPO
    except:
      - buf.build/googleapis/googleapis
      - buf.build/grpc-ecosystem/grpc-gateway
plugins:
  - plugin: buf.build/protocolbuffers/go:v1.30.0
    out: gen/go
    opt: paths=source_relative
  - plugin: buf.build/grpc/go:v1.3.0
    out: gen/go
    opt: paths=source_relative,require_unimplemented_servers=false
  - plugin: buf.build/grpc/python:v1.56.1
    out: gen/python
  - plugin: buf.build/protocolbuffers/python
    out: gen/python
  - plugin: buf.build/grpc-ecosystem/gateway:v2.15.2
    out: gen/go
    opt: paths=source_relative
  - plugin: buf.build/grpc-ecosystem/openapiv2:v2.15.2
    out: third_party/OpenAPI

And finally, this is the pyproject.toml file used to generate the requirements.txt file using poetry:

[tool.poetry]
name = "project"
version = "0.1.14"
description = "..."
authors = ["..."]

[tool.poetry.dependencies]
python = "^3.8"
bleach = "^4.1.0"
coverage = "^5.5.0"
django = "^2.2"
django-categories = "^1.8.0"
django-cors-headers = "^3.8.0"
django-filter = "^2.4.0"
django-fsm = "^2.7.0"
django-fsm-log = "^2.0.0"
django-imagekit = "^4.0.2"
django-log-request-id = "^1.6.0"
django-oauth-toolkit = "^1.5.0"
django-permission = "^1.0.4"         # consider removing, not longer mantained.
django-rest-swagger = "^2.1.2"       # consider removing, unmantained
django-simple-history = "^3.0.0"
django-storages = "^1.11.1"
django-taggit = "^1.5.1"
django-taggit-serializer = "^0.1.7"  # consider removing, last update 4 years ago
djangorestframework = "^3.12.4"
google-api-python-client = "^2.20.0"
google-auth = "^2.0.2"
google-cloud-storage = "^1.42.1"
oauth2client = "^4.1.3"
# Limited by django 2.2.x: https://github.com/psycopg/psycopg2/issues/1293                                                                                     # consider migrating, deprecated
psycopg2 = "2.8.6"
pycryptodome = "^3.10.1"
pytest-django = "^4.4.0"
python-gmaps = "^0.3.1"
pyyaml = "^5.4.1"                                                                                                                      # consider migrating, last update 2016                                                                                         # consider removing or moving to official library
smtpapi = "^0.3.1"                                                                                                                     # consider removing or updating
stripe = "^2.60.0"
unittest-xml-reporting = "^3.0.4"
grpcio = "~1.51.3"
grpcio-tools = "~1.51.3"
filestack-python = "^3.4.0"
django-session-header = { git = "https://github.com/kball/django-session-header.git", branch = "master" }                              # consider removing
django-access-tokens = { git = "https://github.com/sixfeetup/django-access-tokens.git", branch = "master" }                            # consider removing
pillow = "^8.3.2"
google-cloud-error-reporting = "^1.2.2"
django-elasticsearch-dsl-drf = "^0.22.2"
django-elasticsearch-dsl = "^7.2.0"
django-safedelete = "^1.0.0"
django-prometheus = "^2.1.0"
private-apis-python = { git = "https://github.com/PRIVATE_REPO", branch = "alternative-package-name", subdirectory = "gen/python" } # names changed for privacy
responses = "^0.14.0"
kafka-python = "^2.0.2"
sonyflake-py = "^1.3.0"

## Not supported yet: 
# grpcio-testing = "^1.46.3"
gunicorn = "^20.1.0"

[tool.poetry.dev-dependencies]
flake8 = "^3.9.2"
black = { version = "^21.8b0", allow-prereleases = true }
rope = "^0.19.0"

[build-system]
requires = ["poetry>=1.0.0"]
build-backend = "poetry.masonry.api"

Here are the final version in the requirements.txt file:

grpcio-status==1.51.3 ; python_version >= "3.8" and python_version < "4.0"
grpcio-tools==1.51.3 ; python_version >= "3.8" and python_version < "4.0"
grpcio==1.51.3 ; python_version >= "3.8" and python_version < "4.0"
googleapis-common-protos==1.60.0 ; python_version >= "3.8" and python_version < "4.0"
googleapis-common-protos[grpc]==1.60.0 ; python_version >= "3.8" and python_version < "4.0"
proto-plus==1.22.3 ; python_version >= "3.8" and python_version < "4.0"
protobuf==4.24.0 ; python_version >= "3.8" and python_version < "4.0"

The app was working in the 4.26.1 version