Installing j2 on Python 3.6

Hello there,

I am maintaining a docker image that is based on Centos7 and has both Python2.7 and Python3.6 available in it. The image has a step where we install a package named ‘j2’ using pip. However, this now fails when building due to some version mismatches. I could see that j2 is dependent upon Jinja2 which in turn is dependent upon MarkupSafe > 2.0

I could see that the latest MarkupSafe versions 2.1.1 and 2.1.0 both require Python 3.7+ for it to work. Thus I planned on to install MarkupSafe 2.0.1 which works on Python 3.6 as well first so that installing j2 does not try to install the latest MarkupSafe on its own.

However installing MarkupSafe 2.0.1 using pip (from python 2.7) installs it as MarkupSafe 0.0.0.

Installing it using pip3 installs it correctly. However as j2 is a Python 2 only package, I am unable to use pip3 to install j2. Thus pip2 can not see MarkupSafe already installed, it again tries to install the latest MarkupSafe and fails.

Is there a way to fix this issue ?

If j2 requires Python 2.7 your options are either to use Python 2.7 or port j2 to work with Python 3.

The last markupsafe version to support Python 2 is 1.1.1.

What version of pip are you using with Python 2? With 20.3.4 installed for me pip correctly installs markupsafe 1.1.1 rather than the latest version.

Python 2 and Python 3 will use different paths to install packages to, so you can only use pip3 to install packages for Python 3 and vise versa.

Thanks for the reply. You are very much correct. I can install MarkupSafe 1.1.1 with pip2. But problem here is that installation of j2 by default looks for MarkupSafe > 2.0. Since the latest is 2.1.1, it will tried to be installed and fail.

j2 was last updated in 2011. The “project homepage” link on PyPI points to a source repository that no longer exists. I think you’d be better finding a replacement for j2. At this point it’s unlikely that anyone will be able to help you get it working.

I’m assuming you mean this package. Taking a quick look, it hasn’t been maintained for well over decade, and the linked Bitbucket source repo is gone (I’m guessing it was a hg repo deleted as part of Bitbucket dropping Mercurial support years ago).

The packaging uses (unsurprisingly, given its age) a bunch of deprecated stuff and crazy hacks that probably won’t work on any modern-ish Python, and certainly won’t soon, and the setup.py doesn’t contain a python_requires argument (unsurprisingly, given that metadata didn’t even exist yet at the time it was packaged) but does contain a manual install-time check for Python 2.6 or later, presumably including 2.7.

The latest version of Jinja, yes (Jinja 3), but if j2 either upper-capped its Jinja version, or used the python_requires argument to make clear it required Python 2, pip should automatically pull a compatible version.

You could manually install older compatible versions of dependencies on Python 3.6, but that requires that j2 work on Python 3.6, which is not impossible (you may actually be very lucky in this particular case), but doubtful for a >decade old package. More importantly, the packaging weirdness is likely to cause issues, and you’re very likely to run into incompatibilities with a bunch of old stuff dropped in the latest Jinja 3.x releases. What you really need to constrain is the Jinja version to at least <3, along with the updates discussed above if you want it to work with Python 3 and semi-modern packaging (which you should).

Wait, what? You just mentioned MarkupSafe 2.0.1 requires Python 3.6, so why would you expect it to work installing on Python 2.7? Also, your Python 2 pip version is so old that it apparently doesn’t even realize that it installed an incompatible package.

To note, you should be using a venv/virtualenv environment so you don’t mess up your system Python installation, and possibly your OS itself, or run into other incompatibilities there, as well as upgrade (python -m pip install --upgrade pip) to a much less ancient pip version than whatever is bundled with your legacy OS.

Porting it shouldn’t actually be that hard; see below.

Every Python install on your machine is separate, so regardless of version, different Python installations (should) never see each others’ packages.

Most of the code is mostly just a CLI wrapper over Jinja with some helper functionality and extra variables injected, and isn’t that complex. Skimming through it, it looked fairly modern for its time, and I didn’t see that much that wouldn’t work unchanged on even the latest version of Python, aside from a few relatively minor and fixable potential trouble spots.

Packaging-wise it needs a little more work, but that shouldn’t be too tough either. While I suggest following the tutorial of the Packaging User Guide, the main must-do items are:

  • Rename j2j2.py to make it a proper Python module
  • Stick the argument parsing and execution code blocks into a main() function
  • In the setup.py, remove all the hacky code besides the actual setup() call inside the main() function, get rid of all the imports and replace them with from setuptools import setup, and in the setup_args dict, replace the scripts line with entry_points = {'console_scripts': ['j2=j2:main']}.
  • Also in the setup_args dict, get rid of platform=..., and 'argparse' in the install_requires (its in the stdlib now), constrain your version of Jinja appropriately (e.g. "<3") and add a line e.g. python_requires = ... with your minimum required Python version, e.g. “>=3.6”` is a reasonable default for your use case.

There are other things you should do, see the packaging guide for that, but those are the minimum to get it working with semi-modern packaging tooling.

The biggest challenge is likely getting to work with recent versions of Jinja (especially Jinja >=3, which broke backward compat), but you should be able to get away with restricting it to Jinja <3 for now, or perhaps an earlier version.

This is because, as discussed above, you need to constrain the version of jinja2 (which is the direct dependency of the package) to at least <3, and possibly lower, which is what requires MarkupSafe >=2.

So if you install jinja2==2.* first or at the same time, (modern) pip should at least solve correctly, though you might still run into one of other problem I’ve identified aren’t also an issue), and you might need to try an older Jinja if you run into problems at runtime. That said, this won’t work forever, since later versions of your OS might not include, or even be able to install Python 2, since it is EoL and obselete.