Red Hat provide (paid, of course) support for their LTS releases. If you need support, you can get it from them. This request still feels like asking that volunteers give you something for free that you can get already if you’re willing to pay for it…
16 Likes
@pf_moore, what does Red Hat support have to do with this? It’s the same case if you’re using Alma or Rocky, which are free. The point is there are many examples where the 5 year support cycle burdens regular downstream people unnecessarily.
1 Like
Hi, Red Hat engineer here.
A Python LTS release would be against our best interests for the community. We want the community to innovate, thrive and move forward – not to get stuck with an ancient Python version. Backwards compatibility work with an old LTS Python version would hold projects back and be a waste of time. Time is the most valuable resource of upstream developers.
RHEL users are not tight to the Python version of the initial X-stream release. Our Python maintenance team adds new versions of Python to Y-stream releases of RHEL in regular intervals. For example RHEL 8.0 came with Python 3.6, RHEL 8.10 comes with 3.6, 3.8, 3.9, 3.11, and 3.12. RHEL 9.4 has 3.9, 3.11, and 3.12, and will get more Python versions throughout its lifecycle. Our users typically use virtual environments or our UBI Python containers, which come with a virtual env, too.
27 Likes
All the old Python versions are still there along with the old package versions that go with. If you’re wanting to run old Python versions for a long time then you are presumably also not upgrading hardware or OS etc so those old versions should keep working fine.
It might be that more can be done to make it smoother when using older Python versions for example if more Python packages had upper bounds to prevent untested future versions of Python or other packages.
1 Like
I used to work at Red Hat too. But neither Red Hat, nor EPEL create RPMs for most of the Python packages except the initial one that ships. That is one thing that prevents upgrading. And while some people use virtual environments, you need to relocate those package RPMS to build the virtual environments or build new ones for those environments. If you don’t do that, you can’t properly audit your system for rogue software or utilize tooling like fapolicy.
At the end of the day, I’m a package maintainer trying to do right by the people downstream from me. I’m always pushing to run the latest, but I know that’s not always possible for everyone. The lack of support in tooling makes my life harder and discourages me from continuing to contribute.
2 Likes
There is an extremely good reason for dropping support, and it has nothing to do with the feature changes in any release.
If my package states that “we support the Python versions currently supported by the CPython team”, I have established a crystal clear support policy which fits 99% of users.
I can expect that my dependencies will drop old Python versions and that I will as well. It’s all relatively organized and rarely surprises maintainers, simply by following the support strategy of Python itself.
7 Likes
Red Hat provides bugfix and security fix for Python in EPEL.
I don’t understand that Python LTS out of EPEL helps Python in EPEL users.
As someone who’s had to use OSS project in embedded environments, often organizations are willing to pay but the process can be unclear. I remember in my case with OpenWRT an entire Organization was setup just to handle corporate customer’s payment for EOL support.
Perhaps the discussion isn’t a simple “yes should or no we shouldn’t do this” but perhaps "no we don’t do this for the main release process but this could be a way to generate revenue for the organization at large so perhaps we could setup a process ? "
3 Likes
Another former Red Hatter here. elpython-meta/README.md at master · elpython/elpython-meta · GitHub is a concept I wrote up a few years ago to consider a potential approach to “Enterprise Lifecycle Python” that was appropriately respectful of the time and effort of volunteer contributors to the Python ecosystem.
As far as I am aware, none of the Python engineers at our various commercial redistributors actually like that idea.
Instead, they prefer to put their efforts towards making supporting multiple versions less painful through things like backport modules on PyPI, improving the stable ABI in CPython, and ensuring Python packaging is well supported in their environments so users aren’t as dependent on native platform packaging.
Edit: also investment in tech to make “bring your own Python runtime” solutions more acceptable in enterprise environments (such as through Kubernetes and Flatpak for Linux, or conda for cross-platform use)
6 Likes
If you need fapolicyd level of security, then you should not rely on EPEL packages at all. EPEL packages are maintained by the community and come with no security or update guarantees. The quality of EPEL packages vary greatly. Some packagers keep EPEL packages in sync with Fedora sources. Others don’t care about EPEL at all and don’t even backport CVE fixes.
You cannot and must not offload your security and long-term maintenance requirements onto the unpaid community. OSS communities aren’t infinite pools of unpaid labor. Companies have to invest – hire engineers to do the work, sponsor upstream maintainers directly or through other means like PSF, Tidelift or GitHub Sponsors, pay commercial companies like Alma or Red Hat.
8 Likes
While some of the specific technology and product references in Stop Supporting Python 2.6 (For Free) | Curious Efficiency are showing their age, the concept behind that post is evergreen: we set the support lifecycle for CPython releases at 5 years because we think that’s a reasonable limit for what a primarily volunteer based community offering can pragmatically sustain. Anything beyond that requires enough boring grunt work that the only way to reliably make it happen is to start explicitly paying people to provide that extended support.
From the conclusion of my Python 2.6 post:
If users are trying to get long term maintenance support for a community project for free, then upstream developers should be sending a single unified message in response: don’t assume you’ll be able to run new versions of open source software on old platforms unless you’re specifically paying someone to ensure that happens.
… most open source projects are maintained by innovators that are upgrading their technology stacks regularly, where versions of components are already old after 2 years and truly ancient after 5. Expecting open source innovators to provide long term maintenance for free is simply unreasonable - regularly upgrading their own stacks means they don’t need this long term platform support for themselves, so the folks that are seeking it should be expected to pay for it to happen.
Setting that boundary with users as a project maintainer with no external support can be genuinely difficult, so the CPython support lifecycle statements are intended to serve an important secondary purpose: giving maintainers of other components in the Python ecosystem something to point to when setting their own timelines for deciding when to stop maintaining compatibility with particular versions.
12 Likes
Red Hat does NOT provide bugfixes or security fixes for EPEL packages.
Is EPEL commercially supported by Red Hat?
No. EPEL is a volunteer effort from the Fedora community. Just like Fedora itself, Red Hat hosts infrastructure for this project and Red Hat engineers are involved as maintainers and leaders but there are no commercial support contracts or service level agreements provided by Red Hat for packages in EPEL.
3 Likes
I think you’re misunderstanding, or perhaps I didn’t explain it correctly. There is not actually a problem with CPython itself or RHEL/EPEL packages. The problem is with tooling. As a package maintainer who tries to be a good steward, I both maintain support for actively used versions of Python and maintain EPEL packages for the Python packages I maintain (and a few others). However, testing these and trying to move to modern packaging standards is problematic because many tooling maintainers will drop support for Python versions that are no longer maintained by the PSF but are still actively used. As @ncoghlan said, the CPython support lifecycle signals other components in the Python ecosystem about which versions they should support. The idea is to provide an expectation of support for a very small number of Python versions for an extended period of time, so that 1.) package maintainers can migrate to newer tooling 2.) What older versions should be supported in not ambiguous or broad.
Let’s say we decide on a 3 year support cycle for interim releases and a 12 year support cycle for LTS releases. Maybe these are optimistic, but this is for illustrative purposes. In 2034 we’d have 6 supported versions:
3.23: 0 years old
3.22: 1 year old
3.21: 2 years old
3.20: 3 years old [LTS]
3.16: 7 years old [LTS]
3.12: 11 years old [LTS]
In 2035, we’d have 5 supported versions:
3.24: 0 years old
3.23: 1 years old
3.22: 2 year old
3.20: 4 years old [LTS]
3.16: 8 years old [LTS]
Then maintainers who want to support older versions aren’t supporting everything from 3.12 and up, because the community has signaled which older versions to target. Individual maintainers can still choose which versions they support and may choose to only support versions of a certain age or less.
I disagree. fapolicy, like SELinux is a basic security feature that should be on by default on a server. The digital world is getting more and more hostile and you need defense in depth. Yes, the quality of EPEL packages vary, but there aren’t a lot of other resources, so it’s a good place to start because not all organizations have the resources to do it on their own. Ideally they are monitoring those packages and addressing the ones that are outdated. The nice thing is, since EPEL is Open Source, you can submit a PR to update an EPEL package or volunteer to help maintain it. That was actually how I started volunteering with EPEL. My paying job depended on packages in EPEL, but they were out of date. They paid me to backport newer versions and submit them to EPEL. Maybe there’s a business model where this could be provided as a service under a subscription model. If any of you are business-minded and think you can sell it, ping me.
Not all downstream users are big corporate entities with deep pockets. What about schools, non-profits, other open source projects? There are armies of volunteers downstream from us. What motivates me in Open Source is maybe I spend 1 hour doing something, and that saves X number of people X amount of time downstream. That time grows exponentially and when I think about it I feel like I’ve actively had an impact. Maybe they spend that saved time playing Civilization. Or maybe, some of that time goes into another Open Source project that ends up saving me time down the road. And maybe it incidentally makes it easier for some big company to operate and be more secure. I want them to be secure because my data and data of people of care about might be on their servers. I also want them to return the favor with contributions and sponsorships in Open Source. But what they do isn’t going to affect what I do or the responsibility I see for those downstream from me.
2 Likes
What I have done in the past is to keep the tool chain for an old python packaged along side the old python. Then you don’t care if tooling changes. But you may need to patch packages to use the older tooling if upstream has already moved on to newer tooling.
As I understand what you are trying to achieve is using a python LTS as social engineering pressure to persuade packagers to stay on old tooling or multiple tool chains.
This still looks like getting volunteers to do maintenance for enterprises.
5 Likes
Where did you get that from? No, the goals, as I’ve repeatedly stated and restated, are
1.) That the tooling supports actively used Python versions so maintainers that support older Python versions can still use modern tooling and packaging
2.) That use of older versions coalesce around specific versions to reduce the number of targets maintainers need to focus on
The goal here is to help maintainers who are volunteers and to keep them active in the community. Anything else is incidental.
2 Likes
There has been a lot of progress in Python packaging which now results in big improvements so it is understandable that you would want to use this with older versions of Python that predate these improvements. Part of what makes these improvements possible though is eventually dropping support for older versions of things so that new features can be used.
6 Likes
We disagree on how to achieve that goal.
If someone is doing something that nobody reasonable should be asking them to do for free, the best way to help them is to encourage them to stop doing that thing as a volunteer activity. It isn’t to encourage more volunteers to engage in that activity.
Sometimes the thing folks need to hear to help them avoid burnout is just “It’s OK to stop doing the thing you don’t enjoy doing, that’s part of what makes this a volunteer activity instead of a job”.
10 Likes
You clarified that its package maint that matters.
Or you as the person who has decided to continue using an older python version can just keep using local copies of the libraries you were using when you deployed it.
The entire point of an LTS distribution is maximizing stability. This is at odds with wanting to keep receiving changes from libraries.
That doesn’t really work. If I’m using features that don’t exist in 3.11, I can’t support anything less than 3.12, and there’s no motivation to either. This would essentially split the entire ecosystem on libraries that support for 10 years, and get frozen on one python version, and libraries that benefit from improvements to python more frequently.
10 Likes
For what it’s worth, pip’s support policy is to support cPython versions that are not end of life, with the addition that we don’t drop support automatically, we usually wait until download statistics show that usage has dropped to a point where we’re comfortable desupporting. You’re claiming that it’s worth supporting older versions - but the download statistics tell a different story.
Would we support a LTS version of Python if such a thing existed? Maybe - Alyssa’s point about CPython support cycles being a signal for what other projects support is a good one. But if we did, progress on new developments on pip would slow down (and it’s already depressingly slow), because supporting older versions incurs a cost - not least of which is that volunteers aren’t that interested in working on code that has to ignore the last 10 years worth of language improvements.
And you’re welcome to use an older version of pip if you want to support older versions of Python. We won’t stop you. We won’t support you, either, but we only support the absolute latest version of pip anyway, so I’m not sure why that makes a difference - we’re not going to stop releasing new versions of pip…
Sorry if that’s not sufficient for you, but I guess you get what you pay for 
7 Likes