Who works on digital sustainability in the Python world?

Ideas
1

I was at a tech un-conference today and there were a couple Python people in discussions about digital environmental sustainability (of the web specifically but also interested in other areas). This made me wonder who else is working on technology’s climate impact in the Python world? Are there any specific PSF initiatives / interest groups working on this?

I’m interested in any possible existing work. For example:

  • Conference carbon footprints (do PyCons report that?)
  • Carbon footprint of CPython’s Continuous Integration
  • Power consumption benchmarking of different software
  • Feature roadmaps of various open source projects to reduce their users’ carbon footprint

From my side I’m approaching this in the context of Django. There are about 50-500k projects built with the framework. Total carbon footprint is on the order of 100-500k tons of CO2 equivalent per year. So any improvement we could make to the framework would go a long way in reducing the industry’s carbon footprint. Or also making improvements by sharing relevant resources with Django users.

3 Likes
2

It would be nice if mobile apps took lowering their power consumption more seriously.

1 Like
3

I agree with @JamesParrott here

4

How are these numbers estimated? If there was no Django would we save that amount?
I ask because I commented on a LinkedIn post from someone stating that switching to Rust would save power because it is faster, but they gave no figures - as if it is self evident??

I think one needs to measure. If Django were 10% more efficient, leading to a 0.01% power saving, (figures are random), would it be worth developing that efficiency change in Django, (with the added need to debug the changes)?

3 Likes
5

that isn’t per se true. but assuming that an efficient Python implementation (thus it’s not using Django) is compared with an efficient implementation with a language that compiles to native CPU instructions, you’d need less server capacities for a web service with the latter.

this also assumes that the resource exhaustion caused by porting and maintaining that more efficient implementation doesn’t outweigh the savings.

6

But we do need to measure. In a web environment, maybe the need to run a database may take the lions share of resources anyway?

7

regarding your examples, i think it makes sense to categorize these into must-avoid and must-reduce categories.

for shipping speakers and audiences to conferences by aircraft there’s no carbon-free alternative. this just needs to stop. it’s simple.

regarding the computational power consumption, this needs to ensure to use clean energy (note that Microsoft, on whose CI and hosting many rely on, is about to re-open a nuclear power plant far away from their headquarters) and it needs to be massively reduced so that clean energy capacities can replace current use of fossil fuels.

8

Great to see there’s interest in this! I’m not aware of much Python usage in mobile apps but yes that’d certainly be good to be more efficient.

This is based on the Sustainable Web Design model, which attempts to quantify server, network, and client device energy use for websites. Source data is my own crawl of the web + HTTP Archive dataset + CrUX report, I can share more numbers if it helps :slight_smile:

If there was no Django – I presume the majority of those sites would still exist, built with other tools. The emissions would be for those other tools to worry about (and yes could be lower or greater).

Re Rust, yes, it’s pretty clear switching to a more performant programming language for equal use leads to lower energy usage. Most CPUs consume less energy when idling than at 100% utilization (though not nothing), and if CPU usage is lower it also makes it possible for sites to provision a server with less beefy hardware.

However this is one of many factors, not a silver bullet.

In a web environment, maybe the need to run a database may take the lions share of resources anyway?

Yes that’s occasionally the case, depends lots on the architecture. The database often is “always on”, while web servers can be scaled on/off according to traffic.

for shipping speakers and audiences to conferences by aircraft there’s no carbon-free alternative

well, I imagine you’re aware of this already but online events and travel by train are clear alternatives :slight_smile:

9

sorry, that i dodge to answer thouroughly. sure, there’s interest. but we know how this matter is absurdly proritized.

i’m just dropping by to link to this blog post by @brettcannon where he addresses the need to change: Why I put in some effort to lower my carbon footprint

okay, and while i’m here and the post’s title provokes my postition – “some effort” is far too less – i wanna contrast his argument that everybody’s of course better off when the preconditions of human life are preserved evenually. i had this perspective for a long time in me and recently concluded to be honest about it to me: this goal is meanwhile unreachable – what will matter imo over the next decades will be an imperative that the collapse causes as few harm and misery among mankind as doable and that we don’t lose the sense and value of human dignity in the process. i hope i will be wrong with that current assesment (argueing against physics though…), but i honestly think that this possibility needs to be considered for reflections and planning with regards to sustainability of humanity.

1 Like
10

I thought I’d share a quick update for anyone interested in this :slight_smile: I’ll be at FOSDEM this weekend along with other sustainability folks if anyone’s around.

This second one I’m hoping will result in a Python implementation of co2.js which will make it easier for Python projects to quantify their emissions. And Python-friendly API client for live grid data.

I hope people find this interesting – and yeah hope to find other Python people to discuss digital sustainability work at FOSDEM!

1 Like
11

As interest has been expressed, here is down-to-earth summary (a conjecture in fact):

Working on sustainability in context of a computer language is a) costing more in negative environmental effects than doing nothing and b) is negligible compared to other problems which should be addressed instead, distracting attention from actual problems.

While imo there are valid topics around this subject, the current discussions (that I’ve seen) are far from them, for reasons.

If you want the original text, pm me. It does not contain anything remotely controversial. Just a lot of people seem to have thin skin these days.

1 Like
12

could someone please reinstate the comment by @lukjak ?! while it is clearly of polemic nature, it addresses important aspects. imho this thread isn’t disturbed by the argument made. it may evoke more targeted thoughts and exchange.

1 Like
13

It is though. Outright hostility towards people interested in sustainability will keep timid users from contributing.
If the purpose of the post is to advance discussion and not to intimidate people, the same points can be made in a way which accords with the CoC.

To whit:

I would not like to see carbon footprint analysis embedded into applications. I think that’s a mistake which increases the aggregate load that our applications generate in CPU, network usage, storage, etc.

Good practices for writing efficient code can make a difference and I think that’s one of the best ways that the Python community can advance the overall goal of making our software lower impact. The incentives around this are clearly aligned, since faster software is usually nicer for users.

And initiatives like PEP 751 (to pick somewhat at random) may have surprising and outsized impact, as package resolution is a complex and costly process which runs across millions of machines daily.

One of the places which is interesting to look is how projects use CI services. I believe that the cost of a broken package is higher than a few hours of CI time, and so I often define extensive test matrices. But even large and complex CI configurations can often be optimized to avoid repeating steps like building a wheel from source.
If you want to help make things more efficient, plenty of projects could use some help improving their configs in gitlab, github, circleci, etc.

All that said, I’m personally a pessimist about our individual abilities to make a big impact. We should try to be good citizens, and I’m advocating for that, but large institutional changes are needed to disincentivize burning CPU on Litecoin clones and low-value LLM based applications.

4 Likes
14

Agreed here. This can be analyzed at scale more efficiently and with lower impact by simply[1] measuring power consumption.

The major thing we as individual users can do is reduce use. The best methods we have for this are being smarter about what we do and don’t cache, what work actually needs computing eagerly, choosing reasonably performant options, and to not eagerly dismiss small improvements that do all add up all together. With some complicated exceptions related to my prior footnote notwithstanding, performance correlates pretty directly with carbon footprint when it comes to the software side of things.

  1. Those who care rather than dismiss it as wasted effort should know I am hiding a bit in “simply” as the source of energy, and relative load compared to baseline power generation in the locality of consumption matters as well. (ie. scheduling your background jobs at night is more than just a way to disrupt users less) ↩︎

3 Likes
15

Hahaha, at least I can say the purpose of my posts is to find other people working on this in the Python world, and potentially interested in collaborating on some of those topics. That’s about it. Once enough of us stop arguing about whether there are improvements to be made, we can get on with it.

And yeah there’s certainly lots that can be done outside of digital tech and programming languages. But we have lots of devs who are going to be coding in Python, so we might as well help them make that coding low-impact. And who knows, along the way they might become knowledgeable and care more about emissions / environmental footprint outside tech too!

There’s honestly so little happening in this area of tech at the moment that I think there’s a tremendous opportunity for individuals to move the needle, and be the advocates for the larger institutional changes. Just a few practical examples from the list I shared initially:

  • Doing a conference carbon footprint assessment for a PyCon is a one-person job, that as far as I know has never been done before. That kind of work has the potential to get hundreds of people reconsidering their travel habits.
  • Continuous Integration optimizations, definitely, particularly for Python projects looking at the Python versions matrix as others have mentioned.
  • I’ve not seen many Python projects report their carbon footprint. What’s it like for PyPI? What’s it like for NumPy code? I’m sure there are opportunities there to make big relative improvements – but we don’t even have good numbers to understand if we’re talking coffee cups or transatlantic flights.

Even excluding actual changes, it’s certainly possible for a few individuals to learn about methodologies to estimate the energy use and embodied emissions of any infrastructure, put the numbers together, and start proposing improvements.

One very concrete example (no public source to corroborate, sorry): A few months ago two colleagues of mine re-configured 30 of our internal Django apps to switch off outside of office hours. That’s removing 5-10 tons of CO2 emissions per year, not to mention lower hosting costs. That’s about 2-5% of our company’s carbon footprint right there.

In a similar fashion – in Europe, the UK NHS is one of the biggest employers (100k+ people?) and nhs.uk is one of the highest-traffic websites, reported around 1.2B page views / year. That site’s carbon footprint is probably on the order of 1000-2000 tons of CO2 per year. I’d hope they use a lot of caching, but I’m sure a speed bump to CPython, or to Jinja / the Django Templates language, or perhaps better support for newer image formats in Python – any of those things would bring sizeable improvements.

Anyway – at the scale of Django or Python more broadly, there’s millions of servers out there running this tech, so definitely any opportunities to reduce that footprint that is worth considering. Whether that’s by doing the same work on smaller servers, or making sure the servers use renewable energy, or doing less for equal results.

:thinking: citation needed, I guess? The figures I have put digital infrastructure at around 400-800 TWh of global energy use per year. That’s around 2-4% of global electricity use, growing fast, cf. Thinking about using AI? report by the Green Web Foundation. Seems like an actual problem to me? For example if that increase in energy use means more work for coal-fired power plants.

1 Like
16

This sounds like premature optimization. Here are some estimates comparing the carbon footprint of employee commuting to that of the website.

Even small changes can make a difference in the bigger picture of sustainability. Plus, improving website performance can help cut down emissions and make things more efficient overall.

Total NHS commuting footprint: 720,000 tons of CO₂ per year. Total NHS website footprint: 1000-2000 tons of CO₂ per year, approximately 0.2% of the carbon footprint from commuting.

Carbon Footprint Calculation for NHS Employee Commuting (Category 7 - GHG Protocol)

Assumptions:

  1. Total NHS Employees: 1.5 million
  2. Average Round-Trip Commute Distance: 20 km per day
  3. Workdays Per Year: 250 days
  4. Average Car Emissions: 120 g CO₂ per km (for a gasoline car)
  5. Percentage of Employees Commuting by Car: 60% (900,000 employees)
  6. Public Transport (Bus & Train) Emissions:
  • Bus: 80 g CO₂ per km per passenger
  • Train: 40 g CO₂ per km per passenger
  • Carpooling & Walking/Biking ignored for simplicity
  1. 40% Use Public Transport: 20% by bus, 20% by train (600,000 employees total)

Step 1: Emissions for Car Users (60% of Employees)

  • Per employee:
    20 km×120 g CO₂/km=2,400 g (per day)
    • Annual per employee:
      2,400 g×250=600,000 g=600 kg CO₂
    • Total for 900,000 employees:
      900,000×600 kg=540,000,000 kg=540,000 tons of CO₂

Step 2: Emissions for Public Transport Users (40% of Employees)

Bus (20% of Employees = 300,000 people)

  • Per employee:
    20 km×80 g CO₂/km=1,600 g CO₂
    • Annual per employee:
      1,600 g×250=400,000 g=400 kg CO₂
    • Total for 300,000 employees:
      300,000×400 kg=120,000 tons of CO₂

Train (20% of Employees = 300,000 people)

  • Per employee:
    20 km×40 g CO₂/km=800 g CO₂
    • Annual per employee:
      800 g×250=200,000 g=200 kg CO₂
    • Total for 300,000 employees:
      300,000×200 kg=60,000 tons of CO₂

Total NHS Employee Commuting Carbon Footprint

  • Car users: 540,000 tons CO₂
  • Bus users: 120,000 tons CO₂
  • Train users: 60,000 tons CO₂
  • Total NHS commuting footprint: 720,000 tons of CO₂ per year
Assumed Carbon Footprint Calculation for NHS Website (GHG Protocol – Scope 2 & 3 emissions)

Assumptions:

  1. Website Traffic: 1.2 billion page views per year.
  2. Data Transfer per Page Load: 2 MB (assumed average webpage size).
  3. Total Annual Data Transfer:
  • 1.2B page views×2MB=2.4PB (petabytes) per year.
  1. Energy Consumption for Data Transfer:
  • 0.5 kWh per GB (based on standard estimates for internet energy use).
  • 2.4PB=2,400,000GB.
  • 2,400,000×0.5=1,200,000 kWh (1.2 GWh) per year.
  1. Emission Factor for Electricity:
  • 300 g CO₂ per kWh (assuming a mix of renewable and non-renewable energy in UK data centers).
  1. Server & Data Center Overhead:
  • Assume 2x factor for cooling, redundancy, and storage.

Carbon Footprint Calculation:

1. Website Hosting & Data Transfer Emissions

1.2 GWh×300 g CO₂/kWh=360,000 kg CO₂=360 tons CO₂

2. Data Center Overhead (Cooling, Storage, etc.)

360×2=720 tons CO₂

3. User Device Energy Consumption (Client-Side)

  • Assume 50% mobile, 50% desktop.
  • Average device energy use per visit: 0.02 kWh.
  • Total annual user energy: 1.2B×0.02=24 GWh
  • User-side emissions: 24 GWh×300 g CO₂/kWh=7,200 tons CO₂

Final Estimated Carbon Footprint of NHS Website

Source CO₂ Emissions (Tons)
Website Hosting & Data Transfer 360
Data Center Overhead 720
User Device Energy 7,200
Total Estimated Footprint 8,280 tons CO₂ per year

(Disclaimer: LLM tools were used as calculators. All facts are verifiable.)

PS: I am not counting the thousands of cars that passed by my window while I was compiling these calculations.

17

If you want to calculate website emissions, there’s really no need to trust LLMs with those maths! Have a go with Website Carbon Calculator, Ecograder, or GreenFrame. Or read the HTTP Archive’s Sustainability chapter of the Web Almanac.

Oh yes, how I wish we realise in 10 years’ time it was premature optimization! If things turn out that way – happy times. We’ll just enjoy the performance improvements and cost savings and move on to other pursuits. For now it seems to me all the indicators of digital tech’s impact on the environment are pointing at things getting much worse. Before they might some day get better – assuming people do something about the issue.

Re employee commuting and cars – I don’t see the relevance? Transportation is a major source of pollution, and lots of people work on that. But this is the Python forum and I’m a Python dev, hence why I started a thread about digital sustainability. If people want to work on other sustainability topics than those adjacent to Python, that’s great, power to them.

3 Likes
18

Agreed, but this is where making Python (and Python projects such as PyPi) more efficient makes a lot of sense, no? A tiny improvement here has a big impact.

There seems to be a lot of whataboutism going on in this thread. Like sure cars pollute more. Does that mean that nothing else matters?

4 Likes
19

I’ve already done the calculations based on my assumptions and carbon emissions data from reliable sources, which are freely available on government websites. I used an LLM for convenience, and I’ve double-checked the calculations. Everything looks correct, but feel free to point out any errors.

I’m not against the initiative—I just think it’s an example of premature optimization. In the long term, it could end up costing more than focusing on the bigger picture.

Improving productivity through better tools and processes naturally boosts efficiency, resulting in lower energy consumption and, in turn, fewer carbon emissions. While this may not always be the main goal, it is a powerful and indirect outcome of enhancing system productivity.

That said, I don’t believe it’s helpful to belittle the incredible work done by thousands of developers over the past three decades, which resulted in one of the most productive programming languages ever created.

Considering the initiative began three decades ago, many of those developers deserve to be recognized in the hall of fame.

One last thing, though—I’m not entirely sure what you mean by this:

Could you clarify?

1 Like
20

Yes – I meant any efficiency or sustainability improvements to the Django framework can lead to a big reduction in carbon emissions as the project is widely used. This is true for any popular open source project. For web frameworks, efficiency improvements go a long way. Things like:

  • Reducing memory use so live servers can be provisioned on smaller machines
  • General web perf improvements (faster = lower energy use per request, possibly smaller servers) - HTTP/2 or HTTP/3, good HTTP caching & compression
  • Good support for scaling infrastructure up and down according to demand, to avoid over-provisioning

None of those ideas are specific to the Python world, but the implementations will be. For example see Granian. And for a more Django-specific example, research in the suitability of SQLite in production has the potential for people to get the same results as other DBs with lower infrastructure requirements (so lower footprint).

There’s not necessarily a need to focus on sustainability as the driver for those improvements - that can just be an addition to usual benchmarking work, for example having targets on power use or infrastructure requirements :slight_smile:

There’s lots of other opportunities for improvement that are focused on carbon footprint rather than general efficiency. And lots of opportunities to optimize what’s happening in the browsers of the users of those web frameworks. If anyone’s interested back in 2023 I spent time researching a roadmap of improvements to reduce the carbon footprint of Wagtail websites (a Python CMS powering on the order of 10-20k sites), and produced my top 5 list for sites to lower their energy use by 90%.

2 Likes