Wanting a singular packaging tool/vision

Packaging
80

Hey all,

I’ve been following the thread a little and thought it would be useful to give a quick heads-up that I’ve been working at Anaconda for a bit over a year as a tech lead on conda, and we’ve been heads down on catching up with technical and organizational debt of the past ~10 years.

A while ago, I co-founded PyPA and maintained pip/virtualenv in the early days (literally taking them off Ian’s hands), as some maybe still remember. I took the job at Anaconda in particular because I think conda has played an important, parallel role in enabling a huge amount of Python users to solve their tricky packaging problems. The elephant in the room is the continued growth of the Python community and the need for a diverse set of tools (based on standards/specs) to cater to it. I hope to build bridges between conda and the PyPA stack as much as possible to improve the user experience of both ecosystems. I’m very excited to see the current generation of packaging tools like Hatch, PDM and Poetry.

Here are some broad stroke comments/ideas:

  • conda is now a multi-stakeholder OSS project, which is being recognized in recent updates to its governance policy. If anyone from the early days of conda reads this, I hope you enjoy seeing that. Anaconda is still invested (and increasingly so) but it’s not the only stakeholder anymore.
  • The conda maintainers have made the same painful mistakes of over-optimizing for a particular subset of users in the scientific community, like PyPA has done for web/infra users.
  • The problems described in this thread about covering the “full packaging stack, with non-Python dependencies” are real and largely solved by conda (among others), and the main focus is now on keeping up with 3rd party ecosystems like Python/PyPA and catering to the community growth.
  • The growth of conda-forge (and to a lesser extent Anaconda Distribution) continues to require investment into scaling the build and distribution infrastructure, improving build tooling and catering to user needs.
  • conda/PyPA compatibility is not perfect, there is a badly named pip_interop_enabled config option which makes it take Python packaging metadata into account for some subcommands. This will hopefully be improved and become the default in the future.
  • Conda specifications, schemas and enhancement proposals are steadily being written by contributors and voted on by the Conda Steering Council.
  • The discussions I had at SciPy this year with @CAM-Gerlach and @henryiii about trying to align the conda packaging format with the wheels format were mostly about getting the temperature in the room, and I hope to be working with @dholth in the future on it.

Full disclosure, I don’t love engaging with these catch-all threads, since they tend to miss the nuance of the topic and lack actionable results, but I’d be interested in being proven wrong :slight_smile:

So if you have any specific questions about conda, do not hesitate to ask!

15 Likes