[announce] Pybi and Posy

Packaging Announcements
29

I’m really glad to see some innovation in this direction :slight_smile:. Currently working with and distributing python code can be really convoluted and confusing (not only for new users). This is both due to the plethora of tools (that each only do part of what is needed) and the broad spectrum of things python can be used for. My team uses python for

  • libraries
  • backend applications (run as containers)
  • internal tools, mostly for DevOps and automation
  • data science and scientific computing

The tools we currently use are:

  • pyenv to install python and to create some long lived, manually activated python environments for quick and dirty experimentation
  • pdm to manage python projects (libraries and applications)
  • pipx to install some python apps we use (black, copier etc.). Some of these require the use of pipx embed <some_package> to install additional packages alongside the application e.g. jinja2 extensions for copier
  • pyinstaller / pyoxidizer to distribute internal applications to non-developers
  • pip to install requirements.txt with hashes in docker containers
  • conda/anaconda for some data science projects that were just easier to get started with conda (think pytorch, tensorflow etc. ). Also Azure Machine Learning IIRC uses conda behind the scenes and exposes this via the SDK to the user.

Problems we have encountered along the way:

  • getting new developers (e.g. interns, students or apprentices) started. Setting up python to write their first library or application can be a real challenge. Even if you use pyenv to install python, how do you install pdm? Using pipx maybe? How do you install pipx? This feels like going in circles. Surprisingly it is far easier in my experience to get people without any prior python knowledge started. People that had some prior exposure to a mix of apt-get, pip, venv, virtualenv and horrible PATH manipulation are even more difficult to get started. In addition to just showing them (what we believe) is a sane way to setup your workspace you have to actively unlearn bad habits with them.
  • distributing applications to non-developers. Same as the above but even worse. Using pyenv to install python and then pipx to install an application is already too much for most non-developers. We have had mixed success with pyinstaller and pyoxidizer mostly due to dependencies that didn’t want to work properly when bundled this way…
  • switching between “regular” python and conda especially if you only use conda to create new environments every once in a while.
  • dependency conflicts when developing an application, where two dependencies couldn’t be installed due to (unnecessary) upper version bounds on their dependencies. Haven’t had this problem since we switched from poetry to pdm but what I remember is that you basically had to either wait for the maintainers to release a new version with updated dependencies or (temporarily) fork the dependency and bump the sub-dependency yourself. As an application developer I’d really appreciate an escape hatch that let me override whatever the dependency solver thought was correct and just let me use the version that I tell it to.
4 Likes