I would like to point to my Python Packaging Tutorial explaining several common use cases using minimal demo projects.
I am not an expert and assume that some of my solutions might not be the best. So I would appreciate if you can review them.
Background of that project: I am member of FOSS maintainer team and prepare the migration of Back In Time from a makefile based build-system to a modern Python Build system using pyproject.toml & Co. To explore some expect able problems and possible solutions I created this minimal examples. I also do plan a tutorial repo about Debian Python Packaging using the same approach with minimal examples illustrating different use cases.
Thanks in advance,
Christian
PS: Posted this without response also on python-list@python.org and r/pythoncoding and several other places. I was pointed to this discussion forum from the setuptools-people.
On the topic of internationalization, you have basically two approaches: either you add code to compile the MO catalogs during the sdist â wheel build, as you did, or you pre-compile them and add them to the sdist. Iâve personally used the second approach as it felt a bit simpler.
The first approach might be more elegant, but in that case Iâd recommend to use babel in order to not unnecessarily require a Unixy tool on the target system where the sdist is installed.
I also see that you are mentioning python3 -m pip install .. Done outside of a virtual environment, this is not recommended, and will error out by default on recent versions of Debian, Ubuntu and other distros, since it can shadow Python packages installed by the system and thus break system packages written in Python. Either use a virtual environment, or install with pipx instead of pip. See Externally managed environments in the packaging user guide.
Sorry, but I donât understand your second approach. How do I pre-compile? You mean this as a manual step before running pip(x)?
The GNU gettext tools are also available for Windows & Co. But I wouldnât get so far in this tutorial and make it to complex. But I mentioned polib in a code comment in the hatch_build.py file.
IMHO it is more preferable using OS tools instead of increasing Python-package dependencies.
I am aware of PEP 668 and always trust the Python folks when the come to a decision no matter if I understand it or not. But it is new to me and I still struggle with it and donât know how to integrate that into my workflow. I will open a separate thread here about that.
Yes (or before running python -m build or hatch build or similar.)
As you wish. Note, though, that Gettext needs installation on macOS and is, as far as I know, not installed by default on popular Linux distros either, so this isnât really saving a dependency, itâs trading a Python dependency for an OS one.
Issues I see with your tutorial as I come across them:
Here, using helloworld-cli with a hyphen for the name of an import package is a bad idea that encourages bad practices. While itâs not a dealbreaker for a package that only exists to provide a command, having a hyphen in the name of a library package means that import package-name in code that uses the package wonât work, as hyphens are not allowed in Python identifiers. Import package names should always be valid Python identifiers, so change that hyphen to an underscore.
Note that the spelling of the import package is independent of the spelling of the distribution package and the spelling of any commands the package installs, so even if you rename the folder to helloworld_cli, you can still have the project name set to âhelloworld-cliâ with a hyphen, and the package can still provide a command named âhelloworld-cliâ with a hyphen.
âThe solution is the Development Mode âŚâ â Development mode is a solution, not the solution. The general solution is to install your package inside a virtualenv â either using editable mode or not â and do your testing inside the venv. The only advantage of editable mode over non-editable is that changes to the source code are reflected in the environment immediately without needing to reinstall[1], but if youâre using tox or nox or similar to run your tests, your package will get reinstall on each run anyway, so thereâs not much benefit to editable mode.
I would advise eliminating the âStart application âas rootââ section entirely. Installing Python packages as root should not be done lightly, and itâs better to keep that idea out of beginnersâ heads in the first place. You donât even explain what youâre trying to accomplish by installing as root or why itâs necessary for your use case in the first place.
âThe file __main__.py does produce this output with meta data retrieved from pyproject.tomlâ â Technically, the metadata is retrieved from the METADATA file installed alongside the package, which is generated from pyproject.toml, which is not included when installing. By the time hellocentralcli is run, the pyproject.toml file is irrelevant.
PEP 643 & dynamic metadata are irrelevant to retrieving metadata with importlib.metadata. It would be better to just delete the paragraph about dynamic metadata entirely.
Though adding, removing, or renaming files in your projectâs source can require you to reinstall anyway even in editable mode âŠď¸
I understand your point. From my point of view the installation of gettext is a dependency that will be handled by the distros package maintainer. My users are intended to install my âpackagesâ via their official distro repository. From upstream or from PyPi (where I do not publish) is vorbiden by ânormal usersâ.
I fixed this in the README.md. The source itself is correct.
From my point of view the benefit is that I donât need to deal with virtual environments, their side effects and redundant packages. But that topic is discussed in another thread.
I understand your point. But in the background of my real project (Back In Time) there is a real use case for this. That is why this example does exist.
The fact that this repo can be used by other persons as a tutorial is just a secondary side effect. The main purpose of that repo is to support the migration of the Back In Time project from its make-based build system to a modern Python build-system. As a backup application it needs to be able to get started as root.
The repo is GPL. Please feel free to re-use it somewhere else.
The âas rootâ example is not finished yet as the warning boxes trying to indicate. So I am open for alternative solutions reaching the same goals.
I am aware of that. The intention of the repo is not to give deeper understanding of the concepts. There are much better ressources for that and I try to link them.
Can you explain why? I gave my arguments about âdynamic meta dataâ.
Thanks a lot for your feedback. I will take that into account. And you gave me a lot of things to think about. I am still learning (and not surprised by that).
Thatâs not a benefit of editable installs. Editable vs. non-editable installs are orthogonal to virtual environments. If youâre not using a virtual environment, then pip install -e . and pip install . will both install your package system-wide, and both will let you test it with pytest.
If the reason for installing your package as root is solely so that you can run it with sudo[1], I would suggest instead installing as non-root (using pipx or pip install --user or a virtualenv) and passing the absolute path to the installed program to sudo, e.,g., sudo ~/.local/bin/my-command. Depending on how your sudo installation is configured, you may be able to instead preserve your PATH (and HOME?) when sudo-ing, letting you just do sudo my-command, but that may come with subtler security issues.
Then again, youâve also mentioned that your users are intended to install your packages via official distro repositories, so you could just install the distro packages as root (which I believe is the default or only option for most distros) instead of using pip in the first place.
You donât need to give deeper understanding, but you should at least not give incorrect understanding. importlib.metadata does not retrieve anything from pyproject.toml.
Because dynamic metadata has nothing to do with the section youâve placed those arguments in. If you want to say things about dynamic metadata, have a dedicated section for it instead of tacking it on to the end of something unrelated.
Is that why youâre installing as root? The README and your comments here arenât clear. The README should be more explicit about this. âŠď¸
Pretty sure itâs bad practice to run apps as sudo when they can be modified by regular users. Thatâs a privilege escalation just waiting to happen.
Inside your own user directory means itâs just as secure as your own account, which presumably is also the one that gets used for random internet browsing, etc.
Best security practice (which not everyone needs) will be to require sudo to modify the apps that youâre going to run with sudo, and to not pass through environment variables referring to user-modifiable locations.
I really appreciate your contributions to this discussion and dealing with a stubborn one like me.
You gave me a lot new insides and things to think about.
It will take some time to come to a conclusion for my self but I will modify my tutorial repo.
This is exactly how things work, if you use any of Python packaging tools other than the things we actively tell people to not do, e.g. directly do python setup.py install.
Itâs, for example, exactly how wheel-based installs work and how pip would install packages from a source distribution.
This post is among other things some of the worst advice Iâve ever read, a disingenuous take on the history of packaging in the Python ecosystem, and a flat insult to the hundreds of hours of completely volunteer effort contributed in this space.
Iâm glad youâve found a workflow that works for you. Packaging in Python has a long and winding history. We all know itâs a mess that needs improvement, which is why this forum is filled with posts about it. If you wish to contribute, do us all the service of not actively misleading others.
