How to package translation files (.po, .mo) in a future-proof way

Summary of the questions (repeated at the end):

  • How can I include all language files with the “new” setuptools way? Ideally, I would like to avoid having to change the configuration file each time a new language is added, as it used to work.
  • For future-proofing, should I use pyproject.toml instead of setup.cfg?
  • After making the required changes, will manifest.in still be needed?
  • For future-proofing, should I use something other than twine?
  • Should I change the directory structure of my project to put “my_app” under “src” as shown in some examples of setuptools documentation to avoid the problems with seemingly picking up files from my virtual environments?

I have a library which contains translation files. Using

python setup.py sdist bdist_wheel
twine upload dist/*

when uploading to Pypi, I’ve recently started getting deprecation warnings such as the following:

... SetuptoolsDeprecationWarning:
    Installing 'my_app.locales' as data is deprecated, please list it in `packages`.

    ############################
    # Package would be ignored #
    ############################
    Python recognizes 'my_app.fr.LC_MESSAGES' as an importable package,
    but it is not listed in the `packages` configuration of setuptools.

    'my_app.locales' has been automatically added to the distribution only
    because it may contain data files, but this behavior is likely to change
    in future versions of setuptools (and therefore is considered deprecated).

    Please make sure that 'my_app.locales' is included as a package by using
    the `packages` configuration field or the proper discovery methods
    (for example by using `find_namespace_packages(...)`/`find_namespace:`
    instead of `find_packages(...)`/`find:`).

    You can read more about "package discovery" and "data files" on setuptools
    documentation page.

As suggested, I did try to use find_namespace, etc., but it then seem to pick up a whole bunch of unwanted files from my virtual environments (I have many virtual environments set up, mostly to test with different Python versions.) no matter what I tried, and I had to abort the process each time.

My project directory structure looks like the following:

my_app/
    __init__.py
    various_files.py
    various_py_dir/
        ...
    locales/
        my_app.pot
        fr/
            LC_MESSAGES/
                my_app.po
                my_app.mo
        es/
            ...
tests/
    ...
venv_1/
    ...

I have the following in manifest.in:

recursive-include my_app/locales *.*

My setup.cfg includes the following:

[options]
packages = find:
include_package_data = True
zip_safe = False
install_requires =
   ...

[options.packages.find]
exclude =
    dist
    build
    ...
    tests
    tests.*
    *.tests
    *.tests.*
    venv*

[options.entry_points]
console_scripts =
    my_app= my_app.__main__:main

[options.package_data]
* =
    my_app/locales/*
my_app= py.typed

Aside: yes, I know, there’s likely a better way to exclude test files, but simply using tests or tests.*, etc., never seem to exclude all of them; however, the above did work for me.

Questions:

  • How can I include all language files with the “new” setuptools way? Ideally, I would like to avoid having to change the configuration file each time a new language is added as it used to work.
  • For future-proofing, should I use pyproject.toml instead of setup.cfg?
  • For future-proofing, should I use something other than twine?
  • After making the required changes, will manifest.in still be needed?
  • Should I change the directory structure of my project to put “my_app” under “src” as shown in some examples of setuptools documentation to avoid the problems with seemingly picking up files from my virtual environments?

The packaging people (including a main contributor to setuptools) mostly hang in the #packaging category, but I am not sure if this question really belongs over there, let’s hope they will see it here anyway…

Some answers from my side:

  • With proper wildcards (*) it seems to me like it should be possible to build a config that would pick up any newly added languages without changing the packaging configuration files. It seems to me like the current wildcards you displayed in your question should do the job. Or is there an issue there already?

  • For future-proofing, it might be a good idea indeed to switch from setup.cfg to pyproject.toml, although support in setuptools is partly still in “beta” (as of version 65.5.0). So maybe try it now if you can afford, but wait a bit before going all in. We can expand on the details if needed.

  • As far as I know, twine is fine, I am not aware of any reason to change to a different tool.

  • I think that it is probably feasible to do what is expected here without a MANIFEST.in file, but I have not done it myself so do not take my word for it.

  • Personally I find that using the so-called src-layout is always a good idea, because in my experience it prevents quite a range of packaging-related issues and simplifies many things (small non-zero initial cost, but big long-term gain).

Yes, that’s my feeling too, which is why I posted it here instead of in the Packaging category.

Looking back at some of the topics in the packaging category, it seems like this one would fit perfectly as well. I do not know if you can change the category yourself. I believe your question deserves more visibility and answers. Or have you figured things out in the meantime?