Support t-strings in gettext

Ideas
1

Wouldn’t you like to write your i18n code using gettext like this?

msg = _(t'Hello {name}')
msg = ngettext(t'{n} snake', t'{n} snakes', n)
flash(_(t'Category "{cat.title}" moved to "{target_cat.title}"'))

instead of this?

msg = _('Hello {name}').format(name=name)
msg = ngettext('{n} snake', '{n} snakes', n).format(n=n)
flash(_('Category "{}" moved to "{}"').format(cat.title, target_cat.title))

Because the latter isn’t super pretty to begin with and prone to errors such as calling .format() inside the gettext call instead of outside, or using positional multiple placeholders (and some languages may require a different order).

Now to the tricky part: How to convert the expressions from the t-string interpolation to names that are suitable in format strings and translations.

My proposed way to solve this is to derive a format string field name from the expression, and reject anything that’s too complex or too dynamic. My current implementation covers simple cases such as plain variables, accessing attributes and items, and calling a method w/o arguments:

PS: Maybe an interesting fact: A less powerful version of this exists in Jinja2 templates for a very long time - it lets you use plain variables inside {% trans %} and uses the variable name during extraction, but the value during translation.

7 Likes
2

This example cannot be translated because the two fields need to be swapped in some language translations.

This uses the %(name)s style to allow for this and is supported by xgettext(?).

flash(_('Category "%(cat)s" moved to "%(title)s"') % {'cat': cat.title, 'title': target_cat.title})
3

Yes, it was intended as a bad example that is avoided altogether when using t-strings: “or using positional multiple placeholders (and some languages may require a different order).”

The easiest way to do this correctly (w/o t-strings) would be to simply use named placeholders (or explicit positional placeholders ({0} and {1}), but those would easily confuse translators)…

4

We explored the use of t-strings for i18n back when PEP 750 was under development. We determined it was not a good fit and was outside the scope of t-strings, both of which are totally fine. The right tool for the job, IMHO is my library flufl.i18n which was originally developed for GNU Mailman, still used there, but useful in any other context. It’s built on top of stdlib Template strings which uses $-syntax which is much more friendly to translators than %-strings[1]. GNU gettext supports Python directly, so that toolset is well designed for use in i18n’d Python code.

  1. and was developed exactly because translators understandably got %-strings wrong in translations ↩︎

2 Likes
5

We determined it was not a good fit

What made it not a good fit? Simply the fact that more complex expressions would not work well in there?

and was outside the scope of t-strings

Keeping it outside the scope of PEP750 made perfect sense IMHO. Doesn’t sound like a good reason against adding it independently from this…

The right tool for the job, IMHO is my library flufl.i18n

At least in the Python webapp world I have yet to see any major webapps that use anything but Babel… So realistically, pretty much everyone uses something else.

A quick search on GitHub also confirms this: 32 matches for flufl.i18n vs over 125k matches for babel in common Python dependency files

https://github.com/search?q=flufl.i18n+path%3Arequirements.txt&type=Code https://github.com/search?q=flufl.i18n+path%3Asetup.py&type=Code https://github.com/search?q=flufl.i18n+path%3Apyproject.toml&type=Code https://github.com/search?q=babel+path%3Arequirements.txt&type=Code https://github.com/search?q=babel+path%3Asetup.py&type=Code https://github.com/search?q=babel+path%3Apyproject.toml&type=Code

and was developed exactly because translators understandably got %-strings wrong in translations

Yep, they do. But regardless of the syntax, the main mistake I see is people translating placeholders…
In any case, applications like transifex (and probably weblate as well?) tend to highlight placeholders separately to make this less likely to happen.

And in fact, my proposed solution using t-strings actually avoids all the problems that come with more complex format placeholders, because only the name is part of the extracted strings - any format/conversion specs live only in the code, and thus translators never see those. So from the simplicity level it’s literally $foo (w/ flufl.i18n / string.Template) vs {foo} (w/ t-strings). Both looks equally easy to get right (or wrong) for translators… At least babel also adds python-brace-format to the pot file metadata for these strings, so tools used by translators know what kind of placeholders to expect.

1 Like
6

You may want to compare what is possible with t-strings and what fluent does to be capable of handling languages that don’t map 1:1 with English and how it seperates what developers need and what localizers need better than extracting strings from source code in any format. I would not recommend t-strings here, nor would I recommend any large project go with a gettext based solution for new localization (yes, don’t throw away the existing work if you have already managed to get your app localized)

7

Yeah, gettext is not as powerful as you sometimes need for perfect translations since you can’t take into account gender, etc. But realistically it’s still what nearly everyone uses… I don’t think it’s really in scope here?

FWIW I completely agree that t-strings make no sense for this type of translation, where you usually just use some identifier for a message, and have the message w/ all the details, conditions, etc. outside the Python code.

8

Okay, more directly: Even if there weren’t other issues[1] with t-strings for i18n that make it worse than gettext in it’s current form, I wouldn’t think it’s worth changing gettext to support this, partly because it would be ideal if we weren’t pushing users towards gettext-based solutions.

  1. Here’s a direct link to where it was explained in more detail from the discussion during pep 750’s development about it not being a good fit PEP 750: Tag Strings For Writing Domain-Specific Languages - #85 by barry ↩︎

9

To me, this is what makes this proposal better for a PyPI package than stdlib. There is no single obvious answer, and in the stdlib we only have one chance to do it right. A package on PyPI, on the other hand, can be much bolder in picking something that works, change later (with backwards-compatible improvements that don’t require a new Python version, and incompatible changes that won’t hold people back from upgrading all of Python), or even offer customization.

2 Likes
10

Sorry if I’m a little late for the party …

When I first learned about f-strings and started to use them, I thought,
“Cool! What a pity they are not usable for i18n.”

However, they are pretty popular, aren’t they?

When I first learned about t-strings, I thought: Now we can internationalize and localize just as easily!

Existing code, using

print(f'some {placeholder}')

could use

print(_(t'some {placeholder}'))

instead, and be done (as for the code, of course).

Wouldn’t this be cool?!

Sadly, this was not the primary idea (surprisingly, if you ask me). But of course, it can still be made possible.

There would be a few things to consider:

  • Like Barry pointed out, the translation function would need to know about the original shape of the message id, to query the catalog for it. The most convenient solution would be an attribute of the string.templatelib.Template class; but it would be sufficient to have some tstring_to_msgid function.
  • We’d need to create templates from the translations we get from the catalog.
  • Of course we’d need to handle changed positions of the placeholders.
  • We could (for a first shot, at least) constrain the expressions in the translations to equal the expressions (and formats, perhaps) in the input templates.

This doesn’t look like science fiction to me; I’d consider it a flagship use case for t-strings.

And it would be even better once we have dedented multiline strings (d-strings), as of PEP 822:

print(_(dt'''
There was some problem in line {line}:
{msg}
'''))

(which would use a 2-line msgid, with the indentation removed).
(There was an indentation in my input, and I hoped the editor would keep it …)

Since d-strings have been moved to Python 3.16, I suggest to add t-string support to gettext in Python 3.16 as well.

See as well PEP 787 which proposes to enhance the subprocess module in a similar way.

11

Yes, fluent might be superior to gettext, in some ways.

So what?

Gettext is out there, and it is heavily used.

It is frankly not true that gettext wouldn’t take in account languages which “don’t map English 1:1” (very few languages will do so); for example, gettext supports more than two plural forms.

We shouldn’t avoid improvements to the gettext module just because we think there might be some better alternative to that module. The optparse module was deprecated as of Python 2.7, but argparse could never entirely replace it; the deprecation was a mistake, and the module was “mercied” in Python 3.11.

Perhaps gettext can benefit from t-strings support more than fluent could; perhaps fluent could benefit as well (it uses ‘{…}’ placeholders as well, AFAICS).

However, I consider the fluent vs. gettext discussion to be off-topic here.

1 Like
12

As @Tobias42 said gettext works well for translations to many languages. I worked on a project that used gettext in python to translate to 15 languages, lots of european, Japanese, Arabic, Thai, and Chinese variants.

2 Likes
14

The indentation did infact not get saved. Using triple-backticks (```), then a language specifier (pyorpython), followed by a newline and the code, and ended with triple-backticks (```) again should work.

Your example

E.g. this would be the output for your example (assuming 4-space indentation):

print(_(dt'''
    There was some problem in line {line}:
    {msg}
'''))

(Note: I’m sorry if this is a bit off-topic.)

1 Like
15

Ah, thank you!