Adopting/recommending a toml parser?

pip recently switched from pytoml to toml, and flit might well follow it. But I notice that toml has not had a release since 2018, and the pip PR I linked contains a discussion about a newer TOML syntax which Poetry would like to use, but pip would be unable to parse.

@dholth suggested that PyPA should take over maintenance of a toml parser. I think it’s a good idea to ensure there’s one package we can wholeheartedly recommend, because we’ve adopted the format in packaging standards (PEP 517 & 518). I think this should ultimately be a candidate for adding to the standard library, maybe once the TOML specification has reached version 1.0.

The three contenders that I’m aware of:

  • toml (github): last release October 2018, last commit January 2020. Newly vendored in pip.
  • pytoml (github: last release July 2019, last commit July 2019. Marked as unmaintained in the readme.
  • tomlkit (github): last release April 2020, last commit April 2020. Supports newer TOML features than the other two. Extra features to work with comments & layout, beyond simple parse/dump.

Tomlkit is clearly the most actively maintained, and it belongs to @sdispater, who’s also the author of Poetry. However, I imagine that the ability to preserve and manipulate style adds a considerable amount of complexity relative to a library which parses TOML and discards the style. I find it hard to imagine that being added to the standard library, for instance.

4 Likes

I will mention that once TOML hits 1.0 I plan to talk to python-dev about somehow getting a TOML parser into the stdlib. This might also lead to a reckoning over what the future of the stdlib should be. :grin:

3 Likes

I chose pytoml for enscons because its source code seemed easier to read than toml, I understand @takluyver had the same impression. We all chose TOML as a file format because you can have interoperable parsers. It’s not a lot of code.

Doesn’t @pradyunsg maintain one?

Reading through the feature list and implementation I personally find tomlkit to be the best implementation. IMHO ability to pass the roundtrip (read and then write equals the same file) should be offered.

I don’t think so. IIUC @pradyunsg is one of the core members of the TOML spec, but they don’t maintain a reference implementation.

2 Likes

Hi I just want to mention that I have wrapped around a C++ single header library to provide a fast TOML: parser in Python.

The repo is here: https://github.com/bobfang1992/pytomlpp

Thanks Bob. For my purposes, the ease of managing a pure Python package outweighs the performance benefit of a compiled parser, but it’s great to have a compiled option if people need that performance.

If pure speed is what one is looking for there’s https://github.com/samuelcolvin/rtoml too. I tried to compare pytomlpp to it, but did not manage to make pytomlpp work on MacOs.

Interesting, but toml++ uses C++17 – an extremely recent version of the C++ language – , while PEP 7 mandates the use of “C89 with several select C99 features”. So I’m afraid integrating pytomlpp into the stdlib would be a no-no, and would remain so for at least the next 10 years.

As for speed, well, the dominant use case for TOML is configuration files, and I don’t think those are usually performance-critical (unless you’re batch-processing thousands of them, perhaps :slight_smile: ).

2 Likes

Hi, thanks for introducing rtoml to me. Did not see this one before. I am trying to resolve the issue on Mac OS recently and once done I will provide a benchmark. Thanks!

Hey thanks for the reply. Yeah I never expected it will be part of stdlib, as it also requires pybind11 which is not part of stdlib. My aim is to provide an alternative route when perforemance is sensitive.

As for apps that care about toml parsing performance, I worked in a quant fund where toml is used for configuring trading strategies and we have thousands of them and when the app starts it reads in all of the toml configs which take quite a while. This is the mian montive for me to start this project in the first place. I agree this is a quite niche scenario but pytomlpp provides other benefit as well I think – fully compatiable with toml1.0rc1 and stick to only python native types for example.

Thanks!

Hi thanks! Yeah I agree this should not be part of the stdlib, but just thought if I mention it here then when people need a performant toml parser they can discover it!

What’s the easiest way for allowing the user to choose between packages which provide the same interface without code change? Is that possible? Is it a security risk?

This might be Provides-Dist, in theory only though, since there doesn’t seem to be any tool that implements it.

Discussed here, recently:

1 Like

There is now 0.10.1 from May 14, 2020.