Adopting/recommending a toml parser?

If pure speed is what one is looking for there’s https://github.com/samuelcolvin/rtoml too. I tried to compare pytomlpp to it, but did not manage to make pytomlpp work on MacOs.

Interesting, but toml++ uses C++17 – an extremely recent version of the C++ language – , while PEP 7 mandates the use of “C89 with several select C99 features”. So I’m afraid integrating pytomlpp into the stdlib would be a no-no, and would remain so for at least the next 10 years.

As for speed, well, the dominant use case for TOML is configuration files, and I don’t think those are usually performance-critical (unless you’re batch-processing thousands of them, perhaps :slight_smile: ).

3 Likes

Hi, thanks for introducing rtoml to me. Did not see this one before. I am trying to resolve the issue on Mac OS recently and once done I will provide a benchmark. Thanks!

Hey thanks for the reply. Yeah I never expected it will be part of stdlib, as it also requires pybind11 which is not part of stdlib. My aim is to provide an alternative route when perforemance is sensitive.

As for apps that care about toml parsing performance, I worked in a quant fund where toml is used for configuring trading strategies and we have thousands of them and when the app starts it reads in all of the toml configs which take quite a while. This is the mian montive for me to start this project in the first place. I agree this is a quite niche scenario but pytomlpp provides other benefit as well I think – fully compatiable with toml1.0rc1 and stick to only python native types for example.

Thanks!

Hi thanks! Yeah I agree this should not be part of the stdlib, but just thought if I mention it here then when people need a performant toml parser they can discover it!

What’s the easiest way for allowing the user to choose between packages which provide the same interface without code change? Is that possible? Is it a security risk?

This might be Provides-Dist, in theory only though, since there doesn’t seem to be any tool that implements it.

Discussed here, recently:

1 Like

There is now 0.10.1 from May 14, 2020.

It seems that date is getting close. As of October 7, there is a pre-release candidate (1.0.0-rc.3) for TOML.

Again, as the time approaches… I’m looking forward to seeing what @brettcannon is thinking on the topic of the standard library. While “batteries included” served us well for many years, we’re beyond it being an advantage now.

I would also like to here what @brettcannon thinks on the topic of the standard library. :wink: (Seriously, it fluctuates so I don’t even know where I’m going to land after the topic is discussed.)

4 Likes

I agree with this sentiment in general, but it really is hard for stuff like setuptools and pip to take on dependencies, which is why they vendor everything. Considering the universal nature of pyproject.toml, it would be very nice to be able to parse these things without taking on a dependency.

Of course, it’s not likely that we’ll drop support for Python 3.9 for many years from now, and maybe by then the need for pip and setuptools to vendor all their dependencies will have been considerably alleviated, so we’ll gain very little advantage to it.

That said, the other time I find it convenient to work without any dependencies are for little scripts where I essentially replace what would have been a bash script with a Python script — I tend to use only the standard library for this sort of thing, since it’s inconvenient to declare dependencies and build a virtualenv for single standalone .py files. I can imagine a TOML parser / emitter will increasingly be valuable for these sorts of things as well, so I’m +1 on including it, even from someone who is mostly in the “kernel / minimal python” camp.

10 Likes

I think there are still real advantages to having some batteries included, and a file format which we’re using in standards for packaging seems like an excellent candidate. Installable packages play a much bigger role now, but I hope the core developers don’t decide nothing new can be added to the standard library - that would be throwing the baby out with the bathwater.

5 Likes

I didn’t mean that there should be no batteries at all. I’m wary of the notion that “best of breed” libraries should be adopted into the standard library, though.

The bootstrapping requirement for package installation tools (as opposed to build tools & other development tools) is a significant concern, and should guide us in deciding what goes into a minimal installation.

I do believe we should keep in mind that bootstrapping a tool like pip isn’t the same as making all the batteries it depends on part of the standard library, though. A pip that gets packaged with Python does not have to share the pieces that it would otherwise vendor.

Of course, I may have some unusual views on the separation of the interpreter and applications built using the interpreter.

-Fred

1 Like

The TOML specification has now reached version 1.0!

I hope it will be added to the standard library of Python 3.10.

3 Likes

Need to have a (probably massive) discussion about the future of the stdlib, then I will work on getting TOML into the stdlib somehow if it still makes sense.

2 Likes

The second part of the sentence seems to spill the beans about the outcome of the massive discussion :wink:

1 Like

:laughing: I’m actually not going into that discussion with any preconceived notions. I should rephrase it to be more like, “if it still makes sense to pull in a TOML module, I will work on that.”

1 Like

Just to metion rtoml (implemented in Rust), which had a v0.6.1 release 14 days ago.

A Rust implementation might be hard to take since it would not only be a burden on alternative implementations, but that there’s currently no Rust in CPython.

1 Like