TOML development has stalled - implications for pyproject.toml and a way forward?

It looks like TOML development has stalled due to the creator of TOML having moved on to other projects without giving anyone else sufficient rights in the toml-lang organisation on GitHub to add those who have kindly offered to volunteer as maintainers.

Some developers (eg. the author of flake8) have been reluctant to embrace pyproject.toml due to syntax issues which looked likely to be addressed in TOML 1.1. However unless something changes, it seems unlikely that TOML 1.1 will ever see the light of day given the lack of resources to go through the pending PRs.

What does it mean for pyproject.toml if TOML never moves beyond 1.0? If the owner of toml-org remains unresponsive, should the TOML specification repository be forked and if so to where? Should the development of TOML, given it was (as far as I know) popularised by its usage in pyproject.toml, be moved under an appropriate Python packaging group eg. PyPA?

1 Like

I guess it depends on what’s actually missing… JSON hasn’t changed ever, and no-one worries about that. tbh I kind of had the impression that TOML was like… done. Can you (or someone) put together some list of what the big motivating features for TOML 1.1 would be?

7 Likes

I think one of the main improvements is to Allow newlines and trailing comma in inline tables which solves difficulties around nesting. I wrote an example of what that helps with here.

1 Like

There is a list of open issues to consider for TOML 1.1.

(I had to write a separate comment because only 2 links are allowed in one comment)

1 Like

Huh, skimming over those, I think the situation is that Pradyun actually can do a 1.1 release; it’s just been bottlenecked on him finding the time? If the only issue is he doesn’t have the admin bit to rearrange repos or add new maintainers, that’s an issue on some time-frame, but probably not urgent and probably something where we can let him decide how he wants to handle it?

1 Like

I don’t see how time can solve @pradyunsg’s lack of admin permissions to add new maintainers since it requires the owner of toml-org to make changes. I also think that it’s rather unfair to expect one person to handle all the maintenance alone particularly if they have other important projects to look after. Ideally the toml-org owner would respond, but it appears he has moved on and isn’t responding.

I’m not saying we should make him handle all the maintenance alone, just that we can wait for him to figure out what help he needs and ask for it, instead of speculating about it like he’s not here :slight_smile:

1 Like

Fair enough. Thanks for your advice.

Yup! This is pretty much it.

2 Likes

One thing I think it’s important to mention is that while JSON may not have changed for a while, it certainly had a number of releases before it reached a state of completion. Similarly with YAML 1.2 (YAML™ Specification Index). I think it is reasonable that TOML requires further releases before it is complete.

“JSON was first presented to the world at the JSON.org website in 2001. A definition of the JSON syntax was subsequently published as IETF RFC 4627 in July 2006. ECMA-262, Fifth Edition (2009) included a normative specification of the JSON grammar. This specification, ECMA-404, replaces those earlier definitions of the JSON syntax. Concurrently, the IETF published RFC 7158/7159 and in 2017 RFC 8259 as updates to RFC 4627.” (https://www.ecma-international.org/wp-content/uploads/ECMA-404_2nd_edition_december_2017.pdf)

It’s also worth mentioning that some people still do think it would be better to make more changes (such as allowing trailing commas). The fact that no one changed a format to make it better doesn’t mean it’s the best it can be, any more than the fact that no one repaired a run-down house means it’s in pristine condition.

1 Like

Interesting analogy. I wasn’t aware time and the elements could wear out digital stuff.

The fact that no one changed the TOML spec recently suggests either the author either thought the paid of further change want worth the cost, or tired of the project and moved onto other things. Changing it once stabilized (and presumably available in multiple environments) might not be worth the compatibility problems.

I’m of the same mind. With a toml parser now being part of the Python stdlib, we’re looking at ~5 year period before new features can be adopted widely.

Well, if you prefer it can be “just because someone built half a house and stopped before finishing the roof and floor doesn’t mean it’s habitable”. :wink: I just mean that just because no one’s fixing something doesn’t mean it ain’t broke.

1 Like

Here is an example of what would be improved by TOML 1.1. This is what an example configuration looks like in YAML:

tool:
  pydoc-markdown:
    renderer:
      type: mkdocs
      mkdocs_config:
        site_name: HDX Python Scraper
      pages:
        - title: Home
        - title: API Documentation
          children:
            - title: Source Readers
              contents:
                - hdx.scraper.readers.*
            - title: Outputs
              contents:
                - hdx.scraper.jsonoutput.*
                - hdx.scraper.googlesheets.*
                - hdx.scraper.exceloutput.*

This is what it looks like in TOML 1.0:

[tool.pydoc-markdown.renderer]
type = "mkdocs"

[tool.pydoc-markdown.renderer.mkdocs_config]
site_name = "HDX Python Scraper"

[[tool.pydoc-markdown.renderer.pages]]
title = "Home"

[[tool.pydoc-markdown.renderer.pages]]
title = "API Documentation"

[[tool.pydoc-markdown.renderer.pages.children]]
title = "Source Readers"
contents = ["hdx.scraper.readers.*"]

[[tool.pydoc-markdown.renderer.pages.children]]
title = "Outputs"
contents = ["hdx.scraper.jsonoutput.*", "hdx.scraper.googlesheets.*", "hdx.scraper.exceloutput.*"]

TOML 1.1 would allow something much more readable like this:

[tool.pydoc-markdown.renderer]
type = "mkdocs"
mkdocs_config = {
  site_name = "HDX Python Scraper"
}

pages = [
  { title = "Home" },
  {
    title = "API Documentation",
    children = [
      {
        title = "Source Readers",
        contents = [
          "hdx.scraper.readers.*",
        ]
      },
      {
        title = "Outputs",
        contents = [
          "hdx.scraper.jsonoutput.*",
          "hdx.scraper.googlesheets.*",
          "hdx.scraper.exceloutput.*",
        ]
      },
    ]
  },
]
1 Like

Building further on that analogy, I’d like to point out that my family house was never “finished” per the original drawing. That doesn’t mean it’s uninhabitable — on the contrary, I’m grateful that it was never completed because it would be a complete overkill.

I’m not saying that TOML is perfect but I’m not convinced that making major changes for what I understand it’s mostly convenience is worth breaking backwards compatibility. Two important points are:

  1. We’ve “just” managed to get tomllib in, and some projects still rely on the ancient unmaintained toml library. TOML 1.1 would mean that many projects suddenly need to switch back to an external implementation, and then start using tomllib again in a future version of Python.
  2. Versions are hard. People are likely to end up using TOML 1.1 feature unknowingly and breaking stuff that still uses a TOML 1.0 parser.
2 Likes

@mcarans I’m not sure what you’re concerned about here and why you’re signal boosting this thread in multiple places now.

Then pyproject.toml won’t evolve beyond TOML 1.0. I don’t think TOML 1.0 is severly lacking as many speculate/opine – it has limitations and tradeoffs but that’s literally every technology ever and it’s fine for digital infrastructure to not evolve stupidly quickly anyway.

Maybe. This forum isn’t the right place to discuss that.

No. TOML is not a PyPA or Python packaging thing – it’s a language spec that is maintained by a completely separate group. It is a configuration file format, and isn’t tied to PyPA or (for that matter) any specific programming language).

5 Likes

There is json5, which is one groups attempt at a new iteration of JSON.

That being said, I think TOML in it’s current iteration is fine for our use cases. Maybe there’s some things that could be better, and if so hopefully it continues to improve, but if it doesn’t, then I think it’s still perfectly adequate.

4 Likes

I only raised it in one place - on the issue about finding additional maintainers for TOML where I thought some watching that issue might be interested. I understand you’d prefer not to turn it into a bigger discussion.

I think that could have been an argument for using a well established existing format for pyproject rather than a less well known still evolving format, but anyway I really hope the improvements make it into TOML and then into pyproject.toml. Thanks for making the milestone which makes it much easier to track what is going on.

2 Likes

I don’t think it’s particularly useful to re-litigate a decision made in 2016, unless we think that the current situation is problematic enough to warrant the ecosystem wide pain of migration [1]. However, I thought maybe it would be fun to poke at the question a bit and see how well that decision held up.

At the time the only really serious contenders we had were TOML, JSON, configparser flavored INI, YAML, and Python Literals (via ast.literal_eval()), and most of them were reviewed by @njs, and out of that review it became obvious that all of the existing formats had serious short comings, and the state of TOML, even in 2016 at v0.4.0, was in a much better position for our use cases.

AFAIK, TOML is the only one of those, since 2016, to have evolved at all, so all of the shortcomings that existed in 2016 for those other formats still exist, but TOML has evolved to get even better in the intervening 7 years.

It is possible that TOML will now end up in the same state that the other options at the time were in, and end up effectively frozen and in a “finished” state… and that’s just fine? Maybe there are some situations where a better format could be more readable, but we haven’t found that in the packaging space yet, and if it’s bad enough, then those projects don’t have to use pyproject.toml and can use their own file in whatever format they prefer.

Hopefully that doesn’t happen, and TOML continues to evolve to support more use cases in clean ways, but I think, even if we were remaking this decision today, given the options we had then, I think we would still make the same decision.


  1. Spoiler alert, I don’t. ↩︎

5 Likes