PEP 813 - The Pretty Print Protocol (round 2)

At the recently concluded Python core team sprint, @ericvsmith and I worked on some major updates to PEP 813 - The Pretty Print Protocol. We think it’s in pretty[1] good shape and essentially ready to be proposed for Python 3.16. It was a lot of fun clanking away at a collaboration between the two of us, with Eric driving the updated reference implementation and me on the PEP.

Here thusly is the round 2 discussion. Please continue the discussions here; I will ask the moderators to lock the round 1 discussion thread.

The main difference is that we won’t be touching the existing pprint API[2]. Instead we’ll be adding new functionality.

Here’s a summary of the extensive differences from the previous version of the PEP:

  • Add a new default printer, pprint.Printer, and leave pprint.PrettyPrinter unchanged.
  • __pprint__() describes an object’s structure, and the printer makes all layout decisions.
  • Section markers pprint.VALUES, pprint.KWARGS and pprint.ITEMS say how tuples are read, replacing the reading of tuples by their shape and the “false-y” name for positional tuple values.
  • Add pprint.Layout for an object’s chrome, the pprint.DELEGATE marker, and the pprint.Keyword and pprint.Entry types. The markers are PEP 661 sentinels.
  • Specify that the default printer reads the stream permissively, and that other printers may define their own vocabulary.
  • Add pprint.decompose(), and the pprint.LEAF sentinel that it returns for a leaf.
  • Defer a table of markers in the default printer, and a way to change the default printer.
  • Add pprint.DescriptionType for type annotating __pprint__() methods.
  • Add an optional hints attribute to pprint.Layout, which can be used by third-party printers.
  • Add a hug attribute to pprint.Layout, so that defaultdict and similar types keep PrettyPrinter’s layout.
  • A printer returns an iterable of lines, and !p joins them with '\n'.
  • Specify how the default printer lays out, sorts, and limits the depth of nested objects, including keys that wrap.
  • The default printer’s expand option takes a pprint.Expand member: NEVER, FIT or ALWAYS. Its defaults are width=78, indent=4 and expand=Expand.FIT.
  • The default printer’s compact option works with every expand mode.
  • The built-in and standard library types implement __pprint__(). str and bytes describe themselves with the new pprint.BreakableString and pprint.BreakableBytes, and int and float with the new pprint.FormattableNumber, so the printer handles no type natively.
  • The default printer’s number_format option replaces underscore_numbers.
  • Specify when an inherited __pprint__() is used, and explain why.
  • dataclasses and collections.namedtuple generate __pprint__() methods.
  • The !p format spec can be a single replacement field, f'{obj!p:{printer}}', whose value is called without first being converted to a string. Any other spec is a SyntaxError.
  • A printer of None means the default printer, in f-strings, t-strings and PyObject_Pretty().
  • In str.format(), a format spec after !p pads the pretty printed string.
  • T-strings support !p. string.templatelib.Interpolation gets a printer attribute, and string.templatelib.convert handles 'p'.
  • Describe compatibility with Rich, IPython and other pretty printing protocols.
  • Move the Rationale section after the Specification, as PEP 12 now recommends.

  1. pun intended ↩︎

  2. but we are proposing soft-deprecating it ↩︎

13 Likes

Exciting to see how much progress there’s been!

One quick thhought on open-issues:

Names. Layout for the description of an object’s chrome, pprint.DescriptionType for the annotation of a __pprint__() result, … are working names.

I think DescriptionType isn’t as helpful as a Protocol could be

from collections.abc import Iterable
from typing import Protocol


class PrettyPrintable(Protocol):  # or any other name
    def __pprint__(self) -> Iterable[object]: ...


class Bird(PrettyPrintable):
    def __init__(self, *args: object) -> None: ...


bad = Bird()
Show diagnostic
error[call-non-callable]: Cannot instantiate abstract class `Bird`
  --> t.py:13:7
   |
13 | bad = Bird()
   |       ^^^^^^ `__pprint__` is unimplemented
   |
  ::: t.py:6:5
   |
 5 | class PrettyPrintable(Protocol):
 6 |     def __pprint__(self) -> Iterable[object]: ...
   |     --------------------------------------------- `__pprint__` declared as abstract on superclass `PrettyPrintable`
info: `PrettyPrintable.__pprint__` is implicitly abstract because `PrettyPrintable` is a `Protocol` class and `__pprint__` lacks an implementation
 --> t.py:5:7
  |
5 | class PrettyPrintable(Protocol):
  |       ------------------------- `PrettyPrintable` declared here
6 |     def __pprint__(self) -> Iterable[object]: ...
  |


Documentation: https://ty.dev/rules#call-non-callable

__pprint__ as currently described is not something that should have a protocol. It has special inheritance rules similar to __hash__. Hashable is a mess already, we don’t need to implement even more fake protocols.

Also, by design, everything is pretty-printable, so the Protocol would not be useful at the usage site.

To be clear what I mean:

class PrettyPrintable(Protocol):  # or any other name
    def __pprint__(self) -> Iterable[object]: ...


class Bird(PrettyPrintable):
    def __init__(self, *args: object) -> None: ...
    def __repr__(self) -> str: ...
    def __pprint__(self) -> Iterable[object]: ...

class Falcon(Bird):
    def __init__(self, *args: object) -> None: ...
    def __repr__(self) -> str: ...
    # No __pprint__ declaration

The Falcon class doesn’t have a __pprint__ that will be used, so it matching the protocol is confusing.

Note that I believe that the semantics for how inhereitance works here are well justified, same as they are well justified for __hash__, it just should be made clear that there are special rules here, and providing a protocol would signal the exact wrong thing. [1]


  1. if someone cares: Hashable should be a special form, not a Protocol, but that’s off-topic. And I don’t think a similar special form would be justified for PrettyPrintable ↩︎

I think the PEP here uses “protocol” in its general Python sense, it’s not meant to imply a typing.Protocol.

3 Likes

Correct. It’s the same sense as in “the repr protocol”, or “the __format__ protocol”.

Plus, “Protocol” provides a plucky progression of Pythonic patterns, pragmatic practices, and powerfully polymorphic programming paradigms.

14 Likes

Thanks for the update, this looks promising. Please could you update the reference implementation to match round 2?

Modern defaults

This PEP mostly obsoletes my python/cpython#149189 to modernise the defaults. The PEP says:

Three of its defaults differ from PrettyPrinter’s: width is 78, indent is 4, and expand is Expand.FIT. So by default its output stays within the line length that PEP 8 recommends, and comes close to code formatted by tools such as ruff and black.

So:

argument Current PEP 813 (round 2) #149189 Match
expand False Expand.FIT (same as True) True :white_check_mark:
indent 1 4 4 :white_check_mark:
width 80 78 88 :cross_mark:

Width

PEP 8 actually recommends 79 not 78.

And PEP 8 is explicitly the style guide for the standard library. The rest of the world can follow it if they wish, and often do, but are free to deviate. And line length is a common deviation.

Indeed, Black and Ruff default to 88:

Black defaults to 88 characters per line, which happens to be 10% over 80. This number was found to produce significantly shorter files than sticking with 80 (the most popular), or even 79 (used by the standard library). In general, 90-ish seems like the wise choice.

It would be nice if this new API starts with modern defaults rather than (a smaller value than) PEP 8’s 2001 choice.

4 Likes

Ah, thanks for the reminder. I’m closing that PR in favor of @ericvsmith 's as-yet-un-PR’d reference implementation branch.

I’ll chat about it with Eric. 88 is fairly arbitrary so it would only be useful to be compatible with black and ruff. I personally use 100 in code I control, which I guess is “90-ish”-ish. If we do go with PEP 8 compatibility, then yeah we should switch to 79.

1 Like

Just create a PR from the branch please. It’s much easier to review or try out that way.

I like the separation of description and presentation, but why does __pprint__ need to yield a mixture of values, markers, and layout instructions? This seems unnecessarily complicated.

Why not return an object derived from a common pprint.Description base class, with subclasses for sequences and mappings? Each would contain an iterable (which could still be a generator) and the relevant Layout fields.

For example:

def __pprint__(self) -> pprint.Mapping:
    return pprint.Mapping(
        self.items(),
        open="{",
        close="}",
        sortable=True,
    )

This would be easier to type-check, and printers could dispatch on the description type rather than interpret a stateful stream. Mistakes would also be easier to catch instead of silently producing unexpected output.

It would also be easier to extend the protocol by adding attributes or subclasses. Otherwise, aren’t we risking the same kind of versioning mess that pickle has accumulated over the years?

4 Likes

From the open issues:

Whether to add a built-in way to pretty print a value without an f-string. The choices include:

  • A pretty argument to the print() built-in function, as an earlier draft of this PEP proposed. print(obj, pretty=True) would use the default printer, and print(obj, pretty=printer) the given printer, for each of the positional arguments.

  • A new pprint() built-in function. Its name would be shadowed by the common from pprint import pprint, which uses PrettyPrinter rather than the default printer, so the two would print the same object differently.

Yes please!

I’d prefer the second choice. Alternatively, something that can be conveniently imported from pprint.

One of the main reasons I had for python/cpython#149189 was that pprint is often used for debugging because it’s always within arm’s reach: just import pprint as pprint and pprint(my_var).

It’s pretty awkward to type say, print(f"{my_var!p}"), and I’ve managed to get it wrong a couple of times playing around with the reference implementation, which is not a big deal but frustrating. You don’t want to fiddle around when debugging: something like pprint(my_var) would be so much easier.

4 Likes

Thanks for the suggestion Neil. The main reason I’m resisting this is that many printed representations are made of a mix of layouts. Think about a class constructor, which pretty printing often wants to mimic. You start with a sequence of positional arguments, followed by a list of keyword arguments. Or think about representing a defaultdict, where the first argument to the constructor is the default factory, and that’s followed by the mapping’s items[1].

Your example annotates __pprint__() as returning a pprint.Mapping, but neither of the above example representations is a mapping or a sequence. Sure, you could define even more complex description subclasses, or composable ones, but I think at that point the __pprint__() implementation just gets too complicated, as does the return type annotation you’d have to give it. We want to keep the implementation of __pprint__() methods as simple and straightforward as possible.

In a sense, the structured form you want is available from the pprint.decompose() function proposed in the PEP. Printers call it to find an object’s __pprint__() and normalize its stream into a (Layout, items) tuple, where each item is a plain value, a Keyword, or an Entry. So printers don’t interpret a stateful stream directly. They can match on the result of decompose() instead.

Putting a meaningful type annotation on __pprint__() methods is definitely an issue. Since this is primarily a debugging interface, where a malformed description shows up right away in the output, I don’t think it will be a major problem in practice.

On versioning: descriptions are produced and consumed in the same process and never persisted, so I don’t think pickle’s problems carry over. Layout can gain attributes, hints carries anything printer-specific, and new sentinels can be added the same way new classes would be.


  1. the signature is a little more complex, but that’s enough of an explanation to get the gist ↩︎

1 Like

I do like the idea of having pretty printing right at your finger tips. It makes me think of “print debugging” which I still think is a useful technique for “artisanal code”[1].

Hesitations are a) the shadowing of pprint.pprint() which I also currently use as from pprint import pprint and b) the similarity of print() and pprint(). Maybe neither of those are a problem, and I agree that having a separate function is probably better than adding a pretty optional keyword argument to built-in print().

I appreciate the vote for built-in pprint() and the feedback on the experience with !p and f-strings. Maybe that’s enough to move built-in pprint() from Open Issue to proposed.


  1. a term I first heard from @savannahostrowski and which I think is a wonderful counterpart to “clanker code” ↩︎

4 Likes

Sorry for the delay. I’ll catch up on this thread and create a PR this weekend.

Thanks, Barry. You’re right that separate Mapping and Sequence descriptions can’t express mixed layouts. But I think the mixed cases are really nested structures. defaultdict(<class 'list'>, {'a': [1]}) is a call whose second argument is a mapping, and nesting composes naturally if __pprint__() returns typed descriptions instead of yielding a marker stream:

class defaultdict:
    def __pprint__(self) -> pprint.Description:
        return pprint.Composite(
            [self.default_factory, dict.__pprint__(self)],
            hug=True,
        )

class UserDict:
    def __pprint__(self) -> pprint.Description:
        return pprint.Delegate(self.data)

class Bass:
    def __pprint__(self) -> pprint.Description:
        return pprint.Composite([
            self._strings,
            pprint.Keyword('pickups', self._pickups),
            pprint.Keyword('active', self._active, False),
        ])

Composite takes the items plus the current Layout fields as keyword arguments (open, close, sortable, hug, hints and so on), so a separate Layout isn’t needed. Here defaultdict reuses dict’s own description with no copying, and items can still be a generator. This is essentially what decompose() already returns. I’m suggesting authors return that structure directly.

Why I think this is better:

  • It’s type-checkable. Type checkers can verify a description’s shape, and printers can match on its type. Mistakes fail loudly instead of silently rendering as something else.
  • There’s no marker-then-data pattern. A sentinel followed by the data it governs is an idiosyncratic way to encode what Python normally expresses as an object, where the type is the marker and the attributes are the data. The positional rules go away too. Today Layout and DELEGATE count only as the first item, and a 4-tuple under KWARGS silently prints as a tuple.
  • Descriptions compose. A stream can’t embed another object’s description, chrome included, without more markers. Typed descriptions nest the same way the data does.
  • It’s consistent with the rest of the PEP. Keyword, Entry, BreakableString and FormattableNumber are already small typed objects. This applies the same design to the whole description.
  • There’s no parse step. Authors and printers share one vocabulary, and decompose() becomes trivial.

On versioning, my concern isn’t persistence but extensibility. Suppose a future version wants a keyword item with an extra feature, say a hint. With sentinels, that’s a new marker, and every existing printer prints it as a stray value, because there’s no way to say “KEYWORD_V2 is like KEYWORD, but with more.” With types, it’s just class HintedKeyword(Keyword). Existing printers that match on Keyword keep working and ignore the new attribute, and newer printers use it. Inheritance gives you backward-compatible extension for free.

The PEP rejects return types other than plain tuples so that authors don’t have to define a new class. This design doesn’t require that either: authors only instantiate classes that pprint provides, and Keyword and Entry already exist. The remaining cost is writing Keyword('pickups', x) instead of yield 'pickups', x after KWARGS, which seems small for what we gain.

3 Likes

I marked this as draft because the last thing I did at the sprint was have Codex convert some code from Python to C, and I still need to review it.