PEP 561 Clarification Regarding `\n`

For what it’s worth this is not a hypothetical. In this PR (already linked earlier) we merged a py.typed file with the wrong interpretation of the PEP’s wording. It’s been wrong for more than a year until someone finally pointed out the issue. At least 3 people (PR author and two reviewers) found the wrong interpretation reasonable. This might explain our interest in this trivial-looking point.

But I believe it’s all cleared up, thank you for the feedback.

1 Like

It’s not ”wrong” @adeak . As pointed out, mypy does not follow strict adherence to the PEP.

But, to your point, if we have to look to the source of every type checker (mypy, pyre, pyright, etc.) to see how or if they implement this PEP, then what’s the point of the PEP? How am I as an end user to determine which parts of a PEP should be taken very literally and which are “loose”? If a package claims compliance to a PEP, but we can pick and choose what to follow, compliance means nothing to the end user since we cannot rely on any consistent behavior.

I am sure the author never meant for this much discrepancy over such a seemingly small issue, but @adeak is right. It can cause silent failure if we expect to take the PEP at its word.

PEPs in general don’t get changed once accepted. I do agree that the wording in the PEP could be interpreted wrongly (my assumption would be that it means “there should be a line that just contains partial” but it’s worded a bit oddly).

Normally, the right thing to do would be to propose a specific change to the actual documentation of the standard that the PEP proposed[1], but I’m not sure where typing standards are documented (I’m more familiar with packaging ones, which are at packaging.python.org).

If there’s no canonical documentation of this feature outside of the PEP, then I’m not sure what the typing community’s stance is on modifying finalised PEPs.

But either way, I think the intention here is pretty clear at this point. Let’s not make too much of a big deal over a minor imprecision in how it was described - people are human and make mistakes, after all.


  1. On the presumption that it’s just clarifying the wording, not changing the intent ↩︎

2 Likes

@adeak Actually, per the mypy implementation, the original with the \n escape sequence DOES work because strip() will remove all whitespace, including whitespace from escape sequences. We can talk more on the PR, but having \n does not cause anything to break.

Some text seems to be here, though partial isn’t mentioned: Typing Python Libraries — typing documentation

cc: @erictraut & @hauntsaninja as the main authors of that bit of the document.

A

Postel’s law is actually considered to have harmful consequences on protocols, and no longer considered by the ietf to be best practice, especially now that the expectation is that broken things being rejected will cause them to be updated. RFC 9413 - Maintaining Robust Protocols

4 Likes

@guido It’s actually been made clear to me that using partial\n is not recognized by mypy:

If stub_typed_file contains partial\n, fscache.read(stub_typed_file).decode().strip() returns 'partial\\n' and it is not recognized as Partial typed package because 'partial\\n' == 'partial' returns False.

So indeed, as silly as it seems, the PEP is open to misinterpretation.

I was wrong. Apologies @adeak

1 Like

You (or I, I guess) seem very confused between the characters “backslash” followed by “n” and “newline”.

The review comment you linked to includes a literal backslash and a literal “n” character. That’s clearly wrong, and (IMO) obviously not the intent of the PEP.

5 Likes

Per the PEP:

Definition of Terms
The definition of “MAY”, “MUST”, and “SHOULD”, and “SHOULD NOT” are to be interpreted as described in RFC 2119.

Partial Stub Packages
If a stub package distribution is partial it MUST include partial\n in a py.typed file.

Per RFC 2119

MUST This word, or the terms “REQUIRED” or “SHALL”, mean that the
definition is an absolute requirement of the specification.

It is extremely odd the PEP explicitly has \n in it, which is why I deliberately wrote it into the py.typed file. If the PEP requires the file follow POSIX convention, it should be stated thus:

If a stub package distribution is partial it MUST include partial\n in a py.typed file and the py.typed file must end with a newline.

It is not unreasonable to think the py.typed file must have exactly partial\n in it if reading the rules and following them.

Quite frankly, following POSIX convention is kind of an “okay, ya, no duh” thing. I don’t think it needs to be stated in the PEP, but if desired for completeness, it should be written clearly (i.e. state to “use POSIX convention”). This part of the PEP is not written clearly as adding partial\n literally in the py.typed file is a perfectly valid reading of the requirements of the PEP, but it is not what the author intended and is not respected by, at least, mypy.

I think most pythonistas auto-escape \n whenever they read it, and so the idea of reading that literally seems pretty unreasonable. But this convention isn’t explicitly stated anywhere, so it is possible.

It seems like a meta issue here is that this isn’t documented anywhere besides the PEP, and the PEP is considered frozen. Whether or not this is a reasonable issue with the PEP, that seems like a bad way to document a standard in general. Things need to be clarified sometimes.

PEP 561 already has five updates since its last post in the post history. Two of them in the last two years, which is definitely after it was accepted.

How about someone just send the PR to clarify that \n in this case means a newline character and not a literal \ and n. Or remove the \n and add text saying that partial should be on a line by itself. There’s no indication in the document that it should be interpreted as an escape here, and assuming that is quite a stretch, but we know the author and can reasonably assume that was the intent.

5 Likes

pythonistas or Linux users? What about the Windows community?

I’m on it. I’m assuming it would be to GitHub - python/peps: Python Enhancement Proposals, right?

But for people who primarily use Python, this hasn’t made a difference in a long time. You don’t need to worry about that when you’re writing code. So you see a \n in a string and you just think “ah a newline”.

I’m not saying your request for clarification is wrong, that’s just why I think some people view the current language as obvious.

Please see PEP 561: Clarify `\n` by adam-grant-hendry · Pull Request #3428 · python/peps · GitHub

Hi! Original PEP author here! To clear up any confusion, the text under discussion was written with Python universal newlines in mind. In other words, “partial” followed by a newline character, either “\r\n” or “\n” (or I suppose even “\r” if that’s still supported). Sorry if the wording was unclear!

7 Likes

@ethanhs Was that to follow POSIX convention, or to serve another purpose?

And thank you for clarifying!

As he said, it’s to follow Python convention.