This ultimately goes to PEP compliance. In the PR I worked on, I typed in the literal escape sequence, a reviewer asked me to remove it. I pointed back to the PEP, and then the question of whether it was necessary came up and whether different checkers would not interpret the package as partially typed if we didn’t follow the PEP strictly.
I wanted to know if type checkers in general were supposed to rely on the newline somehow (stranger things have happened; e.g. say the checker later needs to append to this file prior to processing and not having a new line would cause an error). In this case, for now, the requirement seems to be convention only, but I would want to know if there was a pragmatic reason for the requirement.
I could dive through the source of all type checkers and see how they decide to implement this rule, but it’s significantly less work to point to the PEP as the single source of truth.
For what it’s worth this is not a hypothetical. In this PR (already linked earlier) we merged a py.typed file with the wrong interpretation of the PEP’s wording. It’s been wrong for more than a year until someone finally pointed out the issue. At least 3 people (PR author and two reviewers) found the wrong interpretation reasonable. This might explain our interest in this trivial-looking point.
But I believe it’s all cleared up, thank you for the feedback.
It’s not ”wrong” @adeak . As pointed out, mypy does not follow strict adherence to the PEP.
But, to your point, if we have to look to the source of every type checker (mypy, pyre, pyright, etc.) to see how or if they implement this PEP, then what’s the point of the PEP? How am I as an end user to determine which parts of a PEP should be taken very literally and which are “loose”? If a package claims compliance to a PEP, but we can pick and choose what to follow, compliance means nothing to the end user since we cannot rely on any consistent behavior.
I am sure the author never meant for this much discrepancy over such a seemingly small issue, but @adeak is right. It can cause silent failure if we expect to take the PEP at its word.
PEPs in general don’t get changed once accepted. I do agree that the wording in the PEP could be interpreted wrongly (my assumption would be that it means “there should be a line that just contains partial” but it’s worded a bit oddly).
Normally, the right thing to do would be to propose a specific change to the actual documentation of the standard that the PEP proposed[1], but I’m not sure where typing standards are documented (I’m more familiar with packaging ones, which are at packaging.python.org).
If there’s no canonical documentation of this feature outside of the PEP, then I’m not sure what the typing community’s stance is on modifying finalised PEPs.
But either way, I think the intention here is pretty clear at this point. Let’s not make too much of a big deal over a minor imprecision in how it was described - people are human and make mistakes, after all.
On the presumption that it’s just clarifying the wording, not changing the intent ↩︎
@adeak Actually, per the mypy implementation, the original with the \n escape sequence DOES work because strip() will remove all whitespace, including whitespace from escape sequences. We can talk more on the PR, but having \n does not cause anything to break.
Postel’s law is actually considered to have harmful consequences on protocols, and no longer considered by the ietf to be best practice, especially now that the expectation is that broken things being rejected will cause them to be updated. RFC 9413 - Maintaining Robust Protocols
@guido It’s actually been made clear to me that using partial\n is not recognized by mypy:
If stub_typed_file contains partial\n, fscache.read(stub_typed_file).decode().strip() returns 'partial\\n' and it is not recognized as Partial typed package because 'partial\\n' == 'partial' returns False.
So indeed, as silly as it seems, the PEP is open to misinterpretation.
You (or I, I guess) seem very confused between the characters “backslash” followed by “n” and “newline”.
The review comment you linked to includes a literal backslash and a literal “n” character. That’s clearly wrong, and (IMO) obviously not the intent of the PEP.
Definition of Terms
The definition of “MAY”, “MUST”, and “SHOULD”, and “SHOULD NOT” are to be interpreted as described in RFC 2119.
Partial Stub Packages
If a stub package distribution is partial it MUST include partial\n in a py.typed file.
Per RFC 2119
MUST This word, or the terms “REQUIRED” or “SHALL”, mean that the
definition is an absolute requirement of the specification.
It is extremely odd the PEP explicitly has \n in it, which is why I deliberately wrote it into the py.typed file. If the PEP requires the file follow POSIX convention, it should be stated thus:
If a stub package distribution is partial it MUST include partial\n in a py.typed file and the py.typed file must end with a newline.
It is not unreasonable to think the py.typed file must have exactly partial\n in it if reading the rules and following them.
Quite frankly, following POSIX convention is kind of an “okay, ya, no duh” thing. I don’t think it needs to be stated in the PEP, but if desired for completeness, it should be written clearly (i.e. state to “use POSIX convention”). This part of the PEP is not written clearly as adding partial\n literally in the py.typed file is a perfectly valid reading of the requirements of the PEP, but it is not what the author intended and is not respected by, at least, mypy.
I think most pythonistas auto-escape \n whenever they read it, and so the idea of reading that literally seems pretty unreasonable. But this convention isn’t explicitly stated anywhere, so it is possible.
It seems like a meta issue here is that this isn’t documented anywhere besides the PEP, and the PEP is considered frozen. Whether or not this is a reasonable issue with the PEP, that seems like a bad way to document a standard in general. Things need to be clarified sometimes.
PEP 561 already has five updates since its last post in the post history. Two of them in the last two years, which is definitely after it was accepted.
How about someone just send the PR to clarify that \n in this case means a newline character and not a literal \ and n. Or remove the \n and add text saying that partial should be on a line by itself. There’s no indication in the document that it should be interpreted as an escape here, and assuming that is quite a stretch, but we know the author and can reasonably assume that was the intent.
But for people who primarily use Python, this hasn’t made a difference in a long time. You don’t need to worry about that when you’re writing code. So you see a \n in a string and you just think “ah a newline”.
I’m not saying your request for clarification is wrong, that’s just why I think some people view the current language as obvious.
Hi! Original PEP author here! To clear up any confusion, the text under discussion was written with Python universal newlines in mind. In other words, “partial” followed by a newline character, either “\r\n” or “\n” (or I suppose even “\r” if that’s still supported). Sorry if the wording was unclear!