Perhaps they should be? With so many people trying to contribute to Python simultaneously, taking each individual contribution in isolation is a recipe for becoming C++ conflicting or inconsistent outcomes.
Nobody has shown that the new function would be any less convoluted (modulo the deprecated-but-not-removed arguments) than the current one. The only proposal that’s been made is we get to give different TypeErrors from each one if the user changes the argument without also changing the function.
But probably nobody has shown it because it’s not related to this PEP, so that’s a good thing - we’re keeping somewhat on topic.
I didn’t see this quote, and certainly wasn’t commenting on it. Though my interpretation here would be that the OS APIs are the pig and subprocess is the lipstick (hence my surprise that the lipstick didn’t manage to hide the different breeds of pig beneath it).
to be pretty convoluted, and explaining why that’s the case to be likely to result in extremely convoluted documentation.
For anyone who hasn’t checked the subprocess docs, the former runs an executable called some command with spaces, whereas the latter runs the command some with arguments ["command", "with", "spaces"].
On POSIX. yes. On Windows they both run some with three arguments. And the fact that it’s different is bad, but adding t-string support here doesn’t actually make the inconsistency worse.
Arguably we could deprecate str-with-spaces on POSIX and change it to imply shlex.split, and then we’d have consistency again.
That’s inconsistent with the documentation, which says:
If passing a single string, either shell must be True (see below) or else the string must simply name the program to be executed without specifying any arguments.
All I was trying to do was point out that I had claimed the new proposal was more convoluted than the existing behaviour. I don’t think you can claim that undocumented behaviour makes the existing behaviour more convoluted, or changing the existing semantics to make things more consistent makes the proposed new behaviour less convoluted.
It’s entirely on topic to flag potential issues with the proposal, and I consider adding yet more complexity to an already complex and easily misunderstood[1] API to be very much a potential issue.
Streamlining the existing API is a useful improvement to consider, and if we did do that, we may well end up with a cleaner basis to add t-string support to, but I’m happy to accept that’s off-topic here, as long as it’s clear that putting that proposal off topic doesn’t somehow magically mean that the existing API is any better as a basis for trying to add t-string support.
as evidenced by the confusion over the case of a single string argument with shell=False↩︎
The documentation for the args parameter to Popen constructor also says “If args is a string, the interpretation is platform-dependent and described below”, so nobody is winning any consistency arguments based on our own docs If the docs were updated to reflect reality, then that earlier section you quoted would be changed.
That’s what I said, yeah. The “nobody has shown it” I was referring to was a complete new API design that somehow has fewer arguments than Popen. At best, it’d end up like the .NET API where you assign a whole lot of attributes to an object rather than (optionally) passing them to a constructor - this is just a complicated task with a huge amount of variation (the current implementation still isn’t sufficient to implement, e.g., the py launcher). I’m highly skeptical that a new API would make the simple cases simpler without excluding other important cases.
plumbum’s approach does indeed look like it would produce quite easy to read code. Even it ends up suffering from an(“over-abundance”, “of”, “explicit”, “quotation”, “marks”, when, “writing”, out, “long”, commands, though).
The suggestion affirms for me though that the most useful core idea the PEP can offer is an as-general-as-we-can-make-it algorithm that:
accepts a template string
quotes all of the input fields
emits the list form of the resulting command invocation (since that is the form that can be safely passed to the widest variety of platform subprocess APIs)
Approaching it that way may also help separate any discussion of the existing cross platform differences in the way list inputs are handled.
No quoting required (step 2), as the list form will get platform-specific quoting when you use it. If the result is a list, then the t-string here is just about the rules for splitting it up into argument-sized chunks.
Unfortunately, splitting without somehow quoting the interpolated fields first will mess up file names that contain whitespace. Some examples (with the incorrect result that trying them with f-strings instead would give you):
fname = "filename with spaces"; shlex.split(t"ls {fname}") should give ["ls", "filename with spaces"], NOT ["ls", "filename", "with", "spaces"]
dirname="some dir"; fname="some file"; shlex.split(t"ls {dirname}/{fname}") should give ["ls", "some dir/some file"], NOT ["ls", "some", "dir/some", "file"]
The key benefit of considering the problem at the shlex.split level, though, is that we’re not escaping for arbitrary shells any more, we’re only escaping for our own shlex.lex algorithm.
For POSIX mode splitting, shlex.quote is already up to that task:
>>> fname="filename with spaces"
>>> shlex.split(f"ls {fname}")
['ls', 'filename', 'with', 'spaces']
>>> shlex.split(f"ls {shlex.quote(fname)}")
['ls', 'filename with spaces']
d = "dir with spaces"
shlex.split(t"ls {d}/file")
--> ["ls", "{d}/file"] # intermediate step
--> ["ls", "dir with spaces/file"]
And then whatever accepts a list knows how to quote/escape each argument (if necessary).
And yes, the splitting logic of a t-string here is platform independent, though it’d be nice for it to be pretty close to “normal” behaviour for as many platforms as possible. The quoting behaviour is entirely platform-specific, because we don’t have to quote arguments that are in a list.
Not to belabour the point, but this is the main advantage of a t-string over an f-string. We get to process the source string (after it’s already split into parts, which is a bit annoying, but manageable) before interpolating. ↩︎
The “bit annoying, but manageable” part of having to actually split each text segment is the aspect I was trying to avoid by quoting the inputs, but I suspect you’re right, and all the ways of potentially avoiding handling the input incrementally are actually worse once the goal is to support both the POSIX and non-POSIX splitting algorithms.
Where it potentially gets complicated is when programmatic quoting gets combined with inline quoting, but it may be that treating the inline quotes the same as any other non-whitespace character will actually give better results than the status quo gives in that case (the last two examples here are unlikely to be what the user wanted):
The main advantage I see for t-strings here would be avoiding mis-usable apis. shlex “works”, but as multiple examples have pointed out, use of it is still error prone. Some of the examples here actually mirror bugs I’ve seen in scripts pushed to production pipelines before. If t-strings can provide a one-shot API that is unambiguous and hard to misuse, that’s a benefit to everyone, reviewers, novices, and everyone in between.
I still think such an API needs to be a new API to have that benefit though, there’s no way to reconcile such a goal with the existing pile of options. It needs to encapsulate every part of this in a way that is resistant to misunderstanding, reliance on platform specific “oh, it works on my machine”, and so on.
For the user explicitly calling shlex.quote, the third ought to be what they get (I’m not sure which of those is actually the real result, but I do note that they’re all f-strings, not t-strings).
But I think it’s actually not all that bad. If you start by substituting each variable with something that never needs quoting, then you can do the normal shlex.split to get arguments, and then go back and substitute the original values into the list.
We would discourage explicit shlex.quote, because the only logical outcome is to get unnecessary quotes, which would then be escaped (or checked and ignored, but this is subprocess and not the t-string) and passed explicitly later on (because they’re part of the variable, not the template):
The examples you give would work equally well with an algorithm that just split on whitespace (while treating substitutions as non-whitespace for the purposes of splitting). Are there any examples that actually need anything more complex? I guess something like
shlex.split(t"'a b' {dirname}/{fname}") -> ["a b", "some dir/some file"]
would be a case that needs “shlex logic”, but are there realistic cases of things like that? Specifically ones where the shlex version is demonstrably better than just switching to list format?
I suppose it doesn’t matter that much though - if this is isolated to the shlex module, it’s clearly marked as both non-portable and relying on our interpretation of the POSIX rules[1]. So while it may not be the most useful feature (shlex isn’t exactly the most popular stdlib module) it will be well defined - and available to people who do get an advantage from its specific feature set.
This would give either ["ls", "-l -h", "some dir/some file"] or ["ls", "", "some dir/some file"] which would be both wrong. Of course it can simply ruled as “don’t to that”, but I just wanted to raise the fact that a substitution can be either a full argument or a part of an argument, but not cover multiple arguments. Contrary to f-string, the special handling of substitutions may have unexpected consequences. Maybe it should be explicit in the documentation of magic_split_function.
One of the advantages of template processors is that it’s relatively straightforward to add further type handling later (as long as unknown input types raise exceptions when processed). For example, if tuples were added as a valid substitution field value type, then interpolating a dynamic list of arguments might look like:
(accepting arbitrary iterables can be a bit more problematic, since that’s such a broad type it can significantly reduce your flexibility in changing the interpretation of other types in later API updates)
But with options being a tuple of string literals we’re going back to reading the noise of quotes and commas between arguments, defeating one of the important purposes of the proposal.
I suggest that we allow nested templates to be flattened instead:
I know a lot of people use subprocess with shell=True even when knowing the security risks because they don’t want to deal with the relative clunkiness of subprocess API’s I/O redirections and would rather take advantage of the convenience of | and > that a shell has to offer.
I agree that subprocess can use a new, higher-level API similar to pathlib as a wrapper library to os.path so that invoking external commands in Python feels a bit more at home for those who are more used to shell scripting.
plumbum is a good step in that direction, but I imagine that with a t-string we can make a library (or expand the existing subprocess) with an arguably friendlier API like this:
from subprocess2 import run, cmd, Result
target = "Python World"
output = Result()
run(
cmd-
t"echo Hello {target}" |
t"cat -n" >> output
)
print(output.stdout) # 1 Hello Python World
Like pathlib’s repurposing of the / operator, repurposing the |, << and >> operators should feel intuitive in this context.
This runs into the same problem that makes a new API that only accepts t-strings, not normal strings, appealing: namely, the behavior changes in a way that requires deep knowledge to spot an issue in a casual reading between a t-string and a string. It’s not an API where accepting either is a good thing as it adds to subtleties reviewers need to be aware of, and makes it impossible for static analysis tools to go “hey you shouldn’t do that”
If you’re suggesting that this new API should accept only t-strings then I agree with you. Notice how I made t"-l -h", t"" and t"cat -n" all template strings in my examples even when they’re fixed values.
If the docs were updated to reflect reality, then that earlier section you quoted would be changed.