To clarify, by if file path contains non-ASCII characters you mean /path/to/data\n-command\n contains undecodable characters, right?
In that case, you’re essentially asking what the canonical encoding is when communicating with a third-party tool—and there is none. Since input/output are fundamentally all bytes, the encoding to use is entirely between the two processes. There are some general guidelines you can follow (UTF-8 for POSIX and UTF-16 on Windows are common), but ultimately you’ll need to refer to either documentation or implementation of the tool to be entirely sure.