Redirecting the error stream to a file in 3.10.0

Python Help
1

I found this issue while redirecting the error stream in python:

import sys
file = open("log", "w")
sys.stderr.write = file.write
variable

now as variable is not defined I must get a NameError, but when I run the script I get the file “log” but it doesn’t contain any value. The file is empty. I think so this is a bug because I got this code from stack overflow, it worked for many people but not for me.

This most likely is a bug in python 3.10.0 (I’m using python 3.10.0) so I posted this here instead of stack overflow, but before reporting I want to be sure this is a bug not my code’s problem.

2

Works fine for me on a mac with 3.10.0. The file log is populated with the traceback.

$ cat log
Traceback (most recent call last):
  File "/Users/python/stderrit.py", line 4, in <module>
    variable
NameError: name 'variable' is not defined. Did you mean: 'callable'?
$ /usr/local/opt/python@3.10/bin/python3 -V
Python 3.10.0
3

Files are bufferred, so the output won’t show up on disk until either
the buffer is full, or you flush the file.

import sys
file = open("log", "w")
sys.stderr.write = file.write
variable

At that point, file log is still zero bytes in size.

file.flush()

And now the file contains the text of the traceback.

4

@steven.daprano it did not work, When should I call the flush function? before raising error or after raising error?

BTW I wanted to give you this info, I think so this might be useful: the file’s write method itself is not working, I just tried using sys.stderr.write method separately and it did not write data, I even tried calling file.write to be sure it’s working but file.write did not write data

5

In 3.9 and later, sys.stderr is line buffered, so any newline should force a flush.

Even without that, the interpreter exiting should cause the flush. I’ve tried the above program on Mac and Windows, 3.6, 3.9, and 3.10. All of them create a populated “log” file with the traceback as contents.

6

Can you show the code that you used to test it?

7

sys.stderr may be line-buffered, but only after its write has been called (by print, logging.Logger.log, etc). In your case, write is overwritten with another write (provided by the user, file.write), so the buffering is configured by your open call.

For stderr redirection, I suggest overwriting sys.stderr completely with line-buffering

import sys
sys.stderr = open("log", "w", buffering=1)
variable

For future reference, to better reproduce stream flushing errors in Python, try to force a seg-fault:

import sys
import ctypes
file = open("log", "w")
sys.stderr.write = file.write
print("spam", file=sys.stderr)
ctypes.string_at(None)  # de-reference null
8

Aha, I see the problem!

sys.stderr.write = file.write

That doesn’t work. It should be:

sys.stderr = file

Now the file will update immediately.

9

I don’t understand. The program appears to create data within the “log” file for me just fine (centos, macos, windows, 3.6 → 3.10). What’s the difference in my attempts and others?

$ cat log
Traceback (most recent call last):
  File "test_stderr.py", line 4, in <module>
    variable
NameError: name 'variable' is not defined

(yes, the method shouldn’t be reassigned, but I can’t get it to fail even with the reassignment in place)

10

What’s your OS and the exact version of Python? Are you running it in an

interactive session? Are you using IDLE? What is your environment?

I have tried the original version (monkey-patching only the write

method) in three versions:

  • Python 2.7: fails with AttributeError;

  • Python 3.10.0: data is written to the log file, but only after flush;

  • Python 3.7.5: data is written to the log file, but only after flush.

I am running on Fedora Linux.

In all cases I am using the default Python interactive interpreter.

11

No idle. No interactive session. I’ve done no patching but left the program exactly as in the first post.
Mac Big Sur w/python 3.10.0, 3.9.5,
Centos 7 w/python 3.6.8
Windows 10 w/python 3.9.5

Here’s the Centos 7 with the OS default 3.6.8 session

[tmp]$ rm log
rm: cannot remove ‘log’: No such file or directory
[tmp]$ cat redir_sterr.py
import sys
file = open("log", "w")
sys.stderr.write = file.write
variable
[tmp]$ python3 -V
Python 3.6.8
[tmp]$ python3 redir_sterr.py
[tmp]$ cat log
Traceback (most recent call last):
  File "redir_sterr.py", line 4, in <module>
    variable
NameError: name 'variable' is not defined
12

@steven.daprano @EpicWink sys.stderr = open("log", "w") did not populate the file after raising the error but it sure did open the file. Even sys.stderr = open("log", "w", buffering = 1) did not work.