I added a print(entry.path) to the `if entry.is_file():’ part, and it basically confirmed what I think is happening. It works fine for directories, happily printing them, but as soon as it encounters a file, is crashes.
You could check in your python script, using if entry.is_symlink().
edit: or use entry.is_dir(follow_symlinks=False) to filter them out.
That tells you the memory of cat command not your script.
To monitor from outside your script replace “self” with the pid of the python process.
But the idea is to read that file in your script and print its value as the code runs.
1 Like
Looks like you have infinite recursion. Would it be worth rewriting your main search loop to use os.walk instead?
It crashes prior to printing out - on hitting the if entry.is_file() part? Try this:
Use glob.glob to see the order of the file objects in that directory. Try moving the first one that is a file (since glob.glob will just give you the filenames for all the files in the directory regardless of whether they are files are not) and see if that helps. I would also say, check the permissions on the file - there may be something screwy with that. In my tests (and mine were on a Mac but I think it would be the same) the file order from glob.glob and os.scandir were the same so it should help you find the file that’s causing the issue.
The name “self” refers to the current process, so you have to make your Python process print it out, not use any external process.
To narrow it down further:
import os
import_path = '/media/user/60/to_import'
def list_files_scandir(path, indent=''):
print(f'{indent}list_files_scandir({path!a})')
with os.scandir(path) as entries:
for entry in entries:
if entry.is_file():
print(f'{indent}file: {entry.path}')
elif entry.is_dir():
print(f'{indent}directory: {entry.path}')
list_files_scandir(entry.path, indent + ' ')
list_files_scandir(import_path)
That is very interesting. This suggests that your problem is not a
Python bug (in Python itself or your code).
On the assumption that you’re on a Linux system, does the output of the
command demsg contain anything interest right after your find blows
up? Eg:
find to_import -type l; dmesg | tail -20
Also, are you pulling data from some external drive? Might it have
filesystem problems?
Do you mean like this?
user@C1:~/sort-qimgv$ cat test3.py
import os
import_path = '/media/user/60/to_import'
def memory():
with open("/proc/self/stat") as f:
print(f.read())
def list_files_scandir(path):
with os.scandir(path) as entries:
for entry in entries:
if entry.is_dir():
memory()
list_files_scandir(entry.path)
else:
print('hello')
memory()
list_files_scandir(import_path)
user@C1:~/sort-qimgv$ python3 test3.py > temp.txt
Killed
user@C1:~/sort-qimgv$
here is the result: Paste ofCode
As you can see, there is no change in the values.
Have a look at sudo dmesg the last few lines may explain why the process was killed.
If that does not help that I would suggest that you run python under gdb and see why the debug thinks the process is killed.
$ gdb python3
(gdb) run test3.py
What does it report?
Frustratingly, the problem has fixed itself overnight. I have updated the post with some details. I’m sorry to leave you hanging. I wish I knew what it was, but now there’s no way to know.
Restarting the computer absolutely CAN change this sort of thing, so I’d file this one under “there was probably something lingering from a previous failure”. If it occurs again, it might be worth investigating; for now, I would say that this isn’t actually a Python problem, just that it was in Python where you saw the results.
1 Like
We know it wasn’t Python bacause it happens with find on the same file
tree.