Process ends with the word 'Killed'

Python Help
1

My problem here is that there doesn’t seem to be any way to find out what has gone wrong. I’m relatively new to programming by the way.

So basically, I have made a simple script that recursively processes all the files under a directory. What I’ve included below is a simplification of the code I was using that I can run to reproduce the same problem. I’m happy to include the full script if that would be better. When I run the code below on almost any directory, it works perfectly. My problem is that my script was designed to work on a specific directory I have, which is full of files nested inside a maze of subdirectories, and although it has been working flawlessly until now, and allowed me to process 1000s of files, it seems to have encountered a specific file that causes a crash.

The details are as follows:

When I run my script, or when I run the snippet below, and import_path is set to this specific directory, the program crashes with nothing but the word ‘Killed’.

The problem directory has been working for a long time until now. Leading me to believe that the problem is caused by a specific file.

I can’t think of a way to get the address of the file, in order to delete it.

I’m running these programs from the command line on my Debian 12 PC

import os

slots = 1
import_path = '/media/user/60/to_import'

def process2(address):
	print(address)

def process(address):
	if address.endswith(('.jpg', '.png', 'jpeg')):
		process2(address)
		global slots
		slots = slots - 1

def list_files_scandir(path):
	with os.scandir(path) as entries:
		for entry in entries:
			if slots > 0:
				if entry.is_file():
					process(entry.path)
				elif entry.is_dir():
					list_files_scandir(entry.path)
			else:
				break

list_files_scandir(import_path)

I would be grateful for any insight.

Edit: I have fixed a typo, and included the full script below:

from PIL import Image
import os
import sqlite3
import hashlib
import shutil

max_files = 100
import_path = '/home/user/sort-qimgv/import'
import_files = os.listdir(import_path)
imported = 0
skipped = 0
db_path = "/home/user/sort-qimgv/database.db"

def hash(filename):
	with open(filename, 'rb', buffering=0) as f:
		return hashlib.file_digest(f, 'sha256').hexdigest()

def thumbnail(file):
		WIDTH = 1920
		HEIGHT = 1280
		img = Image.open(file)
		img.thumbnail((WIDTH, HEIGHT))
		img.save(file)

if len(import_files) == 0: 
	raise Exception("No files to import.")
else: 
	print("Files found.")

# How many slots are free?
conn = sqlite3.connect(db_path)
cur = conn.cursor()
cur.execute("SELECT * FROM `index` WHERE location=0")
rows = cur.fetchall()
slots = max_files - len(rows)
conn.close()
print(str(slots) + ' slots.')

def import_file(file):
	hash_first = hash(file)

	conn = sqlite3.connect(db_path)
	cur = conn.cursor()
	cur.execute("SELECT * FROM hash WHERE hash=?", (hash_first,))
	rows = cur.fetchall()
	conn.close()

	if len(rows) > 0:
		global skipped
		skipped = skipped + 1
		os.remove(file)
		return

	conn = sqlite3.connect(db_path)
	cur = conn.cursor()
	cur.execute("insert into hash (hash) values (?)", (hash_first,))
	conn.commit()

	if os.path.getsize(file) > 1000000:
		thumbnail(file)
		hash_second = hash(file)
		conn = sqlite3.connect(db_path)
		cur = conn.cursor()
		cur.execute("insert into hash (hash) values (?)", (hash_second,))
		conn.commit()

	# find a suitable id
	conn = sqlite3.connect(db_path)
	cur = conn.cursor()
	cur.execute("SELECT MIN(id) + 1 FROM `index` WHERE id + 1 NOT IN (SELECT id FROM `index`)")
	rows = cur.fetchall()
	conn.close()

	value = rows[0][0]
	if value is None:
		newid = 1
	else:
		newid = value

	conn = sqlite3.connect(db_path)
	cur = conn.cursor()
	cur.execute("insert into `index` values (?, 0)", (newid,))
	cur.execute("insert into old_name values (?, ?)", (newid, str(file)))
	conn.commit()	

	new_destination = "/home/user/sort-qimgv/sort/" + str(newid) + os.path.splitext(file)[1]
	shutil.move(file, new_destination)
	global imported, slots
	imported = imported + 1
	slots = slots - 1

def process(address):
	if address.endswith(('.jpg', '.png', 'jpeg')):
		print(address)
		import_file(address)

def list_files_scandir(path):
	with os.scandir(path) as entries:
		for entry in entries:
			if slots > 0:
				if entry.is_file():
					process(entry.path)
				elif entry.is_dir():
					list_files_scandir(entry.path)
			else:
				break

list_files_scandir(import_path)

print(str(imported) + ' imported, ' + str(skipped) + ' skipped.')

Edit 2:
I’ve realised that a lot of what I put in the snippet was extraneous. This is actually all I need to reproduce the bug:

user@C1:~/sort-qimgv$ cat test2.py
import os

import_path = '/media/user/60/to_import'

def list_files_scandir(path):
	with os.scandir(path) as entries:
		for entry in entries:
			if entry.is_file():
				print(entry.path)
			elif entry.is_dir():
				list_files_scandir(entry.path)

list_files_scandir(import_path)
user@C1:~/sort-qimgv$ python3 test2.py
Killed
user@C1:~/sort-qimgv$

Edit 3:

Well, this is kind of embarrassing… What was reliably going wrong yesterday had miraculously fixed itself today, and I have done absolutely nothing except restart my computer.

So I’m totally nonplussed about what was going on, and am now unable to investigate further.

I think it must have has something to do with the way the drive was mounted, due to the spontaneous after a restart. The reason I ruled that out before was that all the other directories on that same drive didn’t have that problem. I also mounted it this time in exactly the same way I always do using Thunar. Anyway, that’s the end. Sorry for the anticlimax. I’m happy to answer any questions, but obviously I can’t do any testing unless it happens again.

2

“Killed” generally means the process died due to SIGKILL, so, basically, something needed your process to die. One possibility is that you consumed so much memory that the system ran out, and needed to remove something (look up the “OOM killer” if you want details on this).

If this very very simple script is able to trigger the problem, it’s unlikely to be a memory issue; but it might be, so let’s start by eliminating this as a possibility. You can read a file called /proc/self/statm to see how much memory your current process is using. Periodically write out the contents of that file, and see whether the numbers seem to keep growing; also, see whether, just before it gets killed, the numbers are huge. The numbers there are measured in pages (probably 4KB each) - you can browse /proc/self/status for a more human-readable version, but for keeping an eye on the growth of memory, I prefer the one-line output format.

I doubt that that’s the problem, but it’s easy enough to check for.

Another thing to consider would be seeing if the file itself is corrupted or damaged in some way. Can you, for example, get the md5sum or sha1sum of the file (using the standard Unix utilities of those names)? That would test the file’s readability, so if there’s any issue with the device or driver, it would show up.

3

In list_files_scandir, if an entry is a file, it calls process, and if the filename ends with one of a number of extensions, the function calls itself, which, in turn, calls itself, etc.

4

You have effectively infinite recursion here:

def process(address):
	if address.endswith(('.jpg', '.png', 'jpeg')):
		process(address)
		global slots
		slots = slots - 1

Once it hits a .jpg, .png, or .jpeg it will recursively call itself forever. I’m guessing that this might be causing an OOM before it hits a RecursionError but I would expect to see that.

5

I’m guessing you never hit the issue prior because this is the first directory you’ve run that script in that has one of those files.

6

You mentioned this is a simplified version–I’m guessing it actually does something else to images when it finds them.

So one explanation is that whatever your process function is really doing is using too much memory, perhaps for a specific file.

1 Like
7

You’re right, but that is an unrelated mistake made by me, not the cause of the original problem.

8

That’s what I thought at first, but no. That’s why I made this simplified version. All it does is print the address of the file, but it still crashes in the same way.

9

I think so too, although I’ve always been running it in the same directory, I think it has come up against something it can’t handle.

10

If I’m understanding the code you put there, it recursively goes through a directory structure looking for the first image it finds and then prints out its full path. Am I missing something?

11

Is it possible you’ve got a cyclic symlink? Perhaps your function has found itself in a loop of scanning itself.

12

I think the issue might be you have a lot of images in there and are opening - and not closing - DB connections. Try removing the conn.close where you check to see if the hash is present and removing all the subsequent calls to conn = sqlite3.connect(db_path) and see if that might help. The idea of the connection is that it should be persistent and that you can do multiple queries with a single connection. Just remember to call conn.close() at each exit point.

13

I had cat /proc/self/statm running in a while loop while I ran my snippet, and there was no noticable change in any column. I probably should have mentioned that it only runs for a very short time:

real	0m0.188s
user	0m0.056s
sys	0m0.131s

As for looking at the file, I wish I could, but there are about a million files in that directory, and I have no idea which one is being chosen by python.

14

I’m reasonably sure you can also reuse the cursor you create with cur = conn.cursor() and just call cur.execute without making a new one each time. I can’t say with any certainty (my own lack of expertise) but it’s possible that references are being kept and things aren’t properly being garbage collected because of this behaviour.

15

That is it exactly. Except what actually happens is this:

user@C1:~/sort-qimgv$ python3 test.py
Killed
16

My debugging suggestion is add prints everywhere. Literally every line should be preceded by a print stating what it’s doing. I would start with the list_files_scandir function and see if you can get a handle one where it’s going in there.
If you’re using an IDE with a debugger, try using the debugger in the IDE. (I know it can be hard for a novice but it will give you the most insight.)
I hope this helps.

1 Like
17

That’s a good idea.

I can’t make sense of the result though, so I’ll leave it here in case anyone can:

import os

slots = 1
import_path = '/media/bruno/60/reddit'

print('a')

def process2(address):
	print(address)

print('b')

def process(address):
	if address.endswith(('.jpg', '.png', 'jpeg')):
		print('j')
		process2(address)
		print('k')
		global slots
		print('l')
		slots = slots - 1
		print('m')

print('c')

def list_files_scandir(path):
	print('e')
	with os.scandir(path) as entries:
		for entry in entries:
			print('f')
			if slots > 0:
				print('g')
				if entry.is_file():
					print('h')
					process(entry.path)
				elif entry.is_dir():
					print('i')
					list_files_scandir(entry.path)
			else:
				break

print('d')

list_files_scandir(import_path)

The output is the infinite repeating patten of ‘fgie’ ended with ‘Killed’.

f
g
i
e
f
g
i
e
f
g
i
e
Killed
18

It’s a good idea, and may well be true. My problem is that when I run find to_import -type l, to find, any symlinks, it crashes with the same Killed. I don’t know what to make of that.

19

That’s good to know, but I have isolated the bug to the first code snippet in my post, which doesn’t contain SQL.

20

Instead of printing single letters, print messages that are more informative, such as the paths.

1 Like