Ouch: several instances of the same key in a dictionary. Duplicate keys in a dictionary

Mabooka-Mabooka · September 29, 2026, 5:02pm

I ran into some very strange behavior.
I have right now a *.json file (that I am willing to upload / show);
that json file is created by a simple single statement
json.dump(d, file);
the object d stored in this file is a single python dictionary;
but that dictionary contains has SEVERAL NON-UNIQUE KEYS.
(?!?!?!?).

Here are some details.
The json file is created / updated by my script that gets information about files (such as stat() or some exiftool info) and then stores this information in a dictionary. The dictionary so looks like
[path] → (file info…).
I hope that is clear.

But today I stumped on the fact that some keys in this dictionary appear more than once!!!
(I stumped on it because something else broke, otherwise I would never ever bother to investigate as I believed that there is no way a diction can have duplicate keys).

There are just a few (24 out of about half a million) such keys; they all correspond to file names that are not ascii (e.g. Cyrillic…), – but still HOW IS THIS POSSIBLE?

BTW I have both $LANG and $LC_ALL set to US.UTF-8.
MAC OS 26.6 Tahoe;
I have Python 3.9.6 and Python 3.13.2 (behavior is same).

Please advise. THANKS!!!

Mabooka-Mabooka · September 29, 2026, 5:11pm

PS: here is my test script and the output from it:

$ cat fucked_up_json_dict.py
#!/usr/bin/env python3

import os, sys, json

def main():
    json_flnm = sys.argv[1]
    with open(json_flnm) as fp:
        j = json.load(fp)
    print(type(j))
    for key in j:
        print(key)

if name == "main":
    main()

$ python  fucked_up_json_dict.py Image_hashes.json | wc -l
416349
$ python  fucked_up_json_dict.py Image_hashes.json | sort | uniq | wc -l
416312

Rosuav · September 29, 2026, 5:22pm

Can you get an example of two keys that simultaneously exist in the dictionary but are showing up as duplicates? There are a few possible reasons for this; one is Unicode normalization, but I don’t know if that’s going to happen with Cyrillic text so I’ll give an example with Latin text instead.

>>> print(ascii(unicodedata.normalize("NFC", "a\u0301")))
'\xe1'
>>> d = {"\xE1": "E1", "a\u0301": "A-0301"}
>>> d
{'á': 'E1', 'á': 'A-0301'}

These are distinct strings - one is a single character, the other is one base character and one combining character - but depending on how your uniq handles this, it’s possible it would interpret them as the same. (Mine doesn’t for this example, which is another reason we need actual examples from your dataset.)

Mabooka-Mabooka · September 29, 2026, 6:58pm

Thank you for replying Chris!
Here are all 24 “bad” keys – they can be obtained simply like this:
python fucked_up_json_dict.py Image_hashes.json | sort | uniq -c | sort -n | tail -25

```
1 /Volumes/Backup_19/obfuscated/path/to/file/IMG_6448.PNG
2 /Volumes/Backup_10/obfuscated/path/to/file/Оля Грицан.png
2 /Volumes/Backup_10/obfuscated/path/to/file/Downloads/в своей студии.jpg
2 /Volumes/Backup_10/obfuscated/path/to/file/Downloads/Чингачгук.png
2 /Volumes/Backup_10/obfuscated/path/to/file/Pictures/SPECIAL/океан.jpeg
2 /Volumes/Backup_10/obfuscated/path/to/file/Pictures/SPECIAL/Family/On-The-Wall/Мы в Виши.jpg
2 /Volumes/Backup_10/obfuscated/path/to/file/Pictures/SPECIAL/Hui/хуйцаа.jpg
2 /Volumes/Backup_10/obfuscated/path/to/file/Pictures/SPECIAL/Мы с Петей.jpg
2 /Volumes/Backup_10/obfuscated/path/to/file/SPECIAL/Весна на женщину.jpg
2 /Volumes/Backup_10/obfuscated/path/to/file/Женя Ю с коровой.jpg
2 /Volumes/Backup_10/obfuscated/path/to/file/Friends/course_2014.jpg
2 /Volumes/Backup_10/obfuscated/path/to/file/Цзы.png
2 /Volumes/Backup_10/obfuscated/path/to/file/Ацик в Вегасе.jpg
2 /Volumes/Backup_10/obfuscated/path/to/file/selected/Unused/Никита с Пу copy.jpg
2 /Volumes/Backup_10/obfuscated/path/to/file/Downloads/Images_etc/Барсик Фофа Гошик Сквор.jpg
2 /Volumes/Backup_10/obfuscated/path/to/file/Downloads/Images_etc/Мгла.jpg
2 /Volumes/Backup_10/obfuscated/path/to/file/Перед Сатино.jpeg
2 /Volumes/Backup_10/obfuscated/path/to/file/Sonia лукавая.jpg
2 /Volumes/Backup_10/obfuscated/path/to/file/лицо.jpeg
2 /Volumes/Backup_19/obfuscated/path/to/file/Family/Пётр и мы.jpg
2 /Volumes/Backup_19/obfuscated/path/to/file/Sonia лукавая.jpg
3 /Volumes/Backup_10/obfuscated/path/to/file/Downloads/Images_etc/Гостиница Дружба 2.jpg
4 /Volumes/Backup_19/obfuscated/path/to/file/Family/просим на новый киндл.jpg
7 /Volumes/Backup_10/obfuscated/path/to/file/Downloads/Images_etc/ну и рожи.jpg
7 /Volumes/Backup_10/obfuscated/path/to/file/Downloads/Images_etc/Нота Фа.jpg
```

Note that the paths are obfuscated but the filenames are as they are.

Not sure how is this uniq’s problem as the strings (filenames) are obtained from the os’s find command.
Thanks!

Mabooka-Mabooka · September 29, 2026, 7:00pm

PS: I am more than willing to send you or whoever from Python community is interested to investigate the whole *.json file, just let me know.

Rosuav · September 29, 2026, 8:35pm

Thanks! Now, the hard part: Can you please open up the original JSON file, find those keys (a matched pair, or a handful of matched pairs), and show us the exact byte representation used? This might be a little trickier as it requires locating things as text, but fetching the undecoded bytes (rather than reencoding). If the JSON file is laid out one per line, it might be possible to find the lines in question, then grab those lines by number after opening the file as a byte stream. Something like:

with open(fn) as f:
    lines = [n for (n, l) in enumerate(f) if "IMG_6448" in l]
with open(fn, "rb") as f:
    print(*[ascii(l) for (n, l) in enumerate(f) if n in lines], sep="\n")

I find IMG_6448 a fasinating example as it doesn’t seem to have any non-ASCII characters in it. Maybe there are some in the path? In any case, seeing the ascii() of the lines in question will be revealing, but if the JSON file is all strung out on a single line, it’ll be harder to pull out the interesting bits.

Sure. If you’re comfortable sending it to me, but not sharing the file with everyone, toss me an email, I’ll see what I can do.

Stefan2 · September 29, 2026, 9:55pm

That isn’t duplicated. Note the “1” in front.

Rosuav · September 29, 2026, 10:00pm

Oh! right. That makes sense. Thanks for spotting that Stefan. Need to switch out for one of the problematic ones, then.

BTW, when you obfuscate the paths, do so after the call to ascii(), to preserve the relevant information.

Stefan2 · September 29, 2026, 10:10pm

Alternative plan:

Print with numbers:

for i, key in enumerate(sorted(j)):
    print(i, key)

sort and uniq ignoring the numbers, make uniq show only duplicates.

Now you have the numbers of the duplicates.

Run the Python script again, but print only the keys with those numbers (in some way revealing more information).

Stefan2 · September 29, 2026, 10:25pm

I think

... | sort -k 2 | uniq -f 1 -D

will do that.

mgedmin · September 30, 2026, 5:31am

Is it possible for there to be newlines in the keys? Because ‘hello\nworld’ and ‘goodbye\nworld’ would each print as two lines, and then uniq would de-duplicate the two copies of “world” into one.

If you repeat the experiment but with print(repr(key)), do you get the same numbers?

facundo · September 30, 2026, 5:32pm

Note that there are corner cases where Python produces JSONs with duplicated keys… e.g.

>>> json.dumps({1:2, "1":3})
'{"1": 2, "1": 3}'
>>> json.loads(_)
{'1': 3}

I tried to push for a fix but the issue was closed as invalid.

Maybe it’s related?

djhenderson · September 30, 2026, 6:11pm

Do you have control over creating the d dictionary that you write out as json?

If you do, I suspect the “duplicate” keys actually have different binary representations in the d dictionary. For instance, you might find the names are Windows wide strings from the os, in UTF-16LE, or the same translated to any of the Unicode forms: ‘NFC’, ‘NFKC’, ‘NFD’, ‘NFKD’ (See the unicodedata module). Or, if the names are passed through the pathlib or come from glob, they may be in normalized Unicode.