Possible data corruption in Python 3.11 on Windows 11

I’m trying to maintain a 25-song rolling playlist in iTunes on Windows via win32com and the iTunes COM interface. I’ve got it working fine, except upon initial running it occasionally fails at adding a random song to the playlist with an error stating the song I’m trying to add may have been deleted. The song list it chooses from is simply the iTunes library list that it loads at the start of the script. It looks at the rating of each song as they’re being loaded and funnels them into different groups based on that rating.

all_songs = itunes.LibraryPlaylist.Tracks
grouped_songs = {}
for song in all_songs:
    if song.Rating > 0 and not song.ExcludeFromShuffle:
        if song.Rating not in grouped_songs:
            grouped_songs[song.Rating] = []
        grouped_songs[song.Rating].append(song)

Then later in the script when I need a new song to be added to the playlist I’m using this to do so:

while len(playlist.Tracks) < 25:
    time.sleep(1)
    # Select a random rating group
    rating = random.choice(rating_weighting)
    # Select a random song from that rating group
    song = random.choice(grouped_songs[rating])
    # Add the song to the playlist if it's not already there
    if song not in playlist.Tracks:
        try:
            playlist.AddTrack(song)
        except pywintypes.com_error as e:
            # I no longer care to see the errors because I can't do anything about them anyway.
            pass

Occasionally during the initial run it will throw an error from the playlist.AddTrack(song) line as if iTunes thinks I’m referencing a file that does not exist and the add attempt fails. After being annoyed at it exiting every time this happened I threw it in a “try” so that it would just make another attempt whenever it ran across a failure. I couldn’t quite figure out why it was failing in the first place despite trying to determine if it was tied to specific songs or if random.choice() was somehow trying to select from outside the data pool. It was tough because once one of the song entries was throwing an error I couldn’t print(“song.Name:”, song.Name) successfully because it would throw an error at that point, too, so I didn’t think it had something to do with iTunes itself once the print commands were also failing. It’ll happily print song.Name for any entries that are still good, but as soon as any of them are bad it’ll throw the error. I managed to track down a few “bad” songs by printing out the index before and index after the bad one and then looking at the entire list of songs to see which songs were in between the two good indexes at the supposedly bad index. Different run attempts would show it successfully adding the bad songs, so it wasn’t tied to some weird data attached to any one song, it seemed, since it would successfully add one of those songs sometimes. The bad songs just seemed to be entirely random. And the thing is, the longer I let this script run the more often it throws failures. I’ve currently got a print statement in there that prints the group number that it is going to select from next so that I can see if it is trying multiple times to add one song. The longer it runs the longer and longer the thing prints the attempts being made when it needs to add another song. And as this happens the successful attempts at adding a song are more and more frequently adding the same smaller group of songs. So it seems as though the data is being corrupted for some reason, and more and more indexes are going bad. The whole script is only about 150 lines if anyone wants to see the whole thing, and not just those snippets. I’m stumped, though I have very little experience as I’m new to python anyway. Any help would be appreciated. Thanks.

Welcome to the forum @_Shorty and thanks for quoting your code correctly first time.

I don’t know the libraries you are using, so here’s a few general observations:

  1. You need to see the error message (and so does anyone here who tries to help you). The try is a good move to keep the program running but for diagnosis a stack dump is invaluable. You should be able to print e and carry on.
  2. The way the error moves around makes me think concurrent access might be the problem. Maybe iTunes is part way through re-organising its tables when you next read the data? If the library is properly designed it should allow some kind of locking to coordinate threads.
  3. Printing strings on Windows is sometimes tricky if Python outputs characters the console cannot represent, even from a valid string. Printing onto sys.stderr can be more robust, or printing the repr() of a string rather than the string itself.

ok, I had a big reply here earlier, but I think it isn’t necessary now, as I think it was going down the wrong track. I think I’ve narrowed it down to these calls.

playlist.Tracks[0].Delete()

I tried paring back the script so the only things it did was connect to iTunes to get the song library list and then do a loop to randomize “song” and print each random song.Name and I could not get it to error out. If I made it loop 1,000 times I got 1,000 successful song.Name prints. Ditto for a 10,000 and 100,000. Alright, so something else it is doing has to be responsible for messing up the data. I started adding some of the other commands back in. When I got to those deletion commands is when it started spitting out errors. Sometimes a small amount, and sometimes a very large amount.

It didn’t seem to matter how many times I executed that delete command, as it didn’t seem to get any worse with each command executed. At least, I couldn’t tell if it did. I tried five attempts each of one deletion, two deletions, five deletions, and 10 deletions. Each of the five attempts seemed to return what looked like a random amount of errors. Sometimes about a hundred, sometimes around 500, sometimes around 900, etc. But it sure seemed to be tied to whether or not I made a deletion call. Calls to retrieve the library list a few times didn’t seem to matter. Nor did any of the other calls seem to matter. But the delete calls will do it relatively reliably. Only one run out of those 20 trials did the use of delete result in no errors. All 19 other runs resulted in a bunch of errors. So I don’t know what’s at fault. Something in the pywin32 library not liking what iTunes says after you make a deletion?

Here’s the code if anybody’s interested in seeing the whole thing.

# This will play a rolling 25-song playlist in iTunes called iTunes DJ, playing any songs with at least 1 star,
# playing songs with higher ratings more often than songs that have lower ratings.  You can add songs to the
# playlist or remove songs from it if you feel like it and it will adjust as needed.  Its goal is to maintain
# a 25-song list at all times and keep it going. It starts playing at the top, but doesn't start deleting
# songs from the list until the currently-playing song is sixth on the list. It then maintains that position
# in order to allow you to see the last five songs that played.  This gives you playback behaviour similar to
# the old Party Shuffle / iTunes DJ feature.  If only we could still add songs and vote for songs from the
# iTunes remote app.  They never should've removed the feature from iTunes in the first place.  It was the
# best feature!  -Clayton Macleod

import os
import pywintypes
import time
import random
import win32com.client # Make sure to install the pywin32 library

os.system('cls')
print("Loading iTunes if not already loaded...")

# Set up the iTunes COM interface
itunes = win32com.client.Dispatch("iTunes.Application")

print("Loading music library...")

# Define the playlist name
playlist_name = "iTunes DJ"

# Check if the playlist already exists, and create it if not
playlist = None
for pl in itunes.LibrarySource.Playlists:
    if pl.Name == playlist_name:
        playlist = pl
        break
if not playlist:
    playlist = itunes.CreatePlaylist(playlist_name)

# Load all songs from the library and group them by rating
all_songs = itunes.LibraryPlaylist.Tracks
grouped_songs = {}
for song in all_songs:
    if song.Rating > 0 and not song.ExcludeFromShuffle:
        if song.Rating not in grouped_songs:
            grouped_songs[song.Rating] = []
        grouped_songs[song.Rating].append(song)

# Define the rating group weighting
rating_weighting = [20, 40, 40, 60, 60, 60, 80, 80, 80, 80, 100, 100, 100, 100, 100]

print("Checking for iTunes DJ playlist and creating if not found...")
# Fill the playlist with songs
print("Ensuring playlist contains at least 25 songs...")
while len(playlist.Tracks) < 25:
    time.sleep(1)
    # Select a random rating group
    rating = random.choice(rating_weighting)
    # Select a random song from that rating group
    song = random.choice(grouped_songs[rating])
    # Add the song to the playlist if it's not already there
    if song not in playlist.Tracks:
        try:
            playlist.AddTrack(song)
        except pywintypes.com_error as e:
            # I no longer care to see the errors because I can't do anything about them anyway.
            pass
            
# Set shuffle and repeat off for the current playlist
playlist.Shuffle = 0
playlist.SongRepeat = 0

# Start playback if necessary
print("Ctrl-C any time to halt this process, after which you can stop or exit iTunes.")
print("You can add or delete any songs you like in the iTunes DJ playlist and it'll get topped up as necessary.")
state = itunes.PlayerState
if itunes.PlayerState == 1 and itunes.CurrentPlaylist.Name == playlist_name:
    print("iTunes DJ is already playing...")
else:
    print("Beginning playback...")
    # Start playback
    playlist.PlayFirstTrack()

# Delete tracks above current song if necessary
current_track = itunes.CurrentTrack
current_index = 0
for i in range(0, playlist.Tracks.Count):
    #print(f"playlist.Tracks[{i}].Name: {playlist.Tracks[i].Name}")
    if playlist.Tracks[i].Name == current_track.Name:
        current_index = i
        break
#print("current_index:", current_index)
#input()
while current_index > 5:
    playlist.Tracks[0].Delete()
    current_index -= 1

print("First five track check completed...")

# Top up the playlist if anything was deleted
while len(playlist.Tracks) < 25:
    time.sleep(1)
    # Select a random rating group
    rating = random.choice(rating_weighting)
    # Select a random song from that rating group
    song = random.choice(grouped_songs[rating])
    #print("song: ", song.Name)
    #input()
    # Add the song to the playlist if it's not already there
    if song not in playlist.Tracks:
        try:
            playlist.AddTrack(song)
        except pywintypes.com_error as e:
            # I no longer care to see the errors because I can't do anything about them anyway.
            pass
            
# Watch for track changes and update the playlist accordingly
while True:
    # Having this set to 6 means it keeps 5 previously played songs above the currently playing track.
    current_track = itunes.CurrentTrack
    current_index = 0
    for i in range(0, playlist.Tracks.Count):
        #print(f"playlist.Tracks[{i}].Name: {playlist.Tracks[i].Name}")
        if playlist.Tracks[i].Name == current_track.Name:
            current_index = i
            break
    while current_index > 5:
        # Delete the top track if there are more than 25 tracks
        playlist.Tracks[0].Delete()
        current_index -= 1
        # Add a new song to the bottom of the playlist if there are less than 25 tracks
        while len(playlist.Tracks) < 25:
            time.sleep(1)
            # Select a random rating group
            rating = random.choice(rating_weighting)
            # Select a random song from that rating group
            song = random.choice(grouped_songs[rating])
            # Add the song to the playlist if it's not already there
            print("Topping up...", rating)
            if song not in playlist.Tracks:
                try:
                    playlist.AddTrack(song)
                except pywintypes.com_error as e:
                    # I no longer care to see the errors because I can't do anything about them anyway.
                    pass
            else:
                # I haven't seen this printed yet so it probably isn't a file-already-in-playlist failure.
                print("Song already in playlist. Skipping...")
            
            # I don't know why it added this.  Seems redundant.
            #if len(playlist.Tracks) >= 25:
                #break
    # Wait for a short time before checking again
    time.sleep(1)

I had a reply, but I guess it was too long after I edited it some. Anyway, I think it was headed down the wrong track. I think I’ve narrowed it down to this call to iTunes to delete a song from the playlist.

playlist.Tracks[0].Delete()

I think the only outside library I’m using is pywin32. (Or was pywintypes external too?)

import os
import pywintypes
import time
import random
import win32com.client # Make sure to install the pywin32 library

I pared the script down to just doing a test loop after it snagged the song library list from iTunes. It would not error out at all. I started adding bits back in, and once I added the delete command back in is when it started throwing errors very reliably. Only 1 of 20 trials with varying numbers of delete commands present resulted in no errors. The other 19/20 trials all threw hundreds of errors in a loop that printed randomized song.Name entries 1000 times. Something in the reply from iTunes when making the deletion calls making something freak out, it would seem. I suppose I’ll go open an issue with the pywin32 project. I think it seems to be an issue for the pywin32 guys from the looks of things, but here is an error message from a playlist add attempt using song.Name:

Traceback (most recent call last):
  File "C:\iTunesDJ-project\iTunesDJ-current working version-debugging4.py", line 138, in <module>
    playlist.AddTrack(song)
  File "<COMObject <unknown>>", line 2, in AddTrack
pywintypes.com_error: (-2147352567, 'Exception occurred.', (0, None, 'The track has been deleted.', None, 0, -1610350078), None)

And when attempting to print song.Name while troubleshooting:

Traceback (most recent call last):
  File "C:\iTunesDJ-project\iTunesDJ-current working version-debugging.py", line 138, in <module>
    print("song.Name:", song.Name)
                        ^^^^^^^^^
  File "C:\Users\Clay\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\win32com\client\dynamic.py", line 628, in __getattr__
    ret = self._oleobj_.Invoke(retEntry.dispid, 0, invoke_type, 1)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pywintypes.com_error: (-2147352567, 'Exception occurred.', (0, None, 'The track has been deleted.', None, 0, -1610350078), None)