Can't figure out this error message "db type could not be determined"

Python Help
1

Complete noob here. I’ve taken some code straight out of the programming for beginner’s book, modified the variable and file names to those that I’m using, left the rest of the syntax as is, and I get an error that I can’t understand. It’s actually four errors, but only one references my code. The other three reference Lib files. The line of code in question is:

fstooges = shelve.open(“3 Stooges.dat”) [I previously created the .dat file]

The errors I get at that line are:

File “[filename and path]”, line 14, in <module>fstooges = shelve.open(“3 Stooges.dat”)

File “[filename and path]\Lib\shelve.py”, line 250, in openreturn DbfilenameShelf(filename, flag, protocol, writeback)

```File “[filename and path]\Lib\shelve.py”, line 227, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)```

`File “[filename and path]Lib\dbm_init_.py”, line 89, in open
raise error[0](“db type could not be determined”)
dbm.error: db type could not be determined```

I began by importing the ‘shelve’ module, so it’s not missing that. I’m not sure where I’m running off the rails here. Appreciate any interpretation and assistance.

Thank you.

2

How did you create 3 Stooges.dat? Does it have any contents? The shelve module only recognizes a few key/value pair file formats I think. It’s been a couple decades since I used it (long before Python 3 or sqlite3 were available), but gdbm, bsddb and Python’s dumbdbm formats come to mind.

Edit: also, what does this code print?

import dbm
dbm.whichdb("3 Stooges.dat")
1 Like
3

Hello,

can you check that the current working directory matches the location of your Python module via this snippet:

import os
print(os.getcwd())

Does it match the current location of your test file?

If not, you can change the working directory so that it matches the directory where your .dat file is located via the following commands (arbitrary directory for demo purposes - provide your own):

 os.chdir(r'C:\Desktop\myTestModule')

Then verify that changes were implemented:

import os
print(os.getcwd())
4

Strange things happening here:

  1. Instead of .dat, I changed it to .txt and still got the same error.

  2. The file is indeed in the correct location, but when I open it to see what it contains, it’s just a couple of long strings of Chinese text, nothing like what I created.

  3. All of the pickle methods are working, it’s just the shelve that is not.

Here is the full block of code. I’m going right by the book, but maybe I missed something or have something out of place.

import pickle, shelvestooges = [“Moe”, “Larry”, “Curly”]
adj = [“bad haircut”, “dim”, “clumsy”, “ugly”]
fstooges = open(“3 Stooges.dat”, “wb”)

pickle.dump(stooges, fstooges)
pickle.dump(adj, fstooges)
fstooges.close()

fstooges = open(“3 Stooges.dat”, “rb”)
stooges = pickle.load(fstooges)
print(stooges)

fstooges = shelve.open(“3 Stooges.dat”)    #this line causes the error

Hoping this can help. Thanks for your help.

5

You forgot to include:

fstooges.close()

from your last open pair. You are attempting to open a file that is already open.

It is better to use the with / open file context manager so that you don’t run into this problem. Once executing the body statements, it will automatically close the file for you even in the event when there are exceptions (i.e., errors).

Something like this:

with open(“3 Stooges.dat”, “rb”) as fstooges:
    pickle.dump(stooges, fstooges)
    pickle.dump(adj, fstooges)

with open(“3 Stooges.dat”, “rb”) as fstooges:
    stooges = pickle.load(fstooges)
    print(stooges)

Here is a Youtube tutorial on this subject that may be helpful:

What is the purpose of the last line?:

fstooges = shelve.open(“3 Stooges.dat”)

You open the file without accompanying body statements or closing file statement - it just hangs there.

6

Thanks Paul, I hadn’t come across with yet in my learning, so good to know.

As far as that last line, there was more code after it, but since that was the line where the error was happening, I didn’t include anything after it. After making your suggested edits, I’m still getting the error at that line and still getting a .dat file with all Chinese characters. Can I use with for the shelve command as well?

7

You need to import the shelve library package.

import shelve

or:

import pickle, shelve

shelvestooges = ['Moe', 'Larry', 'Curly']
adj = ['bad haircut', 'dim', 'clumsy', 'ugly']

with open('3 Stooges.pkl', 'wb') as fstooges:
    pickle.dump(stooges, fstooges)  # dump pickeled data to stooges
    pickle.dump(adj, fstooges)      # Serialize list and write to file fstooges

with open('3_Stooges.pkl', "rb") as fstooges:

    while True:
        
        try:
            obj = pickle.load(fstooges)
            print(obj)

        except EOFError:
            # End of file reached, no more objects to load
            break


fstooges = shelve.open('3 Stooges.pkl')    #this line causes the error

I would recommend using the with / open context manager with the last line as well.

8

I already had pickle and shelve imported, so that’s good. On the final line, you have the file type as .pkl; is that necessary for shelving?

I tried with/open, but I’m just confusing myself now. Here is what I have at the moment, with error handling commented out:

#try:
    #fstooges = shelve.open("3 Stooges.pkl")   #creates an error
    with open("3 Stooges.pkl", "rb") as fstooges:
            fstooges["adj"] = ["bad haircut", "dim", "clumsy", "ugly"] <--line 18
            fstooges.sync()
            print("adj =", fstooges["adj"])
#except Exception as errmsg:
    #print(errmsg)
fstooges.close()

The error I get now is:

File "[My path and filename]", line 18, in <module>
    fstooges["adj"] = ["bad haircut", "dim", "clumsy", "ugly"]
TypeError: '_io.BufferedReader' object does not support item assignment

Completely lost now. I googled TypeError, but didn’t get much out of it. What’s wrong this time?

9

You can also use .bin (binary) and txt (text) file suffixes. Experiment (swap them out and run your script) .

Don’t use empy space when creating file names. Use an underscore in place instead.

For example. Instead of this file.txt, use this_file.txt.

The error tells you exactly what the issue is:

object does not support item assignment

… because of this line:

fstooges["adj"] = ["bad haircut", "dim", "clumsy", "ugly"]
  1. The file is in read mode via rb. What you are attemping to do is a write.
  2. The data stored in the file is NOT in normal Python variable object mode. It is stored in a serialization format. Thus, you cannot fetch or reference the list object as you would normally in Python.
10

Good catch on the file name convention, I know better than that.

I tried other file extensions, but it makes no difference, still the same output. Changing the “r” to “w” results in an empty .dat file instead of giving me all the Chinese characters.

Not following your #2 comment. Can you please explain and let me know how I can modify the code to do a proper shelve? I think I’ve got the pickle part down, just struggling with the shelve part.

11

pickle and shelve are different ways to store data. You can’t mix them.

A shelve file acts like a dict:

# Write the data.
fstooges = shelve.open("3 Stooges.dat")
fstooges["stooges"] = stooges
fstooges.close()

# Read the data.
fstooges = shelve.open("3 Stooges.dat")
stooges = fstooges["stooges"]
print(stooges)
fstooges.close()

When you open a shelve file, it’ll complain if the file already exists but it isn’t a valid shelve file.

That’s what’s happening - you’ve just made a pickle file and are then trying to re-use that same file as a shelve file.

You could, for example, use different extensions for the 2 kinds of file, such as .pkl for pickle files and .slv for shelve files. That would provide a hint as to what format it is and reduce the chance of mixing them.

12

If you are familiar with the UTF-8 character coding standard, you’ll understand that every character (including those of different languages) are represented by an equivalent character in the Unicode standard (this is a whole new discussion by the way - opening Pandora’s box as they say). So, when pickle serializes your data into a file, it saves the data in binary format. Now consider the number of languages there are in the world. There are thousands (7000+). It is inevitable that different languages have similar sounding words - albeit having a different meaning. The same holds true here. Although the serialized data is being saved in your file in binary format, it has a different meaning to both pickle and to the file which interprets the serialized data. This is why the characters appear as Chinese even though they aren’t meant to be. Make sense?

When using the shelve library, you cannot use existing file names to save data to - give it a different name. The exception is to use a different file suffix.

As they say, when in Rome, do as the Romans. So, sometimes libraries lay down rules by which to go by and us users have to abide by them.

Here is an example. Write to the file named 4_Stooges.txt and then read the contents of the file to verify its contents.

# Provide a unique file name other than existing pickle file name
with shelve.open('4_Stooges.txt') as fstooges:
    fstooges["adj"] = ["bad haircut", "dim", "clumsy", "ugly"]

# Read value stored in shelve file for verification
with shelve.open('4_Stooges.txt') as fstooges:
    print(f'\nfstooges[adj] value stored in shelve file -->: {fstooges["adj"]}')
13

OK, it’s all coming into view now. I had some things out of sequence, and wasn’t adding anything to the shelve file, thinking it would be taken from the initial pickle file. Got it all straight and I’m not getting errors anymore.

The only question I have now is why the shelve file doesn’t take its data from the pickle file. I’m sure the reason is something simple and understanding it will help me put the pieces together.

Thanks for all the help!

14

When working strictly with the pickle library, data is serialized into binary format. If you save many objects into a file, you can’t fetch them by reference as you would normal Python code. The shelve library serves as an interface or liaison between the user and the pickle library allowing objects to be referenced as dictionary keys (see the code snippet from my last post).

In the attached sketch, it demonstrates how the shelve library is essentially like a wrapper for the pickle library making it easier to interface with for the user.

pickle_shelve_libraries
pickle_shelve_libraries731×475 6.8 KB

For example, to fetch an object using the pickle library, you may use something like this (more complex than using the shelve library):

def fetch_object(filename, obj_fetch):
    
    with open(filename, "rb") as fstooges:

        while True:

            try:
                obj = pickle.load(fstooges)

                if obj == obj_fetch:
                    print(f'Object fetched: {obj_fetch}')

            except EOFError:
                # End of file reached, no more objects to load
                break

fetch_object('3_Stooges.txt', shelvestooges)

Using the shelve library simplies the code.

Hope this helps.

15

When working only with the pickle module, data is serialized into a binary stream. That means if you save multiple objects into a single file, you can’t directly access them by key — you have to read them sequentially with pickle.load() until the end of the file.

The shelve module, on the other hand, acts as an interface or a wrapper around pickle. It combines pickle with a simple key–value database (based on dbm) and lets you access stored objects just like you would access elements in a dictionary.

That’s why the shelve file doesn’t “take” its data from a regular pickle file — their internal structures are completely different.

  • A pickle file is just a continuous stream of serialized Python objects.

  • A shelve file is a small database where each object is stored under a unique key.

So, if you already have a pickle file, you’ll need to load the data and then save it into a shelve database manually, like this:

import pickle
import shelve

# Load objects from pickle
with open("data.pkl", "rb") as f:
    data = pickle.load(f)

# Save them into shelve
with shelve.open("data_shelve.db") as db:
    db["dataset"] = data

Now you can access the data directly by key:

with shelve.open("data_shelve.db") as db:
    print(db["dataset"])

If your pickle file contains multiple objects, you can convert them all like this:

import pickle
import shelve

def convert_pickle_to_shelve(pickle_file, shelve_file):
    with open(pickle_file, "rb") as f:
        objs = []
        while True:
            try:
                objs.append(pickle.load(f))
            except EOFError:
                break

    with shelve.open(shelve_file) as db:
        for i, obj in enumerate(objs):
            db[f"obj_{i}"] = obj

convert_pickle_to_shelve("old_data.pkl", "new_data.db")

So, in short:

  • pickle = raw serialization (no indexing)

  • shelve = dictionary-like database built on top of pickle

  • They can’t read each other’s files directly because their structures differ

You just need to load from pickle and then re-save to shelve — after that, everything will work as expected.

16

Thanks Misha, that explains it rather nicely, and the sample code will definitely come in handy. I’m getting a little better every day at reading code and knowing what it says. Still a long way to go, but I’ve found a good community here to help me along.

Appreciate everyone’s assistance!