Large, low-maintenance database recommendation?

Python Help
1

I’m looking for a database library (probably a key-value store, or something like it).

I need it to scale to multiple terabytes, and I need it to be sturdy (no unrecoverable databases).

I’d like it to be:

  1. low maintenance
  2. fast
  3. single file per table?
  4. concurrent readers and writers
  5. no daemon to fire up?
  6. able to store nulls in keys and values would be nice

It doesn’t necessarily have to shard to multiple servers. In fact, I’d probably avoid a database library that required that.

I’m almost happy with anydbm+gdbm, but it’s a bit slow for large tables, and I don’t think it’s concurrent. And I’m almost happy with dbhash, but I’m not sure what maintenance is like for it, and I don’t think it’s concurrent either - and more importantly it seems to be moribund.

I know that shelve and Samba’s trivial db (tdb, not the same as text db) exist, but I have no experience with them.

The general trend seems to be away from single-table key-value stores of yore, and toward things that require a server process. Why?

Any suggestions?

Thanks!

2

Multiple terabytes? I’d stick that into PostgreSQL. You’d like it to be:

  1. low maintenance - that’s a relative term, but a properly-deployed PGSQL server shouldn’t require too much hassle
  2. fast - highly respected in the industry
  3. single file per table? - not sure, but you can architect it any way you like
  4. concurrent readers and writers - no problems here
  5. no daemon to fire up? - ehh, this one it fails on
  6. able to store nulls in keys and values would be nice - check

So it does lose out on point 5, but you’re looking at multiple terabytes, so I’m hoping a Postgres back end is worth that cost. (And for the record, on my Linux system, I can easily forget that I have Postgres installed - it starts automatically and doesn’t consume significant resources when not in use.) Having an external daemon is the easiest way to manage concurrent readers and writers, since it means that there’s a dedicated broker that keeps track of everything.

1 Like
3

This: A Guide to Partitioning Data In PostgreSQL | Severalnines
…sadly says:
The maximum table size allowed in a PostgreSQL database is 32TB, however unless it’s running on a not-yet-invented computer from the future, performance issues may arise on a table with only a hundredth of that space.

1 Like
4

I haven’t tried it at the scale you’re talking about but I’ve had good luck with DiskCache: Disk Backed Cache — DiskCache 5.6.1 documentation.

Typically dbs have their own process to allow that process to offload some processing from the app using it. Also because that process can synchronize requests and allow access via sockets to other hosts.