Ability to catch sqlite3 error log

sqlite3 has a callback facility wherein the engine can call into the application to report error & diagnostic conditions with a numeric/text message. sqlite authors promote using this hook to C/C++ developers as supplying valuable information for understanding errors (and some performance optimization decisions!). However, the cpython _sqlite module does not expose this function. So python sqlite3 users cannot take advantage.

It would be great if, similar to the sqlite3 .set_trace_callback(), a similar callback existed for the error log. One complication is that this callback is documented to be as restricted in context as a signal handler, so presumably some queuing / mapping would be necessary.

NB: This is not equivalent to existing error RC’s from sqlite3 functions, which are already mapped to exceptions, but rather akin to logging records.

https://www.sqlite.org/errlog.html

cc: @erlendaasland

1 Like

The problem with the log callback, is that you have to register it with sqlite3_config() which must be called before SQLite initialisation. Currently, SQLite is initialised during module init (import sqlite3), so it is not trivial to expose such functionality. One option could be to not initialise SQLite during module init, and instead initialise lazily, but we’d still have to be sure that no database connections are open and no SQLite APIs are in use, so it has a very rigid contract with the user.

Note that the other callbacks (trace, authorizer, progress) work with database handles, so they do not have the restrictions of sqlite3_config().

Also notice that even though the SQLITE_CONFIG_LOG config option is currently marked as an “anytime configuration option”, that feature is not guaranteed. Quoting the SQLite docs:

The set of anytime configuration options can change (by insertions and/or deletions) from one release of SQLite to the next.

New configuration options may be added in future releases of SQLite. Existing configuration options might be discontinued.

Even with that limitation (“anytime” but who knows whether forever), this would be useful. Since sqlite3 does report errors if such calls are mistimed (change from “anytime”), the python wrapper could turn the call into an exception.

I suggest you check out the apsw[1] module; it has built-in support for the SQLite error log mechanism:

https://rogerbinns.github.io/apsw/tips.html#diagnostics-tips


  1. Another Python SQLite Wrapper ↩︎

It is not such large problem. You can always register a callback which does nothing by default. But if you then register a Python callable, it will call that callable.

The question is how much an empty log callback affects performance?

1 Like

That’s a possibility.

I’ll create a proof-of-concept branch, so we can benchmark it.

Here’s something to play around with:

https://github.com/erlend-aasland/cpython/pull/new/sqlite/errlog

1 Like

Thanks for the reference. A different flavour (requiring quite a bit of code porting) - and fast moving (so more version conditionals).

Definitely. apsw is free to do whatever they want. sqlite3 OTOH is bound by rigid backwards compatibility requirements and the DB API, so it is a lot less flexible. That does not mean we’re opposed to improve it, or (slowly) add new features, though :slight_smile:

FWIW, I think apsw is a very good piece of softwre.