Adds a database retry decorator to capture and retry exceptions
rooted in SQLite locking. These locking errors are rooted in
the fact that essentially, we can only have one distinct writer
at a time. This writer becomes transaction oriented as well.
Unfortunately with our green threads and API surface, we run into
cases where we have background operations (mainly, periodic tasks...)
and API surface transacations which need to operate against the DB
as well. Because we can't say one task or another (realistically
speaking) can have exclusive control and access, then we run into
database locking errors.
So when we encounter a lock error, we retry.
Adds two additional configuration parameters to the database
configuration section, to allow this capability to be further
tuned, as file IO performance is *surely* a contributing factor
to our locking issues as we mostly see them with a loaded CI
system where other issues begin to crop up.
The new parameters are as follows:
* sqlite_retries, a boolean value allowing the retry logic
to be disabled. This can largely be ignored, but is available
as it was logical to include.
* sqlite_max_wait_for_retry, a integer value, default 30 seconds
as to how long to wait for retrying SQLite database operations
which are failing due to a "database is locked" error.
The retry logic uses the tenacity library, and performs an
expoential backoff. Setting the amount of time to a very large
number is not advisable, as such the default of 30 seconds was
deemed reasonable.
Change-Id: Ifeb92e9f23a94f2d96bb495fe63a71df9865fef3
091edb0631