This patch provides the plumbing for implementing storage
sharding across multiple backends. Sharding is agnostic to
storage driver type and transport type. The new feature is
optional, and disabled by default.
The design eschews placing any kind of sharding reverse proxy
in the network, allowing the storage drivers to continue
communicating directly with their respective backends.
Sharding can be enabled by setting the global "sharding"
option to True. Future patches will add a sharding section to
the config that can be used to tweak the way sharding works when
it is enabled.
Storage drivers are managed by a Catalog class. The Catalog is
responsible for registering and deregistering queues in the
catalog backend, and for looking up an appropriate driver,
according to which shard a particular queue has been assigned.
In the future, this design will make it straightforward to map
individual queues to different storage backends, according to user
preference.
FWIW, I considered enabling sharding by inserting the routing driver
as the last stage in the storage pipeline. However, it felt like
a hack for the following reasons:
* Doing so orphaned the regular, solitary driver that was
still always loaded at the end of the pipeline.
* Since the bootstrap was not aware of the sharding driver,
it could not be used to provide setup, so the catalog
object had to be turned into a singleton and options
had to always be loaded from the global config.
* The driver would have to be added to each controller
pipeline, and would have to always be the last stage in
the pipeline. Introducing a simple "sharded" boolean option
seemed to be a more straightforward, less error-prone way
for operators to enable sharding.
Partially-Implements: blueprint storage-sharding
Change-Id: I5190211e81fe4acd311b2cfdd0bae806cc3fec81
e0892978cd