From the early days of Valet, there has been a known O(n)
inefficiency concerning Music query filtering. This is further
exacerbated by the instantiation of Music-backed ORM objects,
leading to a O(n*m) inefficiency.
This commit is a mitigation that greatly increases the likelihood
of O(1) filtering performance. valet-engine job status check
efficiency is also improved as a result.
This commit also fixes a condition where valet-api may create
multiple valet-engine placement requests for a stack id that
already has a request in process (e.g., if nova-scheduler calls
valet-api multiple times due to a retry by nova-controller).
IMPORTANT: When applying this commit, perform the following changes.
1: In valet.conf, ensure "tries" and "interval" are set to
100 and 0.1, respectively. (If they're commented out, change
the commented versions. There is no need to uncomment them.)
[music]
tries = 100
interval = 0.1
2: Manually create secondary keys in the valet-api cassandra tables.
For instance, given a keyspace of "valet_aic", execute these commands
in cqlsh on the cassandra server used by valet-api.
CREATE INDEX ON valet_aic.plans (stack_id);
CREATE INDEX ON valet_aic.placements (plan_id);
CREATE INDEX ON valet_aic.placements (orchestration_id);
CREATE INDEX ON valet_aic.placements (resource_id);
Until Music is updated to handle this automatically, these commands
must be executed every time the tables are dropped/re-added.
3: Determine the realistically expected number of simultaneous plan
requests for valet-api. Ensure the server running valet-api is sized
appropriately, set the httpd configuration's thread count to match
this number (perhaps a few extra), and restart the daemon.
For example, to specify 10 threads in Apache2 httpd, edit the
WSGIDaemonProcess directive for valet-api (this is one line):
WSGIDaemonProcess valet user=ubuntu group=ubuntu
threads=10 python-home=/opt/stack/heat.venv
4: To start with a clean slate, clear out any residual placement
requests and responses in the valet-engine cassandra tables.
For instance, given a keyspace of "valet_aic", execute these commands
in cqlsh on the cassandra server used by valet-engine.
TRUNCATE valet_aic.placement_requests;
TRUNCATE valet_aic.placement_results;
Change-Id: I76528c9b81dc451241ecc547cadc18cc4b1284df
0ff46ffb36