openstack/qa-specs
Code Issues Proposed changes

Add qa spec for bp:stress-api-tracking

Browse Source 
This patch adds the qa spec file for the
stress-api-tracking blueprint.

It has been expanded to include the tracking of api
calls from any tempest test, not just the stress tests.

Partially implements blueprint stress-api-tracking

Change-Id: I744f38612c885b455af0c505696768d716d3918e
This commit is contained in:
Ramy Asselin 2014-06-03 11:21:23 -07:00
parent e2a02d4dea
commit f5a6eb8763
1 changed files with 145 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

145
specs/stress-api-tracking.rst Normal file
View File

@ -0,0 +1,145 @@
::
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
..
==================================
Implement API Tracking
==================================
Previous BP URL: https://blueprints.launchpad.net/tempest/+spec/stress-api-tracking
Problem description
===================
API Call success/failure statistics provide valuable information in test
scenarios. However, manually tracking these in test cases is tedious and
reduces the readability of tests.
Proposed change
===============
Create a decorator that can be applied to select clients that will
automatically monitor the number of calls, successes, and failures of its public methods.
Create a new class called "Trace" with a single public method called "decorate" that will
wrap any desired class's public methods with the following functionality:
#. When such methods return normally the call is successful
#. When the method raises an exception
* If a method in the stack trace is white-listed, the call is successful.
* If no methods in the stack trace are white-listed, the call is a failure.
Example expected (and configurable) whitelisted methods could include:
* is_resource_deleted
* wait_for_server_termination
For each decorated method, maintain a dictionary that tracks:
#. The number of successfull calls
#. The number of failed calls
#. The list of observed stacktraces for the failed calls
Client APIs can then be decorated by e.g.::
def setUpTracking(self, trace):
"""add stats tracking decorator to services."""
if self.manager:
if self.manager.servers_client:
trace.decorate(self.manager.servers_client)
if self.manager.network_client:
trace.decorate(self.manager.network_client)
if self.manager.limits_client:
trace.decorate(self.manager.limits_client)
if self.manager.keypairs_client:
trace.decorate(self.manager.keypairs_client)
if self.manager.quotas_client:
trace.decorate(self.manager.quotas_client)
if self.manager.flavors_client:
trace.decorate(self.manager.flavors_client)
if self.manager.floating_ips_client:
trace.decorate(self.manager.floating_ips_client)
if self.manager.snapshots_client:
trace.decorate(self.manager.snapshots_client)
if self.manager.volumes_client:
trace.decorate(self.manager.volumes_client)
if self.manager.volume_types_client:
trace.decorate(self.manager.volume_types_client)
Run the test as usual.
Finally, the Stats class provides a basic implementation to log collected statistics as well as return the
statistics object for custom logging and other use cases.
At the end of a stress test run, print out the statistics gathered e.g. using LOG.info() logging::
2014-03-13 15:23:40.729 12273 INFO tempest.stress.driver [-] [VolumesClientJSON:get_volume] 6225 pass, 0 fail
2014-03-13 15:23:40.729 12273 INFO tempest.stress.driver [-] [VolumesClientJSON:create_volume] 151 pass, 0 fail
2014-03-13 15:23:40.730 12273 INFO tempest.stress.driver [-] [SnapshotsClientJSON:create_snapshot] 278 pass, 0 fail
2014-03-13 15:23:40.730 12273 INFO tempest.stress.driver [-] [SnapshotsClientJSON:delete_snapshot] 260 pass, 0 fail
2014-03-13 15:23:40.730 12273 INFO tempest.stress.driver [-] [VolumesClientJSON:delete_volume] 133 pass, 0 fail
2014-03-13 15:23:40.731 12273 INFO tempest.stress.driver [-] [ServersClientJSON:detach_volume] 583 pass, 0 fail
2014-03-13 15:23:40.731 12273 INFO tempest.stress.driver [-] [VolumesClientJSON:list_volumes] 2833 pass, 0 fail
2014-03-13 15:23:40.731 12273 INFO tempest.stress.driver [-] [ServersClientJSON:attach_volume] 595 pass, 0 fail
2014-03-13 15:23:40.732 12273 INFO tempest.stress.driver [-] [ServersClientJSON:get_server] 11 pass, 0 fail
2014-03-13 15:23:40.732 12273 INFO tempest.stress.driver [-] [SnapshotsClientJSON:wait_for_snapshot_status] 278 pass, 0 fail
2014-03-13 15:23:40.732 12273 INFO tempest.stress.driver [-] [VolumesClientJSON:wait_for_volume_status] 1607 pass, 0 fail
2014-03-13 15:23:40.732 12273 INFO tempest.stress.driver [-] [ServersClientJSON:wait_for_server_status] 2 pass, 0 fail
2014-03-13 15:23:40.732 12273 INFO tempest.stress.driver [-] [ServersClientJSON:create_server] 2 pass, 0 fail
2014-03-13 15:23:40.732 12273 INFO tempest.stress.driver [-] [SnapshotsClientJSON:list_snapshots] 2833 pass, 0 fail
2014-03-13 15:23:40.733 12273 INFO tempest.stress.driver [-] [SnapshotsClientJSON:get_snapshot] 550 pass, 0 fail
New classes could be located in module:
common/tracking.py
Implementation
==============
Assignee(s)
-----------
Ramy Asselin <ramy.asselin@hp.com>
Walter A. Boring IV <walter.boring@hp.com>
Milestones
----------
Target Milestone for completion:
Juno-2
Work Items
----------
[walter-boring] API Statistics Wrapper: DONE
[ramy-asselin] Stabilize API Statistics Wrapper: DONE
[walter-boring] Cinder Stress Test: DONE
[ramy-asselin] Stabilize Cinder Stress Tests: DONE
[ramy-asselin] Cinder CHO Test: DONE
[ramy-asselin] Cinder CHO Stabilize: DONE
[ramy-asselin] DOCUMENTATION: TODO
[ramy-asselin] Test Cases: TODO
[ramy-asselin] Code Cleanup: TODO
Current Solution
----------------
API Tracking: https://review.openstack.org/#/c/90449/