x/browbeat
Code Issues Proposed changes

Basic test based return codes for Browbeat

Browse Source 
This adds a very simple check for failed tests before Browbeat's exit
if failed tests are found Browbeat will exit with a return code of one.
This will provide Browbeat CI failure when tests fail without interuppting
ongoing tests if a single one fails.

Change-Id: I5382f684fe03d85692a275dc5c03a136004f34d9
This commit is contained in:
jkilpatr 2017-02-14 11:51:47 -05:00
parent d707d4bc6d
commit c0715bff39
2 changed files with 23 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
browbeat.py
View File

@ -111,7 +111,12 @@ def main():
_logger.info("Saved browbeat result summary to {}".format(
os.path.join(result_dir,time_stamp + '.' + 'report')))
lib.WorkloadBase.WorkloadBase.print_summary()
_logger.info("Browbeat Finished, UUID: {}".format(browbeat_uuid))
if lib.WorkloadBase.WorkloadBase.failure > 0:
_logger.info("Browbeat Finished with Failures, UUID: {}".format(browbeat_uuid))
sys.exit(1)
else:
_logger.info("Browbeat Finished Successfully, UUID: {}".format(browbeat_uuid))
sys.exit(0)
if __name__ == '__main__':
sys.exit(main())

17
doc/source/usage.rst
View File

@ -144,6 +144,23 @@ using some simple searches such as:
shaker_uuid: c918a263-3b0b-409b-8cf8-22dfaeeaf33e AND record.concurrency:1 AND record.test:Bi-Directional
Interpreting Browbeat Results
=============================
By default results for each test will be placed in a timestamped folder `results/` inside your Browbeat folder.
Each run folder will contain output files from the various workloads and benchmarks that ran during that Browbeat
run, as well as a report card that summarizes the results of the tests.
Browbeat for the most part tries to restrict itself to running tests, it will only exit with a nonzero return code
if a workload failed to run. If, for example, Rally where to run but not be able to boot any instances on your cloud
Browbeat would return with RC 0 without any complaints, only by looking into the Rally results for that Browbeat run
would you determine that your cloud had a problem that made benchmarking it impossible.
Likewise if Rally manages to run at a snails pace, Browbeat will still exit without complaint. Be aware of this when
running Browbeat and take the time to either view the contents of the results folder after a run. Or setup Elasticsearch
and Kibana to view them more easily.
Working with Multiple Clouds
============================