Andrey Kurilin eef45465ca Configure gate-rally-dsvm-verify
Recently, we added new job "gate-rally-dsvm-verify" for Rally[1].
This job implied functional testing integration Rally with Tempest[2].

This patch adds tests/ci/rally-verify.sh script, which is used by new job.

rally-verify.sh does:
- tempest installation
- run "rally verify start" twice and print results
- compare results of two verifications
- list verifications
- generate html page based on results

To implement gate-rally-dsvm-verify some changes were requered to
existing Rally code:
- Added ability for rally/ui/utils.py to accept arguments to render html-pages
- Fixed logging debug-messages in tempest verifier
- Fixed check "is debug mode turned on or not"(also, added hacking rule for it)

TODO for future patches:
- add launch of rally task for Tempest
- add launch of random test set
- add check for successful tests

[1] https://review.openstack.org/#/c/137232
[2] https://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler

Closes-Bug: #1400465
Closes-Bug: #1400518

Change-Id: I8e1fbab22c2da109bbc442f040fe259e5d22a62a
2014-12-12 01:28:01 +02:00
..
2014-10-07 13:50:40 +00:00
2014-12-12 01:28:01 +02:00
2014-12-12 01:28:01 +02:00

Rally Style Commandments

Rally Specific Commandments

* [N30x] - Reserved for rules related to mock library
  • [N301] - Ensure that assert_* methods from mock library is used correctly
  • [N302] - Ensure that nonexistent "assert_called" is not used
  • [N303] - Ensure that nonexistent "assert_called_once" is not used
* [N310-N314] - Reserved for rules related to logging
  • [N310] - Ensure that rally.log is used instead of rally.openstack.common.log
  • [N311] - Validate that debug level logs are not translated
  • [N312] - Validate correctness of debug on check.
* [N32x] - Reserved for rules related to assert* methods
  • [N320] - Ensure that assertTrue(isinstance(A, B)) is not used
  • [N321] - Ensure that assertEqual(type(A), B) is not used
  • [N322] - Ensure that assertEqual(A, None) and assertEqual(None, A) are not used
  • [N323] - Ensure that assertTrue/assertFalse(A in/not in B) are not used with collection contents
  • [N324] - Ensure that assertEqual(A in/not in B, True/False) and assertEqual(True/False, A in/not in B) are not used with collection contents