Andrey Kurilin eef45465ca Configure gate-rally-dsvm-verify
Recently, we added new job "gate-rally-dsvm-verify" for Rally[1].
This job implied functional testing integration Rally with Tempest[2].

This patch adds tests/ci/rally-verify.sh script, which is used by new job.

rally-verify.sh does:
- tempest installation
- run "rally verify start" twice and print results
- compare results of two verifications
- list verifications
- generate html page based on results

To implement gate-rally-dsvm-verify some changes were requered to
existing Rally code:
- Added ability for rally/ui/utils.py to accept arguments to render html-pages
- Fixed logging debug-messages in tempest verifier
- Fixed check "is debug mode turned on or not"(also, added hacking rule for it)

TODO for future patches:
- add launch of rally task for Tempest
- add launch of random test set
- add check for successful tests

[1] https://review.openstack.org/#/c/137232
[2] https://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler

Closes-Bug: #1400465
Closes-Bug: #1400518

Change-Id: I8e1fbab22c2da109bbc442f040fe259e5d22a62a
2014-12-12 01:28:01 +02:00
..
2014-12-12 01:28:01 +02:00
2014-12-12 01:28:01 +02:00
2014-11-29 22:36:51 -08:00

Testing

Please, don't hesitate to write tests ;)

Unit tests

Files: /tests/unit/*

The goal of unit tests is to ensure that internal parts of the code work properly. All internal methods should be fully covered by unit tests with a reasonable mocks usage.

About Rally unit tests:

  • All unit tests are located inside /tests/unit/*
  • Tests are written on top of: testtools, fixtures and mock libs
  • Tox is used to run unit tests

To run unit tests locally:

$ pip install tox
$ tox

To run py26, py27 or pep8 only:

$ tox -e <name>

#NOTE: <name> is one of py26, py27 or pep8

To get test coverage:

$ tox -e cover

#NOTE: Results will be in /cover/index.html

To generate docs:

$ tox -e docs

#NOTE: Documentation will be in doc/source/_build/html/index.html

Functional tests

Files: /tests/functional/*

The goal of functional tests is to check that everything works well together. Fuctional tests use Rally API only and check responses without touching internal parts.

To run functional tests locally:

$ source openrc
$ rally deployment create --fromenv --name testing
$ tox -e cli

#NOTE: openrc file with OpenStack admin credentials

Rally CI scripts

Files: /tests/ci/*

This directory contains scripts and files related to the Rally CI system.

Rally Style Commandments

File: /tests/hacking/checks.py

This module contains Rally specific hacking rules for checking commandments.