openstack/rally-openstack
Code Issues Proposed changes
ec1f9e4e01
rally-openstack/tests/ci
History
Andrey Kurilin bb21495db7 [CLI] group commands in help message 
In subcommand(for example in verification) we have several groups of
methods(managements, launchers, results). It would be nice to split
these groups in help message.

Before:

   compare           Deprecated. Use `rally verify results' instead.
   detailed          Display results table of a verification with detailed errors.
   discover          Show a list of discovered tests.
   genconfig         Generate Tempest configuration file.
   import            Import Tempest tests results into the Rally database.
   install           Install Tempest.
   installplugin     Install Tempest plugin.
   list              List verification runs.
   listplugins       List all installed Tempest plugins.
   reinstall         Uninstall Tempest and install again.
   results           Display results of verifications.
   show              Display results table of a verification.
   showconfig        Show Tempest configuration file.
   start             Start verification (run Tempest tests).
   uninstall         Remove the deployment's local Tempest installation.
   uninstallplugin   Uninstall Tempest plugin.
   use               Set active verification.

After:

   genconfig         Generate Tempest configuration file.
   install           Install Tempest.
   installplugin     Install Tempest plugin.
   listplugins       List all installed Tempest plugins.
   reinstall         Uninstall Tempest and install again.
   showconfig        Show Tempest configuration file.
   uninstall         Remove the deployment's local Tempest installation.
   uninstallplugin   Uninstall Tempest plugin.

   discover          Show a list of discovered tests.
   start             Start verification (run Tempest tests).

   compare           Deprecated. Use `rally verify results' instead.
   detailed          Display results table of a verification with detailed errors.
   import-results    Import Tempest tests results into the Rally database.
   list              List verification runs.
   results           Display results of verifications.
   show              Display results table of a verification.
   use               Set active verification.

Also this change transforms all _ to - in cli methods names.

Change-Id: I292e71d159ee35e933119f7fb57209f071aa37d4
2016-11-28 22:58:14 +03:00
..
hooks [ci] add test hook for certificaiton task 2016-11-11 01:58:07 +02:00
__init__.py Reorganize test module structure 2014-10-07 13:50:40 +00:00
cover.sh Add timeout for tests execution 2016-09-30 17:42:16 +03:00
osresources.py Add CinderVolumes.create_and_accept_transfer 2016-11-13 21:46:53 -05:00
pytest_launcher.py Merge "Using sys.exit(main(sys.argv)) exit the program" 2016-10-05 18:18:32 +00:00
rally_gate_functions.sh [CLI] group commands in help message 2016-11-28 22:58:14 +03:00
rally_verify.py Fix rally verify ci test 2016-11-23 20:30:31 +03:00
rally-gate.sh [ci] add test hook for certificaiton task 2016-11-11 01:58:07 +02:00
README.rst [CLI] Fix mentions of deprecated command `rally verify compare' 2016-10-20 12:12:38 +03:00
render.py [Reports] Get rid of Mako in reports, finally 2016-10-19 15:29:03 +03:00
sync_requirements.py Sync requirements before release 2016-10-11 19:22:10 +03:00
test_install.sh Make sure we don't have a stale package cache 2016-04-05 09:28:54 -04:00
wip-rally-gate.py [CLI] group commands in help message 2016-11-28 22:58:14 +03:00

README.rst

Rally Gate Jobs

For each patch submitted for review on Gerrit, there is a set of tests called gate jobs to be run against it. These tests check whether the Rally code works correctly after applying the patch and provide additional guarantees that it won't break the software when it gets merged. Rally gate jobs contain tests checking the codestyle (via pep8), unit tests suites, functional tests and a set of Rally benchmark tasks that are executed against a real devstack deployment.

rally-gate.sh

This script runs a set of real Rally benchmark tasks and fetches their results in textual / visualized form (available via a special html page by clicking the corresponding job title in Gerrit). It checks that scenarios don't fail while being executed against a devstack deployment and also tests SLA criteria to ensure that benchmark tasks have completed successfully.

rally-integrated.sh

This script runs a functional tests suite for Rally CLI. The tests call a range of Rally CLI commands and check that their output contains the expected data.

rally-verify.sh

This script runs various "rally verify" commands. This set of commands allow us to perform Tempest tests of OpenStack live cloud and display verification results. The verification results obtained by running various "rally verify <cmd>" commands including "start", "show", "list" are compared using the "rally verify results" command, which are then saved in csv, html and json formats in the "rally-verify" directory. Jenkins uses this script by running the 'gate-rally-dsvm-verify' job.

test_install.sh

This script tests the correct working of the install_rally.sh, used for the installation of Rally. Jenkins tests this script by running it against Centos6 and Ubuntu 12.04 in the corresponding jobs 'gate-rally-install-bare-centos6' and 'gate-rally-install-bare-precise'.

Jenkins

Jenkins is a Continuous Integration system which works as the scheduler. It receives events related to proposed changes, triggers tests based on those events, and reports back. For each patch that is uploaded for review on Gerrit, Jenkins runs it against the various rally gate jobs listed below along with their functions and local equivalents:

  • gate-rally-pep8 : code style check (equal to tox -epep8)
  • gate-rally-docs : documention generation (equal to tox -edocs)
  • gate-rally-python27 : unit tests against python27 (equal to tox -epy27)
  • gate-rally-python34(non-voting) : unit tests against python34 ( equal to tox -epy34) (non-voting since not all tests pass this)
  • rally-coverage : generates unit test coverage (equal to tox -cover)
  • gate-rally-install-bare-centos6 : testing of test_install.sh(described above) against Centos
  • gate-rally-install-bare-precise : testing of test_install.sh(described above) against Ubuntu 10.04
  • gate-rally-dsvm-rally : runs rally-gate.sh(described above) against OpenStack deployed by devstack with nova-network (It is standard dsvm job)
  • gate-rally-dsvm-neutron-rally : runs rally-gate.sh against OpenStack deployed by devastack with neutron
  • gate-rally-dsvm-cli : runs rally-integrated.sh ( equal to tox -ecli)
  • gate-rally-dsvm-verify(non-voting) : runs rally-verify.sh and tests Rally and Tempest integration in all possible ways
  • gate-rally-tox-self(non-voting) : not yet used

and a success in these tests(except non-voting) would mean that the patch is approved by Jenkins.