118 lines
4.2 KiB
ReStructuredText
118 lines
4.2 KiB
ReStructuredText
..
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may
|
|
not use this file except in compliance with the License. You may obtain
|
|
a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
License for the specific language governing permissions and limitations
|
|
under the License.
|
|
|
|
.. _verification-reports:
|
|
|
|
====================
|
|
Verification reports
|
|
====================
|
|
|
|
Rally stores all verifications results in its DataBase so that you can access
|
|
and process results at any time. No matter what verifier you use, results will
|
|
be stored in a unified way and reports will be unified too.
|
|
|
|
We support several types of reports out of the
|
|
box: :include-var:`rally.cli.commands.verify.DEFAULT_REPORT_TYPES`; but our
|
|
reporting system is pluggable so that you can write your own plugin to build
|
|
some specific reports or to export results to the specific system (see
|
|
:ref:`howto-add-new-reporting-mechanism` for more details`).
|
|
|
|
.. contents::
|
|
:depth: 2
|
|
:local:
|
|
|
|
HTML reports
|
|
------------
|
|
|
|
HTML report is the most convenient type of reports. It includes as much as
|
|
possible useful information about Verifications.
|
|
|
|
Here is an example of HTML report for 3 verifications.
|
|
It was generated by next command:
|
|
|
|
.. code-block:: console
|
|
|
|
$ rally verify report --uuid <uuid-1> <uuid-2> <uuid-3> --type html \
|
|
--to ./report.html
|
|
|
|
|
|
.. image:: ../images/Report-Verify-for-4-Verifications.png
|
|
:align: center
|
|
|
|
The report consists of two tables.
|
|
|
|
First one is a summary table. It includes base information about
|
|
verifications: UUIDs; numbers of tests; when they were launched; statuses; etc.
|
|
Also, you can find detailed information grouped by tests statuses at the right
|
|
part of the table.
|
|
|
|
If the size (height) of the summary table seems too large for you and hinders
|
|
to see more tests results, you can push "Toggle Header" button.
|
|
|
|
The second table contains actual verifications results. They are grouped by
|
|
tests names. The result of the test for particular verification overpainted by
|
|
one of the next colours:
|
|
|
|
* *Red* - It means that test has "failed" status
|
|
* *Orange* - It is "unexpected success". Most of the parsers calculates it just
|
|
like failure
|
|
* *Green* - Everything is ok. The test succeeded.
|
|
* *Yellow* - It is "expected failure".
|
|
* *Light Blue* - Test is skipped. It is not good and not bad
|
|
|
|
Several verifications comparison is a default embedded behaviour of reports.
|
|
The difference between verifications is displayed in brackets after actual
|
|
test duration. Sign **+** means that current result is bigger that standard by
|
|
the number going after the sign. Sign **-** is an opposite to **+**. Please,
|
|
note that all diffs are comparisons with the first verification in a row.
|
|
|
|
Filtering results
|
|
"""""""""""""""""
|
|
|
|
You can filter tests by setting or removing a mark from check box of the
|
|
particular status column of the summary table.
|
|
|
|
.. image:: ../images/Report-Verify-filter-by-status.png
|
|
:align: center
|
|
|
|
Tests Tags
|
|
""""""""""
|
|
|
|
Some of the tests tools support tests tagging. It can be used for setting
|
|
unique IDs, groups, etc. Usually, such tags are included in test name. It is
|
|
inconvenient and Rally stores tags separately. By default they are hidden, but
|
|
if you push "Toggle tags" button, they will be displayed under tests names.
|
|
|
|
.. image:: ../images/Report-Verify-toggle-tags.png
|
|
:align: center
|
|
|
|
Tracebacks & Reasons
|
|
""""""""""""""""""""
|
|
|
|
Tests with "failed" and "expected failure" statuses have tracebacks of
|
|
failures. Tests with "skipped", "expected failure", "unexpected success" status
|
|
has "reason" of events. By default, both tracebacks and reasons are hidden,
|
|
but you can show them by clicking on the appropriate test.
|
|
|
|
.. image:: ../images/Report-Verify-tracebacks.png
|
|
:align: center
|
|
|
|
.. image:: ../images/Report-Verify-xfail.png
|
|
:align: center
|
|
|
|
Plugins Reference for all out-of-the-box reporters
|
|
--------------------------------------------------
|
|
|
|
.. generate_plugin_reference::
|
|
:base_cls: Verification Reporter
|