Propose release notes for Rally 0.10.0
Change-Id: Idd1a72f17756f275ba453959b35fadf6d650fe55
This commit is contained in:
Andrey Kurilin
parent
04382bc6c9
commit
e97e312fc7
520
doc/release_notes/archive/v0.10.0.rst
Normal file
520
doc/release_notes/archive/v0.10.0.rst
Normal file
@ -0,0 +1,520 @@
|
|||||||
|
=============
|
||||||
|
Rally v0.10.0
|
||||||
|
=============
|
||||||
|
|
||||||
|
Overview
|
||||||
|
--------
|
||||||
|
|
||||||
|
+------------------+-----------------------+
|
||||||
|
| Release date | **10/20/2017** |
|
||||||
|
+------------------+-----------------------+
|
||||||
|
|
||||||
|
* Ability to use OpenStack deployments without admin credentials
|
||||||
|
* Better validation process of tasks
|
||||||
|
* New task format (with an ability to set description for workloads)
|
||||||
|
* New JSON report
|
||||||
|
* ElasticSearch exporter
|
||||||
|
* `OSProfiler support
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/quick_start/tutorial/step_11_profiling_openstack_internals.html>`_
|
||||||
|
* Restructure of configuration options.
|
||||||
|
|
||||||
|
Details
|
||||||
|
-------
|
||||||
|
|
||||||
|
Command Line Interface
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
* Introduce `rally task import
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/cli_reference.html#rally-task-import>`_
|
||||||
|
command for importing task results into database.
|
||||||
|
|
||||||
|
* Extend tags support for tasks. Now you can specify several tags for a single
|
||||||
|
task using `--tag argument
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/cli_reference.html#task-start-tag>`_.
|
||||||
|
Also filtering tasks by tags is now available.
|
||||||
|
|
||||||
|
* Move DB category from ``rally-manage db`` to `rally db
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/cli_reference.html#category-db>`_ and
|
||||||
|
introduce `rally db show
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/cli_reference.html#rally-db-show>`_
|
||||||
|
command for printing the used connection string.
|
||||||
|
|
||||||
|
Deployments
|
||||||
|
~~~~~~~~~~~
|
||||||
|
|
||||||
|
This release we started a huge work related to simplification of deployment
|
||||||
|
component of Rally. There is a good progress which includes several nice
|
||||||
|
features:
|
||||||
|
|
||||||
|
* The format.
|
||||||
|
"ExistingCloud" deployment type covers 99.99% cases and is used as a base for
|
||||||
|
all new things. Also, it will be extended to support different platforms
|
||||||
|
soon. The new format looks like (for OpenStack case):
|
||||||
|
|
||||||
|
.. code-block:: json
|
||||||
|
|
||||||
|
{
|
||||||
|
"openstack": {
|
||||||
|
"admin": {
|
||||||
|
"username": "admin",
|
||||||
|
"password": "changeme",
|
||||||
|
"tenant_name": "foo",
|
||||||
|
},
|
||||||
|
"auth_url": "https://example.com",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
* admin user is optional in case of setting existing users.
|
||||||
|
From the beginning, setting admin credentials was a required section of Rally
|
||||||
|
deployment configuration. Even with introducing existing users feature, this
|
||||||
|
behaviour left.
|
||||||
|
Finally, we finished a big code refactoring and admin credential become
|
||||||
|
optional section. If a set of plugins for particular workload doesn't require
|
||||||
|
admin user, you can launch this task at deployment without setting it.
|
||||||
|
|
||||||
|
The information about the requirements of plugins you can find at
|
||||||
|
`Plugins Reference page
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html>`_ (see
|
||||||
|
``Requires platform(s):`` section at the bottom of each plugin).
|
||||||
|
|
||||||
|
* Originally, Rally project was designed to check performance of OpenStack and
|
||||||
|
we succeeded in building awesome tool. We do not plan to stop and just want
|
||||||
|
to inform about our future plans to expand a number of supported platforms.
|
||||||
|
Subscribe to our `GitHub organization
|
||||||
|
<https://github.com/xrally>`_ to not miss new plugins.
|
||||||
|
|
||||||
|
Task component
|
||||||
|
~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
* The new task format is introduced. It includes a bunch of improvements,
|
||||||
|
unification, etc. All the docs and samples will be updated soon.
|
||||||
|
|
||||||
|
As for now, you can check `a spec
|
||||||
|
<https://github.com/openstack/rally/blob/0.10.0/doc/specs/implemented/new_rally_input_task_format.rst>`_
|
||||||
|
for this big change.
|
||||||
|
|
||||||
|
* SLA failure_rate max=0 become a default if nothing else is specified.
|
||||||
|
|
||||||
|
* Totally reworked atomic actions. The atomic actions now supports nested
|
||||||
|
actions which allows to measure durations inside the scenario even more
|
||||||
|
precise. You can find them in HTML report or in our new json report
|
||||||
|
(see ``rally task report --json``).
|
||||||
|
|
||||||
|
* Generation of names for new resources takes care about particular workload
|
||||||
|
id, so it helps to provide a better cleanup and prepare for new feature -
|
||||||
|
disaster cleanup.
|
||||||
|
|
||||||
|
Plugins
|
||||||
|
~~~~~~~
|
||||||
|
|
||||||
|
We started supporting discovering plugins by entry-points, so you can easily
|
||||||
|
deliver your custom plugins as a simple python package.
|
||||||
|
|
||||||
|
To make you package after-discoverable, you need to specify the proper
|
||||||
|
entry-point at your setup.cfg file:
|
||||||
|
|
||||||
|
.. code-block::
|
||||||
|
|
||||||
|
rally_plugins =
|
||||||
|
path=package_name
|
||||||
|
|
||||||
|
**Deployment Engines**:
|
||||||
|
|
||||||
|
Remove serverproviders & rarely used deployers
|
||||||
|
|
||||||
|
Unfortunately, seems like nobody is using deployers for deploying
|
||||||
|
their clouds and mostly people would like just to execute their code.
|
||||||
|
|
||||||
|
1) Remove server provides
|
||||||
|
2) Remove engines that uses server providers
|
||||||
|
|
||||||
|
**OpenStack clients**:
|
||||||
|
|
||||||
|
* Deprecate EC2 client. It wasn't used in any of plugins and doesn't support
|
||||||
|
keystone v3
|
||||||
|
|
||||||
|
* Move ``rally.osclients`` module to ``rally.plugins.openstack.oscliens``
|
||||||
|
|
||||||
|
**Scenarios**:
|
||||||
|
|
||||||
|
The old way to describe scenario plugin via method is finally removed.
|
||||||
|
Initially Rally scenario plugins were methods of special class, like below:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
from rally.task import scenario
|
||||||
|
|
||||||
|
class SomeBasicClass(scenario.Scenario):
|
||||||
|
|
||||||
|
@scenario.configure()
|
||||||
|
def some_scenario(self, arg1):
|
||||||
|
"""An implementation of SomeBasicClass.foo scenario."""
|
||||||
|
|
||||||
|
@scenario.configure()
|
||||||
|
def another_scenario(self):
|
||||||
|
"""Implementation of another scenario, SomeBasicClass.bar."""
|
||||||
|
|
||||||
|
However to unify scenarios with other plugins we moved to model where
|
||||||
|
plugin is class. It was done long time ago.
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
from rally.task import scenario
|
||||||
|
|
||||||
|
@scenario.configure(name="CustomName")
|
||||||
|
class Some(scenario.Scenario):
|
||||||
|
|
||||||
|
def run(self, arg1):
|
||||||
|
"""An implementation of the scenario."""
|
||||||
|
|
||||||
|
We had a bunch of code that was used for transition and backward compatibility
|
||||||
|
that we finally removed.
|
||||||
|
|
||||||
|
|
||||||
|
* *NEW!!*
|
||||||
|
|
||||||
|
- `CinderQos.create_and_get_qos
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cinderqos-create-and-get-qos-scenario>`_
|
||||||
|
|
||||||
|
- `CinderQos.create_and_list_qos
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cinderqos-create-and-list-qos-scenario>`_
|
||||||
|
|
||||||
|
- `CinderQos.create_and_set_qos
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cinderqos-create-and-set-qos-scenario>`_
|
||||||
|
|
||||||
|
- `CinderQos.create_qos_associate_and_disassociate_type
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cinderqos-create-qos-associate-and-disassociate-type-scenario>`_
|
||||||
|
|
||||||
|
- `CinderVolumeTypes.create_and_get_volume_type
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-and-get-volume-type-scenario>`_
|
||||||
|
|
||||||
|
- `CinderVolumeTypes.create_and_list_volume_types
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-and-list-volume-types-scenario>`_
|
||||||
|
|
||||||
|
- `CinderVolumeTypes.create_and_update_encryption_type
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-and-update-encryption-type-scenario>`_
|
||||||
|
|
||||||
|
- `CinderVolumeTypes.create_and_update_volume_type
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-and-update-volume-type-scenario>`_
|
||||||
|
|
||||||
|
- `CinderVolumeTypes.create_get_and_delete_encryption_type
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-get-and-delete-encryption-type-scenario>`_
|
||||||
|
|
||||||
|
- `CinderVolumeTypes.create_volume_type_add_and_list_type_access
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-volume-type-add-and-list-type-access-scenario>`_
|
||||||
|
|
||||||
|
- `Dummy.openstack
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#dummy-openstack-scenario>`_
|
||||||
|
|
||||||
|
- `GlanceImages.create_and_deactivate_image
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#glanceimages-create-and-deactivate-image-scenario>`_
|
||||||
|
|
||||||
|
- `GlanceImages.create_and_download_image
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#glanceimages-create-and-download-image-scenario>`_
|
||||||
|
|
||||||
|
- `GlanceImages.create_and_get_image
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#glanceimages-create-and-get-image-scenario>`_
|
||||||
|
|
||||||
|
- `GlanceImages.create_and_update_image
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#glanceimages-create-and-update-image-scenario>`_
|
||||||
|
|
||||||
|
- `K8sPods.create_pods
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#k8spods-create-pods-scenario>`_
|
||||||
|
|
||||||
|
- `K8sPods.create_rcs
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#k8spods-create-rcs-scenario>`_
|
||||||
|
|
||||||
|
- `K8sPods.list_pods
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#k8spods-list-pods-scenario>`_
|
||||||
|
|
||||||
|
- `ManilaShares.create_and_extend_share
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#manilashares-create-and-extend-share-scenario>`_
|
||||||
|
|
||||||
|
- `ManilaShares.create_and_shrink_share
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#manilashares-create-and-shrink-share-scenario>`_
|
||||||
|
|
||||||
|
- `ManilaShares.create_share_then_allow_and_deny_access
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#manilashares-create-share-then-allow-and-deny-access-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronBGPVPN.create_and_delete_bgpvpns
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-delete-bgpvpns-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronBGPVPN.create_and_list_bgpvpns
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-list-bgpvpns-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronBGPVPN.create_and_list_networks_associations
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-list-networks-associations-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronBGPVPN.create_and_list_routers_associations
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-list-routers-associations-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronBGPVPN.create_and_update_bgpvpns
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-update-bgpvpns-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronBGPVPN.create_bgpvpn_assoc_disassoc_networks
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-bgpvpn-assoc-disassoc-networks-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronBGPVPN.create_bgpvpn_assoc_disassoc_routers
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-bgpvpn-assoc-disassoc-routers-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronNetworks.create_and_show_ports
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronnetworks-create-and-show-ports-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronNetworks.create_and_show_routers
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronnetworks-create-and-show-routers-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronNetworks.create_and_show_subnets
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronnetworks-create-and-show-subnets-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronNetworks.set_and_clear_router_gateway
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronnetworks-set-and-clear-router-gateway-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronSecurityGroup.create_and_delete_security_group_rule
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronsecuritygroup-create-and-delete-security-group-rule-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronSecurityGroup.create_and_list_security_group_rules
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronsecuritygroup-create-and-list-security-group-rules-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronSecurityGroup.create_and_show_security_group
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronsecuritygroup-create-and-show-security-group-scenario>`_
|
||||||
|
|
||||||
|
- `NeutronSecurityGroup.create_and_show_security_group_rule
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronsecuritygroup-create-and-show-security-group-rule-scenario>`_
|
||||||
|
|
||||||
|
- `NovaServerGroups.create_and_delete_server_group
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservergroups-create-and-delete-server-group-scenario>`_
|
||||||
|
|
||||||
|
- `NovaServerGroups.create_and_get_server_group
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservergroups-create-and-get-server-group-scenario>`_
|
||||||
|
|
||||||
|
- `NovaServers.boot_and_get_console_url
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservers-boot-and-get-console-url-scenario>`_
|
||||||
|
|
||||||
|
- `NovaServers.boot_server_and_attach_interface
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservers-boot-server-and-attach-interface-scenario>`_
|
||||||
|
|
||||||
|
- `NovaServers.boot_server_and_list_interfaces
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservers-boot-server-and-list-interfaces-scenario>`_
|
||||||
|
|
||||||
|
- `NovaServers.boot_server_attach_volume_and_list_attachments
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservers-boot-server-attach-volume-and-list-attachments-scenario>`_
|
||||||
|
|
||||||
|
* *UPDATED!!*
|
||||||
|
|
||||||
|
- The new argument ``properties`` is added to scenario
|
||||||
|
`IronicNodes.create_and_list_node
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#ironicnodes-create-and-list-node-scenario>`_
|
||||||
|
|
||||||
|
|
||||||
|
* *DELETED*
|
||||||
|
|
||||||
|
Fuel and Nova-Network are not alive any more. So we removed those scenarios.
|
||||||
|
If any of those scenarios a critical for you, please contact us.
|
||||||
|
|
||||||
|
- `FuelEnvironments.create_and_delete_environment
|
||||||
|
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#fuelenvironments-create-and-delete-environment-scenario>`_
|
||||||
|
|
||||||
|
- `FuelEnvironments.create_and_list_environments
|
||||||
|
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#fuelenvironments-create-and-list-environments-scenario>`_
|
||||||
|
|
||||||
|
- `FuelNodes.add_and_remove_node
|
||||||
|
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#fuelnodes-add-and-remove-node-scenario>`_
|
||||||
|
|
||||||
|
- `NovaFloatingIpsBulk.create_and_delete_floating_ips_bulk
|
||||||
|
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novafloatingipsbulk-create-and-delete-floating-ips-bulk-scenario>`_
|
||||||
|
|
||||||
|
- `NovaFloatingIpsBulk.create_and_list_floating_ips_bulk
|
||||||
|
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novafloatingipsbulk-create-and-list-floating-ips-bulk-scenario>`_
|
||||||
|
|
||||||
|
- `NovaNetworks.create_and_delete_network
|
||||||
|
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novanetworks-create-and-delete-network-scenario>`_
|
||||||
|
|
||||||
|
- `NovaNetworks.create_and_list_networks
|
||||||
|
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novanetworks-create-and-list-networks-scenario>`_
|
||||||
|
|
||||||
|
- `NovaSecGroup.boot_and_delete_server_with_secgroups
|
||||||
|
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-boot-and-delete-server-with-secgroups-scenario>`_
|
||||||
|
|
||||||
|
- `NovaSecGroup.boot_server_and_add_secgroups
|
||||||
|
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-boot-server-and-add-secgroups-scenario>`_
|
||||||
|
|
||||||
|
- `NovaSecGroup.create_and_delete_secgroups
|
||||||
|
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-create-and-delete-secgroups-scenario>`_
|
||||||
|
|
||||||
|
- `NovaSecGroup.create_and_list_secgroups
|
||||||
|
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-create-and-list-secgroups-scenario>`_
|
||||||
|
|
||||||
|
- `NovaSecGroup.create_and_update_secgroups
|
||||||
|
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-create-and-update-secgroups-scenario>`_
|
||||||
|
|
||||||
|
**Validators**:
|
||||||
|
|
||||||
|
The validators refactoring was a long-term task which blocked us to abandon
|
||||||
|
alignment to only OpenStack platform and requirements of setting admin
|
||||||
|
credentials. In this release, we made a great progress and fixed a lot of
|
||||||
|
issues and blockers which made possible to refactor validators.
|
||||||
|
Now validation step is equal for all types of plugins (Scenario, SLA, Context,
|
||||||
|
Hooks, Runners, etc).
|
||||||
|
|
||||||
|
The old way to add validator for scenario is deprecated. The new unified way
|
||||||
|
looks like:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
import time
|
||||||
|
|
||||||
|
from rally.common import validation
|
||||||
|
from rally.task import scenario
|
||||||
|
|
||||||
|
@validation.add("number", param_name="timeout", minval=0)
|
||||||
|
@scenario.configure(name="Foo.bar")
|
||||||
|
class FooScenario(scenario.Scenario):
|
||||||
|
def run(self, timeout):
|
||||||
|
time.sleep()
|
||||||
|
|
||||||
|
The old validators from ``rally.task.validators`` module is deprecated too, see
|
||||||
|
equivalents which can be used with ``add`` decorator:
|
||||||
|
|
||||||
|
- required_openstack --> `required_platform
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-platform-validator>`_
|
||||||
|
with setting platform argument to "openstack"
|
||||||
|
|
||||||
|
- external_network_exists ->`external_network_exists
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#external-network-exists-validator>`_
|
||||||
|
|
||||||
|
- file_exists ->`file_exists
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#file-exists-validator>`_
|
||||||
|
|
||||||
|
- flavor_exists ->`flavor_exists
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#flavor-exists-validator>`_
|
||||||
|
|
||||||
|
- image_exists ->`image_exists
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#image-exists-validator>`_
|
||||||
|
|
||||||
|
- image_valid_on_flavor ->`image_valid_on_flavor
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#image-valid-on-flavor-validator>`_
|
||||||
|
|
||||||
|
- number ->`number
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#number-validator>`_
|
||||||
|
|
||||||
|
- required_api_versions ->`required_api_versions
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-api-versions-validator>`_
|
||||||
|
|
||||||
|
- required_cinder_services ->`required_cinder_services
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-cinder-services-validator>`_
|
||||||
|
|
||||||
|
- required_clients ->`required_clients
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-clients-validator>`_
|
||||||
|
|
||||||
|
- required_contexts ->`required_contexts
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-contexts-validator>`_
|
||||||
|
|
||||||
|
- required_neutron_extensions ->`required_neutron_extensions
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-neutron-extensions-validator>`_
|
||||||
|
|
||||||
|
- required_param_or_context ->`required_param_or_context
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-param-or-context-validator>`_
|
||||||
|
|
||||||
|
- required_services ->`required_services
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-services-validator>`_
|
||||||
|
|
||||||
|
- restricted_parameters ->`restricted_parameters
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#restricted-parameters-validator>`_
|
||||||
|
|
||||||
|
- validate_heat_template ->`validate_heat_template
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#validate-heat-template-validator>`_
|
||||||
|
|
||||||
|
- volume_type_exists ->`volume_type_exists
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#volume-type-exists-validator>`_
|
||||||
|
|
||||||
|
- workbook_contains_workflow -> `workbook_contains_workflow
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#workbook-contains-workflow-validator>`_
|
||||||
|
|
||||||
|
- network_exists is removed, since we do not find any customers for it.
|
||||||
|
Please contact us if it was useful for you.
|
||||||
|
|
||||||
|
- validate_share_proto is removed in favor of enum validator
|
||||||
|
|
||||||
|
Fixed bugs
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
* [plugins] JSON schema of `servers
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#servers-context>`_
|
||||||
|
context allows to transmit a list of nics in two formats. First one is a
|
||||||
|
format that novaclient expects to see (each network should be represented
|
||||||
|
like ``{"nic-id": "the id of the network"}``). The second one is more
|
||||||
|
user-friendly - just list of strings (each network should be represented
|
||||||
|
just by id of the network). Unfortunately, the second case was not covered
|
||||||
|
by our code base.
|
||||||
|
|
||||||
|
Also, the first described format works in limited cases due to bad
|
||||||
|
serialization.
|
||||||
|
|
||||||
|
`Launchpad bug-report #1695245
|
||||||
|
<https://bugs.launchpad.net/rally/+bug/1695245>`_
|
||||||
|
|
||||||
|
* [deployment] ~/rally/.openrc not working for keystone v3
|
||||||
|
|
||||||
|
`Launchpad bug-report #1683820
|
||||||
|
<https://bugs.launchpad.net/rally/+bug/1683820>`_
|
||||||
|
|
||||||
|
* [plugins] Failed to list volumes in case of missed name in the object.
|
||||||
|
|
||||||
|
* [backported into 0.9.1][deployment] Credentials is not updated as soon as
|
||||||
|
deployment is recreated. Need to call recreate request twice.
|
||||||
|
|
||||||
|
`Launchpad bug-report #1675271
|
||||||
|
<https://bugs.launchpad.net/rally/+bug/1675271>`_
|
||||||
|
|
||||||
|
* [backported into 0.9.1][plugins] Scenario `IronicNodes.create_and_list_node
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#ironicnodes-create-and-list-node-scenario>`_
|
||||||
|
had a wrong check that list of all nodes contains newly created one.
|
||||||
|
|
||||||
|
* [backported into 0.9.1][task][cleanup] Do not remove quotas in case of
|
||||||
|
existing users
|
||||||
|
|
||||||
|
* [backported into 0.9.1][task][cleanup] Various traces of neutron resources
|
||||||
|
|
||||||
|
* [backported into 0.9.1][core] Keystone v3, authentication error for Rally
|
||||||
|
users if the value of project_domain_name of admin user isn't equal "default"
|
||||||
|
|
||||||
|
`Launchpad bug-report #1680837
|
||||||
|
<https://bugs.launchpad.net/rally/+bug/1680837>`_
|
||||||
|
|
||||||
|
* [backported into 0.9.1][task] Scenario `NovaHosts.list_and_get_hosts
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novahosts-list-and-get-hosts-scenario>`_
|
||||||
|
obtains hostname for all hosts. But it fails in some environments if host is
|
||||||
|
not compute.
|
||||||
|
|
||||||
|
`Launchpad bug-report #1675254
|
||||||
|
<https://bugs.launchpad.net/rally/+bug/1675254>`_
|
||||||
|
|
||||||
|
* [backported into 0.9.1][verification] Rally fails to run on systems on which
|
||||||
|
python-virtualenv is not installed
|
||||||
|
|
||||||
|
`Launchpad bug-report #1678047
|
||||||
|
<https://bugs.launchpad.net/rally/+bug/1678047>`_
|
||||||
|
|
||||||
|
* [backported into 0.9.1][verification] CLI `rally verify rerun
|
||||||
|
<https://rally.readthedocs.io/en/0.9.1/verification/cli_reference.html#rally-verify-rerun>`_
|
||||||
|
fails with TypeError due to wrong integration with Rally API.
|
||||||
|
|
||||||
|
* [plugins] Rally fails while creating neutron router on the clouds where
|
||||||
|
ext-gw-mode extension is not installed.
|
||||||
|
|
||||||
|
* [plugins] Scenario `CinderVolumes.create_nested_snapshots_and_attach_volume
|
||||||
|
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumes-create-nested-snapshots-and-attach-volume-scenario>`_
|
||||||
|
fails on a big load due to the fact that one server is used for several
|
||||||
|
iterations. In such case we are facing 2 issues: the maximum number of
|
||||||
|
volumes per VM is 26 (which is a number of available names for volumes);
|
||||||
|
detaching volume of one iteration can block attaching of other iterations.
|
||||||
|
|
||||||
|
`Launchpad bug-report #1708160
|
||||||
|
<https://bugs.launchpad.net/rally/+bug/1708160>`_
|
||||||
|
|
||||||
|
|
||||||
|
Thanks
|
||||||
|
~~~~~~
|
||||||
|
|
||||||
|
2 Everybody!
|
||||||
@ -1 +1 @@
|
|||||||
./archive/v0.9.2.rst
|
./archive/v0.10.0.rst
|
||||||
157
doc/specs/implemented/improve_atomic_actions_format.rst
Normal file
157
doc/specs/implemented/improve_atomic_actions_format.rst
Normal file
@ -0,0 +1,157 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
..
|
||||||
|
This template should be in ReSTructured text. The filename in the git
|
||||||
|
repository should match the launchpad URL, for example a URL of
|
||||||
|
https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named
|
||||||
|
awesome-thing.rst . Please do not delete any of the sections in this
|
||||||
|
template. If you have nothing to say for a whole section, just write: None
|
||||||
|
For help with syntax, see http://sphinx-doc.org/rest.html
|
||||||
|
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
|
||||||
|
|
||||||
|
|
||||||
|
=============================================
|
||||||
|
New Atomic actions format in workload results
|
||||||
|
=============================================
|
||||||
|
|
||||||
|
Currently atomic actions data in workload results is insufficient,
|
||||||
|
therefore some new features can not be implemented.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
The main problem is that current format does not support nested
|
||||||
|
atomic actions.
|
||||||
|
|
||||||
|
Also, atomic actions data does not include timestamps for each action
|
||||||
|
start and end time. Having this data will allow us to inspect atomic
|
||||||
|
actions runtime better and generate detailed reports.
|
||||||
|
|
||||||
|
Since word "atomic" means something that can not be split into parts
|
||||||
|
and we introduce nested atomic actions, we should use different term
|
||||||
|
instead of "atomic actions".
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
Term "atomic actions" should be renamed to just "actions".
|
||||||
|
|
||||||
|
Change actions results schema from type "object" to "array"
|
||||||
|
and extend it with timestamps and nested actions.
|
||||||
|
|
||||||
|
Nested actions will be represented by "children" key and have
|
||||||
|
unlimited nesting.
|
||||||
|
|
||||||
|
With timestamps, there is no need to save durations anymore,
|
||||||
|
so get rid of this value.
|
||||||
|
|
||||||
|
Since this change is not backward compatible, we need to create
|
||||||
|
a database migration script. The migration will use iteration start
|
||||||
|
timestamp as start timestamp for first action and then calculate
|
||||||
|
further timestamps based on actions order and their durations.
|
||||||
|
|
||||||
|
Benefits of new format
|
||||||
|
----------------------
|
||||||
|
|
||||||
|
Nested actions will make actions measurement more detailed and flexible
|
||||||
|
since we could have data what sub-actions were run during specific action
|
||||||
|
runtime, without complicated changes at code.
|
||||||
|
|
||||||
|
Start and end timestamps will provide us with accurate information
|
||||||
|
about action runtime within the whole iteration and ability to create
|
||||||
|
`Gantt charts <https://en.wikipedia.org/wiki/Gantt_chart>`_.
|
||||||
|
|
||||||
|
Schema modification
|
||||||
|
-------------------
|
||||||
|
|
||||||
|
Schema location is *rally.common.objects.task.TASK_RESULT_SCHEMA
|
||||||
|
["properties"]["result"]["properties"]["atomic_actions"]*
|
||||||
|
|
||||||
|
should be moved to *rally.common.objects.task.TASK_RESULT_SCHEMA
|
||||||
|
["properties"]["result"]["properties"]["actions"]*
|
||||||
|
|
||||||
|
and changed:
|
||||||
|
|
||||||
|
AS IS:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
{
|
||||||
|
"type": "object"
|
||||||
|
}
|
||||||
|
|
||||||
|
Here keys are actions names, and values are their durations.
|
||||||
|
Actions data is actually represented by collections.OrderedDict,
|
||||||
|
so we have real order saved.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
OrderedDict([("keystone.create_tenant", 0.1234),
|
||||||
|
("keystone.create_users", 1.234)])
|
||||||
|
|
||||||
|
TO BE:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
{
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"name": {"type": "string"}, # name of action
|
||||||
|
"started_at": {"type": "number"}, # float UNIX timestamp
|
||||||
|
"finished_at": {"type": "number"}, # float UNIX timestamp
|
||||||
|
"children": {"$ref": "#/"},
|
||||||
|
},
|
||||||
|
"required": ["name", "started_at", "finished_at", "children"],
|
||||||
|
"additionalProperties": False
|
||||||
|
},
|
||||||
|
"minItems": 0
|
||||||
|
}
|
||||||
|
|
||||||
|
Example how this data can be represented:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
[{"name": "keystone.create_tenant",
|
||||||
|
"started_at": 1455281370.288397,
|
||||||
|
"finished_at": 1455281372.672342,
|
||||||
|
"children": []},
|
||||||
|
{"name": "keystone.create_users",
|
||||||
|
"started_at": 1455281372.931324,
|
||||||
|
"finished_at": 1455281373.375184,
|
||||||
|
"children": []}]
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
Alexander Maretskiy <amaretskiy@mirantis.com>
|
||||||
|
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
- Rename atomic actions into actions
|
||||||
|
- Improve actions results format
|
||||||
|
- Create a DB migartion that transforms results to new format
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
None
|
||||||
348
doc/specs/implemented/new_rally_input_task_format.rst
Normal file
348
doc/specs/implemented/new_rally_input_task_format.rst
Normal file
@ -0,0 +1,348 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
..
|
||||||
|
This template should be in ReSTructured text. The filename in the git
|
||||||
|
repository should match the launchpad URL, for example a URL of
|
||||||
|
https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named
|
||||||
|
awesome-thing.rst . Please do not delete any of the sections in this
|
||||||
|
template. If you have nothing to say for a whole section, just write: None
|
||||||
|
For help with syntax, see http://sphinx-doc.org/rest.html
|
||||||
|
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
|
||||||
|
|
||||||
|
|
||||||
|
====================================
|
||||||
|
Make the new Rally input task format
|
||||||
|
====================================
|
||||||
|
|
||||||
|
Current Rally format is not flexible enough to cover all use cases that are
|
||||||
|
required. Let's change it!
|
||||||
|
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
Why do we need such fundamental change?
|
||||||
|
|
||||||
|
- Multi scenarios load generation support.
|
||||||
|
This is very important, because it will allow to use Rally for more
|
||||||
|
real life load generation. Like making load on different components
|
||||||
|
and HA testing (where one scenario tries for example to authenticate
|
||||||
|
another is disabling controller)
|
||||||
|
|
||||||
|
- Ability to add require meta information like (title and descriptions)
|
||||||
|
That are required to generate clear reports
|
||||||
|
|
||||||
|
- Fixing UX issues. Previous format is very hard for understanding and
|
||||||
|
end users have issues with understanding how it works exactly.
|
||||||
|
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
Make a new format that address all issues.
|
||||||
|
|
||||||
|
|
||||||
|
Old format JSON schema:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
{
|
||||||
|
"type": "object",
|
||||||
|
"$schema": "http://json-schema.org/draft-04/schema",
|
||||||
|
"patternProperties": {
|
||||||
|
".*": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"args": {
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
"runner": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"type": {"type": "string"}
|
||||||
|
},
|
||||||
|
"required": ["type"]
|
||||||
|
},
|
||||||
|
"context": {
|
||||||
|
"type": "object"
|
||||||
|
},
|
||||||
|
"sla": {
|
||||||
|
"type": "object",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"additionalProperties": False
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
Old format sample:
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
---
|
||||||
|
<ScenarioName>:
|
||||||
|
-
|
||||||
|
args: <dict_with_scenario_args>
|
||||||
|
runner: <dict_with_runner_type_and_args>
|
||||||
|
context:
|
||||||
|
<context_name>: <dict_with_context_args>
|
||||||
|
...
|
||||||
|
sla:
|
||||||
|
<sla_name>: <sla_arguments>
|
||||||
|
-
|
||||||
|
-//-
|
||||||
|
-
|
||||||
|
-//-
|
||||||
|
<AnotherScenarioName>:
|
||||||
|
-//-
|
||||||
|
|
||||||
|
Every element of list corresponding to <ScenarioName> is separated task,
|
||||||
|
that generates environment according to context, generates load using
|
||||||
|
specified runner that runs multiple times <ScenarioName> with it's args.
|
||||||
|
|
||||||
|
|
||||||
|
New format JSON schema:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
{
|
||||||
|
"type": "object",
|
||||||
|
"$schema": "http://json-schema.org/draft-04/schema",
|
||||||
|
"properties": {
|
||||||
|
"version": {"type": "number"},
|
||||||
|
"title": {"type": "string"},
|
||||||
|
"description": {"type": "string"},
|
||||||
|
"tags": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {"type": "string"}
|
||||||
|
},
|
||||||
|
|
||||||
|
"subtasks": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"title": {"type": "string"},
|
||||||
|
"description": {"type": "string"},
|
||||||
|
"tags": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {"type": "string"}
|
||||||
|
},
|
||||||
|
|
||||||
|
"run_in_parallel": {"type": "boolean"},
|
||||||
|
"workloads": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"scenario": {"type": "object"},
|
||||||
|
"runner": {"type": "object"}
|
||||||
|
"sla": {"type": "object"},
|
||||||
|
"contexts": {"type": "object"}
|
||||||
|
},
|
||||||
|
"required": ["scenario", "runner"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"contexts": {"type": "object"}
|
||||||
|
},
|
||||||
|
"required": ["title", "workloads"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": ["title", "tasks"]
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
New format sample:
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# Having Dictionary on top level allows us in future to add any new keys.
|
||||||
|
# Keeping the schema of format more or less same for end users.
|
||||||
|
|
||||||
|
# Version of format
|
||||||
|
version: 2
|
||||||
|
|
||||||
|
# Allows to set title of report. Which allows end users to understand
|
||||||
|
# what they can find in task report.
|
||||||
|
title: "New Input Task format"
|
||||||
|
|
||||||
|
# Description allows us to put all required information to explain end
|
||||||
|
# users what kind of results they can find in reports.
|
||||||
|
description: "This task allows you to certify that your cloud works"
|
||||||
|
|
||||||
|
# Explicit usage "rally task start --tag" --tag attribute
|
||||||
|
tags: ["periodic", "nova", "cinder", "ha"]
|
||||||
|
|
||||||
|
subtasks:
|
||||||
|
# Note every task is executed serially (one by one)
|
||||||
|
#
|
||||||
|
# Using list for describing what subtasks to run is much better idea then
|
||||||
|
# using dictionary. It resolves at least 3 big issues:
|
||||||
|
#
|
||||||
|
# 1) Bad user experience
|
||||||
|
# 1.1) Users do not realize that Rally can run N subtask
|
||||||
|
# 1.2) Keys of Dictionary were Scenario names (reasonable question why?!)
|
||||||
|
# 1.3) Users tried to put N times same k-v (to run one subtask N times)
|
||||||
|
# 2) No way to specify order of scenarios execution, especially in case
|
||||||
|
# where we need to do chain like: ScenarioA -> SecnearioB -> ScenarioA
|
||||||
|
# 3) No way to support multi scenario load, because we used scenario name
|
||||||
|
# as a identifier of single task
|
||||||
|
-
|
||||||
|
# title field is required because in case of multi scenario load
|
||||||
|
# we can't use scenario name for it's value.
|
||||||
|
title: "First task to execute"
|
||||||
|
description: "We will stress Nova" # optional
|
||||||
|
|
||||||
|
# Tags are going to be used in various rally task reports for filtering
|
||||||
|
# and grouping.
|
||||||
|
tags: ["nova", "my_favorite_task", "do it"]
|
||||||
|
|
||||||
|
# The way to execute scenarios (one by one or all in parallel)
|
||||||
|
run_in_parallel: False
|
||||||
|
|
||||||
|
# Single scenario load can be generated by specifying only one element
|
||||||
|
# in "workloads" section.
|
||||||
|
workloads:
|
||||||
|
-
|
||||||
|
scenario:
|
||||||
|
NovaServers.boot_and_delete:
|
||||||
|
image:
|
||||||
|
name: "^cirros$"
|
||||||
|
flavors:
|
||||||
|
name: "m1.small"
|
||||||
|
runner:
|
||||||
|
constant:
|
||||||
|
times: 100
|
||||||
|
concurrency: 10
|
||||||
|
# Subtask success of criteria based on results
|
||||||
|
sla:
|
||||||
|
# Every key means SLA plugin name, values are config of plugin
|
||||||
|
# Only if all criteria pass task is marked as passed
|
||||||
|
failure_rate:
|
||||||
|
max: 0
|
||||||
|
|
||||||
|
# Specification of context that creates env for scenarios
|
||||||
|
# E.g. it creates users, tenants, sets quotas, uploads images...
|
||||||
|
contexts:
|
||||||
|
# Each key is the name of context plugin
|
||||||
|
|
||||||
|
# This context creates temporary users and tenants
|
||||||
|
users:
|
||||||
|
# These k-v will be passed as arguments to this `users` plugin
|
||||||
|
tenants: 2
|
||||||
|
users_per_tenant: 10
|
||||||
|
|
||||||
|
# This context set's quotas for created by `users` context tenants
|
||||||
|
quotas:
|
||||||
|
nova:
|
||||||
|
cpu: -1
|
||||||
|
|
||||||
|
-
|
||||||
|
title: "Second task to execute"
|
||||||
|
description: "Multi Scenario load generation with common context"
|
||||||
|
|
||||||
|
run_in_parallel: True
|
||||||
|
|
||||||
|
# If we put 2 or more scenarios to `scenarios` section we will run
|
||||||
|
# all of them simultaneously which allows us to generate more real life
|
||||||
|
# load
|
||||||
|
workloads:
|
||||||
|
-
|
||||||
|
scenario:
|
||||||
|
CinderVolumes.create_and_delete:
|
||||||
|
size: 10
|
||||||
|
runner:
|
||||||
|
constant:
|
||||||
|
times: 100
|
||||||
|
concurrency: 10
|
||||||
|
sla:
|
||||||
|
failure_rate:
|
||||||
|
max: 0
|
||||||
|
-
|
||||||
|
scenario:
|
||||||
|
KeystoneBasic.create_and_delete_users:
|
||||||
|
name_length: 20
|
||||||
|
runner:
|
||||||
|
rps:
|
||||||
|
rps: 1
|
||||||
|
times: 1000
|
||||||
|
sla:
|
||||||
|
max_seconds_per_iteration: 10
|
||||||
|
-
|
||||||
|
scenario:
|
||||||
|
PhysicalNode.restart:
|
||||||
|
ip: "..."
|
||||||
|
user: "..."
|
||||||
|
password: "..."
|
||||||
|
runner:
|
||||||
|
rps:
|
||||||
|
rps: 10
|
||||||
|
times: 10
|
||||||
|
sla:
|
||||||
|
max_seconds_per_iteration: 100
|
||||||
|
# This scenario is called in own independent and isolated context
|
||||||
|
contexts: {}
|
||||||
|
|
||||||
|
# Global context that is used if scenario doesn't specify own
|
||||||
|
contexts:
|
||||||
|
users:
|
||||||
|
tenants: 2
|
||||||
|
users_per_tenant: 10
|
||||||
|
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
No way
|
||||||
|
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
boris-42 aka Boris Pavlovic
|
||||||
|
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
- Implement OLD -> NEW format converter
|
||||||
|
|
||||||
|
- Switch task engine to use new format. This should affect only task engine
|
||||||
|
|
||||||
|
- Implement new DB schema format, that will allow to store multi-scenario
|
||||||
|
output data
|
||||||
|
|
||||||
|
- Add support for multi scenario results processing in rally task
|
||||||
|
detailed|sla_check|report
|
||||||
|
|
||||||
|
- Add timestamps to task, scenarios and atomics
|
||||||
|
|
||||||
|
- Add support for usage multi-runner instance in single task with
|
||||||
|
common context
|
||||||
|
|
||||||
|
- Add support for scenario's own context
|
||||||
|
|
||||||
|
- Add ability to use new format in rally task start.
|
||||||
|
|
||||||
|
- Deprecate OLD format
|
||||||
|
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
None
|
||||||
206
doc/specs/implemented/pluggable_validators.rst
Normal file
206
doc/specs/implemented/pluggable_validators.rst
Normal file
@ -0,0 +1,206 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
..
|
||||||
|
This template should be in ReSTructured text. The filename in the git
|
||||||
|
repository should match the launchpad URL, for example a URL of
|
||||||
|
https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named
|
||||||
|
awesome-thing.rst . Please do not delete any of the sections in this
|
||||||
|
template. If you have nothing to say for a whole section, just write: None
|
||||||
|
For help with syntax, see http://sphinx-doc.org/rest.html
|
||||||
|
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
|
||||||
|
|
||||||
|
|
||||||
|
=================================
|
||||||
|
Rally Task Validation Refactoring
|
||||||
|
=================================
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
* Current validator system is pluggable - but it doesn't use our plugin
|
||||||
|
mechanism which creates problems (e.g. validators are imported directly and
|
||||||
|
used in code, instead of using their names, which doesn't allow to rename
|
||||||
|
them or move without breaking backward compatibility).
|
||||||
|
|
||||||
|
* Current mechanism of validation leads to a lot of OpenStack related code in
|
||||||
|
the Rally task engine.
|
||||||
|
|
||||||
|
* It's hard to use the same validators for different types of plugins, current
|
||||||
|
approach is used only for scenarios.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
To create unified validation mechanism that can be used for all types of
|
||||||
|
future deployments and type of plugins in the same way. So we will be able
|
||||||
|
to remove `OpenStack related code <https://github
|
||||||
|
.com/openstack/rally/blob/be8cd7bff6de9b3e83dd31005ae5d07ca1c86b9e/rally
|
||||||
|
/task/engine.py#L188-L278>`_ from the task engine, and create a bunch of
|
||||||
|
common validators (e.g. jsonschema) that can be used by any
|
||||||
|
plugin.
|
||||||
|
As a bonus of refactoring, it allows us to switch to common mechanism of
|
||||||
|
plugins.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
No way
|
||||||
|
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Here is an example of base class for all pluggable validators.
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
import abc
|
||||||
|
import six
|
||||||
|
|
||||||
|
from rally.common.plugin import plugin
|
||||||
|
from rally.task import validation
|
||||||
|
|
||||||
|
|
||||||
|
def configure(name, platform="default"):
|
||||||
|
return plugin.configure(name=name, platform=platform)
|
||||||
|
|
||||||
|
@six.add_metaclass(abc.ABCMeta)
|
||||||
|
@configure(name="base_validator")
|
||||||
|
class Validator(plugin.Plugin):
|
||||||
|
|
||||||
|
def validate(self, cache, deployment, cfg, plugin_cfg):
|
||||||
|
"""
|
||||||
|
Method that validates something.
|
||||||
|
|
||||||
|
:param cache: this is cross validator cache where different
|
||||||
|
validators could store information about
|
||||||
|
environment like initialized OpenStack clients,
|
||||||
|
images, etc and share it through validators.
|
||||||
|
E.g. if your custom validators need to perform 200
|
||||||
|
OpenStack checks and each validator plugin need to
|
||||||
|
initialize client, Rally will take extra 2 minutes
|
||||||
|
for validation step. As well, its not efficient to
|
||||||
|
fetch all image each time if we have image related
|
||||||
|
validators.
|
||||||
|
:param deployment: Deployment object, deployment which would be
|
||||||
|
used for validation
|
||||||
|
:param cfg: dict, configuration of subtask
|
||||||
|
:param plugin_cfg: dict, with exact configuration of the plugin
|
||||||
|
"""
|
||||||
|
pass
|
||||||
|
|
||||||
|
def add(name, **kwargs):
|
||||||
|
"""
|
||||||
|
Add validator instance to the validator plugin class meta.
|
||||||
|
|
||||||
|
Get validator class by name. Initialize an instance. Add validator
|
||||||
|
instance to validators list stored in the Validator meta by
|
||||||
|
'validator_v2' key. This would be used to iterate and execute through
|
||||||
|
all validators used during execution of subtask.
|
||||||
|
|
||||||
|
:param kwargs: dict, arguments used to initialize validator class
|
||||||
|
instance
|
||||||
|
:param name: str, name of the validator plugin
|
||||||
|
"""
|
||||||
|
validator = Validator.get(name)(**kwargs)
|
||||||
|
|
||||||
|
def wrapper(p):
|
||||||
|
p._meta_setdefault("validators_v2", [])
|
||||||
|
p._meta_get("validators_v2").append(validator)
|
||||||
|
return p
|
||||||
|
|
||||||
|
return wrapper
|
||||||
|
|
||||||
|
|
||||||
|
@abc.abstractmethod
|
||||||
|
def validate(plugin, deployment, cfg, plugin_cfg):
|
||||||
|
"""
|
||||||
|
Execute all validate() method of all validators stored in meta of
|
||||||
|
Validator.
|
||||||
|
|
||||||
|
Iterate during all validators stored in the meta of Validator and
|
||||||
|
execute proper validate() method and add validation result to the
|
||||||
|
list.
|
||||||
|
|
||||||
|
:param plugin: is plugin class instance that has validators and should
|
||||||
|
be validated
|
||||||
|
:param deployment: Deployment object, deployment which would be
|
||||||
|
used for validation
|
||||||
|
:param cfg: dict, configuration of subtask
|
||||||
|
:param plugin_cfg: dict, with exact configuration of the plugin
|
||||||
|
"""
|
||||||
|
results = []
|
||||||
|
cache = {}
|
||||||
|
|
||||||
|
for v in plugin._meta_get("validators_v2"):
|
||||||
|
try:
|
||||||
|
v.validate(cache, deployment, cfg, plugin_cfg)
|
||||||
|
except Exception as e:
|
||||||
|
results.append(validation.ValidationResult(is_valid=False,
|
||||||
|
msg=e))
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
New design allows us to use the same validator and same validation mechanism
|
||||||
|
for different types of plugins (context, sla, runner, scenarios) which was not
|
||||||
|
possible before. For example, we could implement jsonschema validation as a
|
||||||
|
plugin.
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
import jsonschema
|
||||||
|
|
||||||
|
@configure(name="jsonschema")
|
||||||
|
class JsonSchemaValidator(Validator):
|
||||||
|
|
||||||
|
def __init__(self, schema=None):
|
||||||
|
super(JsonSchemaValidator, self).__init__()
|
||||||
|
self.schema = schema or {}
|
||||||
|
|
||||||
|
def validate(self, cache, deployment, cfg, plugin_cfg):
|
||||||
|
jsonschema.validate(plugin_cfg, self.schema)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@validator.add("jsonschema", schema="<here_json_schema>")
|
||||||
|
class SomeContext(base.Context):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class SomeScenario(base.Scenario):
|
||||||
|
|
||||||
|
@validator.add("jsonschema", schema="<here_json_schema>")
|
||||||
|
def some_function(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
|
||||||
|
- boris-42 <bpavlovic@mirantis.com>
|
||||||
|
- rvasilets <rvasilets@mirantis.com>
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
- Create validation module with base plugin and method of adding validators
|
||||||
|
|
||||||
|
- Add support to task engine of new validation mechanism
|
||||||
|
|
||||||
|
- Port all old validators to new mechanism
|
||||||
|
|
||||||
|
- Deprecate old validation mechanism
|
||||||
|
|
||||||
|
- Remove deprecated in new release
|
||||||
|
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
None
|
||||||
109
doc/specs/implemented/task_and_verification_export.rst
Normal file
109
doc/specs/implemented/task_and_verification_export.rst
Normal file
@ -0,0 +1,109 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
..
|
||||||
|
This template should be in ReSTructured text. The filename in the git
|
||||||
|
repository should match the launchpad URL, for example a URL of
|
||||||
|
https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named
|
||||||
|
awesome-thing.rst . Please do not delete any of the sections in this
|
||||||
|
template. If you have nothing to say for a whole section, just write: None
|
||||||
|
For help with syntax, see http://sphinx-doc.org/rest.html
|
||||||
|
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
|
||||||
|
|
||||||
|
|
||||||
|
====================================================
|
||||||
|
Export task and verifications into external services
|
||||||
|
====================================================
|
||||||
|
|
||||||
|
Currently Rally stores all information about executed tasks and verifications
|
||||||
|
in its database and it is also able to provide this data in JSON format or
|
||||||
|
in the form of HTML reports. There is a request for Rally to export this data
|
||||||
|
into external services (like test management system or Google Docs)
|
||||||
|
via its API.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
There are many, including a lot of proprietary, test management systems
|
||||||
|
in the market available as SaaS and/or On-Premises, like TestRail, TestLink,
|
||||||
|
TestLodge etc, which objective is to manage, organize and track all testing
|
||||||
|
efforts.
|
||||||
|
|
||||||
|
Most of the systems provide an API for importing test data. The systems also
|
||||||
|
possess data model somewhat similar to Rally's one.
|
||||||
|
It usually includes (among others) models for project, test suite test case,
|
||||||
|
test plan and test execution results.
|
||||||
|
|
||||||
|
It is suggested to provide Rally users an ability to export information about
|
||||||
|
testing their environments into such test management systems in order
|
||||||
|
to integrate testing via Rally into rest of their testing activities.
|
||||||
|
|
||||||
|
Since different test management systems have alike yet different API
|
||||||
|
for the purpose it is reasonable to implement this export functionality via
|
||||||
|
plugins.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
1. Implement a base class Exporter for an export plugin at
|
||||||
|
*rally/task/exporter.py*.
|
||||||
|
|
||||||
|
..code-block:: python
|
||||||
|
|
||||||
|
class Exporter(plugin.Plugin):
|
||||||
|
def export(self, task, connection_string):
|
||||||
|
...
|
||||||
|
|
||||||
|
2. Implement a CLI command of the form
|
||||||
|
|
||||||
|
..code-block:: shell
|
||||||
|
|
||||||
|
rally task export <UUID> <CONNECTION_STRING>
|
||||||
|
|
||||||
|
3. Implement a base class VerifyExporter for an export plugin at
|
||||||
|
*rally/verify/exporter.py*.
|
||||||
|
|
||||||
|
..code-block:: python
|
||||||
|
|
||||||
|
class VerifyExporter(plugin.Plugin):
|
||||||
|
def export(self, verification, connection_string):
|
||||||
|
...
|
||||||
|
|
||||||
|
4. Implement a CLI command of the form
|
||||||
|
|
||||||
|
..code-block:: shell
|
||||||
|
|
||||||
|
rally verify export <UUID> <CONNECTION_STRING>
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
No way
|
||||||
|
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
|
||||||
|
rvasilets <rvasilets@mirantis.com>
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
- Implement plugin base class
|
||||||
|
|
||||||
|
- Implement CLI command
|
||||||
|
|
||||||
|
- Implement plugin for TestRail
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
None
|
||||||
Loading…
x
Reference in New Issue
Block a user