Provide efficacy indicators
When the admin trigger an audit using Watcher, we provide a list of actions that should be run to attain a specific goal (defined in the audit template), Today, there is no incentive to give a predicted result of the optimization after running the actions plan. This feature will give the admin an estimated gain of running Watcher on the infrastructure. Change-Id: I774b6bde9329135d7cb9c85387ea40f9ab59d49f Implements: blueprint bp/efficacy-indicator
This commit is contained in:
vmahe
parent
cf70ad13ab
commit
34946cc8a4
@ -1,36 +1,47 @@
|
||||
@startuml
|
||||
class "Base Strategy" {
|
||||
String goal
|
||||
ActionPlan execute()
|
||||
#Goal _goal
|
||||
#Solution _solution
|
||||
#String _name
|
||||
#String _display_name
|
||||
#boolean _pre_execute()
|
||||
#boolean _do_execute()
|
||||
#boolean _post_execute()
|
||||
+String get_name()
|
||||
+String get_display_name()
|
||||
+String get_goal_name()
|
||||
+String get_goal_display_name()
|
||||
+Solution execute()
|
||||
}
|
||||
class Goal {
|
||||
+String get_name()
|
||||
+String get_display_name()
|
||||
}
|
||||
class "Energy Base Strategy" {
|
||||
String goal="Reduce Energy"
|
||||
}
|
||||
class "IO Base Strategy" {
|
||||
String goal="Reduce Disk IO"
|
||||
}
|
||||
class "Network Bandwidth Base Strategy" {
|
||||
String goal="Reduce Network Bandwidth"
|
||||
}
|
||||
class IndicatorSpecification {
|
||||
String name
|
||||
String description
|
||||
String type
|
||||
String unit
|
||||
String valueRange
|
||||
boolean isMandatory = true
|
||||
-String _name
|
||||
-String _description
|
||||
-String _type
|
||||
-String _unit
|
||||
-String _value_range
|
||||
-boolean _is_mandatory = true
|
||||
}
|
||||
class EnergyEfficacySpecification {
|
||||
ArrayList : IndicatorSpecification[] indicatorsDescription
|
||||
boolean checkEfficacyIndicators(ActionPlan)
|
||||
Object getGlobalEfficacy(ActionPlan)
|
||||
-Dictionary : IndicatorSpecification[] _indicators_description
|
||||
+boolean check_efficacy_indicators(Solution)
|
||||
+Object get_global_efficacy(Solution)
|
||||
}
|
||||
class Efficacy {
|
||||
HashMap : IndicatorValue[] indicators
|
||||
-Dictionary : IndicatorValue[] _indicators
|
||||
}
|
||||
class IndicatorValue {
|
||||
String name
|
||||
Object value
|
||||
-String _name
|
||||
-Object _value
|
||||
}
|
||||
"Base Strategy" <|-- "IO Base Strategy"
|
||||
"Base Strategy" <|-- "Energy Base Strategy"
|
||||
@ -38,10 +49,12 @@ class IndicatorValue {
|
||||
"Energy Base Strategy" <|-- "Energy Strategy A"
|
||||
"Energy Base Strategy" <|-- "Energy Strategy B"
|
||||
"Energy Base Strategy" <|-- "Energy Strategy C"
|
||||
"Energy Base Strategy" --> "EnergyEfficacySpecification"
|
||||
Goal --> "EnergyEfficacySpecification" : is evaluated with
|
||||
EnergyEfficacySpecification "1" *-- "many" IndicatorSpecification
|
||||
"Base Strategy" ..> ActionPlan : generates
|
||||
ActionPlan --> Efficacy
|
||||
"Base Strategy" --> Solution : generates
|
||||
Solution --> Efficacy
|
||||
Efficacy "1" *-- "many" IndicatorValue
|
||||
"Base Strategy" --> Goal : achieves
|
||||
note top of "Energy Base Strategy" : In this base class, get_goal_name() \
|
||||
\nreturns "REDUCE_ENERGY".
|
||||
@enduml
|
||||
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 44 KiB After Width: | Height: | Size: 62 KiB |
@ -26,19 +26,23 @@ Problem description
|
||||
Today, Watcher does not provide efficacy indicators to give more accuracy to
|
||||
an action plan. An administrator would require to have an idea of the level
|
||||
of optimization Watcher can provide. The indicators must be related to the
|
||||
initial goal set in the audit template,for example if the goal is "thermal
|
||||
initial goal set in the audit template, for example if the goal is "thermal
|
||||
optimization", we should provide an estimated average inlet temperature of the
|
||||
cluster (or an estimated decrease in % of the average inlet temperature).
|
||||
If the `Strategy`_ applied is "basic consolidation", we should be able to
|
||||
provide an estimated average CPU load after running the optimization.
|
||||
These indicators must be computed by the chosen strategy.
|
||||
|
||||
There should also be a way to compare the efficacy of different strategies for
|
||||
a given optimization `goal`_. Therefore, all the strategies associated to a
|
||||
given goal should provide the same efficacy indicators so that the
|
||||
`Administrator`_ can decide which strategy is the best. In the longer run, it
|
||||
will also enable Watcher to automatically decide which strategy to use (via
|
||||
some *StrategySelector* component in the `Watcher Decision Engine`_).
|
||||
In later versions of Watcher, there should also be a way to compare the
|
||||
efficacy of different `Strategies`_ for a given optimization `goal`_.
|
||||
Therefore, all the strategies associated to a given goal should provide the
|
||||
same efficacy indicators so that the `Administrator`_ can decide which strategy
|
||||
is the best.
|
||||
|
||||
In an even longer run, it will enable Watcher to automatically decide:
|
||||
|
||||
* which strategy to use (via some *StrategySelector* component in the
|
||||
`Watcher Decision Engine`_). See `Strategy Selector blueprint`_.
|
||||
* or which planner implementation is the most efficient. See
|
||||
`Planner Selector blueprint`_.
|
||||
|
||||
|
||||
Use Cases
|
||||
@ -48,8 +52,8 @@ As an administrator, I would like to have efficacy indicators to be able to
|
||||
judge if the proposed optimization fit my objectives before running the
|
||||
recommended action plan.
|
||||
|
||||
As an administrator, I would like to be able to compare the efficacy of
|
||||
several strategies for the same optimization goal
|
||||
As an administrator, I would like to be able to compare in a fair way the
|
||||
efficacy of several strategies for the same optimization goal
|
||||
|
||||
As a strategy developer, I will provide a list of indicators that can be
|
||||
evaluated during the execution of the audit and displayed to the administrator
|
||||
@ -58,6 +62,13 @@ with the corresponding action plan.
|
||||
As a developer, I would like to make sure that all strategies associated to
|
||||
a given goal provide the same list of efficacy indicators.
|
||||
|
||||
As an administrator, I would like to be able to update a given goal and its
|
||||
associated optimization strategies. In that case, all the previously launched
|
||||
audits and action plans which are still in recommended or pending state should
|
||||
be cancelled (in order to make it clear that the estimated efficacy indicators
|
||||
are not relevant anymore and that a new audit should be launched with the new
|
||||
deployed version).
|
||||
|
||||
|
||||
Project Priority
|
||||
-----------------
|
||||
@ -67,9 +78,12 @@ Not relevant because Watcher is not in the big tent so far.
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
An hashmap should be added to the `action plan`_ providing a list of efficacy
|
||||
indicators with a name and a value (the name of the indicator is used as the
|
||||
key for the hash map and should be unique).
|
||||
A dictionary should be added to the `action plan`_ providing a list of efficacy
|
||||
indicators with a name and a value (the name is used as the key for each
|
||||
indicator and should be unique).
|
||||
|
||||
The same dictionary should also be added to the `solution`_ generated by
|
||||
each `Strategy`_ (which contains a non scheduled list of actions).
|
||||
|
||||
We propose to use the `DDD Specification Pattern`_ to express the constraints
|
||||
regarding efficacy indicators for a given `goal`_.
|
||||
@ -88,19 +102,57 @@ contain the following properties for each expected efficacy indicator:
|
||||
|
||||
Each strategy associated to a given XYZ goal should have access to a
|
||||
singleton of the **XYZEfficacySpecification** class in order to make sure
|
||||
the generated action plan contains the expected efficacy indicators. In other
|
||||
the generated solution contains the expected efficacy indicators. In other
|
||||
words, in each strategy, it should be possible to call the
|
||||
**checkEfficacyIndicators(ActionPlan)** method of the
|
||||
**check_efficacy_indicators(Solution)** method of the
|
||||
**XYZEfficacySpecification** which returns a boolean saying whether all
|
||||
mandatory efficacy indicators are provided in a given `action plan`_ and
|
||||
whether the indicator values are compliant with the expected types and ranges.
|
||||
mandatory efficacy indicators are provided in a given `solution`_ and
|
||||
whether the indicator values are compliant with the expected types and ranges
|
||||
(we may rely on `Voluptuous Python data validation library`_ to manage the
|
||||
schema definition and validation of indicators).
|
||||
|
||||
It should be possible to call a **getGlobalEfficacy(ActionPlan)** method from
|
||||
the **XYZEfficacySpecification** class which returns a global score of efficacy
|
||||
for a given `action plan`_. This global score would be computed using a
|
||||
ponderated sum of the different efficacy indicators.
|
||||
It should be possible to call a **get_global_efficacy(Solution)** method from
|
||||
the **XYZEfficacySpecification** class (or directly from the **XYZGoal** class)
|
||||
which returns a global score of efficacy for a given `solution`_. This global
|
||||
score would be computed using a weighted sum of the different efficacy
|
||||
indicators.
|
||||
|
||||
Below you will find a class diagram showing the hierarchy of Strategies for
|
||||
For example, for the **BASIC_CONSOLIDATION** goal, the following efficacy
|
||||
indicators would be returned:
|
||||
|
||||
- **nb_migrations**: the number of virtual machine migrations
|
||||
- **nb_empty_nodes**: the number of empty nodes (i.e. where all the virtual
|
||||
machines have been migrated to some other machines).
|
||||
|
||||
And the global score of efficacy would be computed with the following formula:
|
||||
|
||||
::
|
||||
|
||||
global_score = (nb_empty_nodes/nb_migrations*100)
|
||||
|
||||
Of course, in this case, we should handle the case when **nb_migrations** is 0.
|
||||
|
||||
Note that for simpler implementation, it is also possible to put directly these
|
||||
methods in the **XYZGoal** class (which would then act as the specification
|
||||
class for the XYZ goal efficacy).
|
||||
|
||||
Note also that whenever a **XYZGoal** class is updated, Watcher should
|
||||
automatically change to a **CANCELLED** state the previously generated audit
|
||||
and action plans . This would make sure that we can handle the consistency
|
||||
between the indicators described in the goal and those returned in the
|
||||
`action plan`_. See `State machine diagrams`_ for more details on the different
|
||||
possible states of the audit or an action plan objects.
|
||||
|
||||
All those efficacy indicators are set in the **_post_execute()** method of the
|
||||
strategy, i.e. just after a `solution`_ has been found in the **_do_execute()**
|
||||
method.
|
||||
|
||||
Note that the **_pre_execute()** method of each strategy will be the place
|
||||
where all necessary data is fetched and prepared in order to compute a
|
||||
`solution`_ (i.e. build a model of the data-center, get some previously
|
||||
stored metrics, ...).
|
||||
|
||||
Below you will find a class diagram showing the hierarchy of `Strategies`_ for
|
||||
several goals and how they are related to efficacy specification classes:
|
||||
|
||||
.. image:: ../../../doc/source/images/class_diagram_efficacy_indicator.png
|
||||
@ -124,7 +176,19 @@ The following data object will be impacted:
|
||||
* **ActionPlan**:
|
||||
|
||||
* We may need to store in the database a list of efficacy indicators
|
||||
associated with the action plan (hashmap)
|
||||
associated with the action plan
|
||||
|
||||
A new **Goal** object should be stored in the Watcher database with the
|
||||
following information:
|
||||
|
||||
* the unique name of the goal
|
||||
* the translated name of the display name of the goal
|
||||
* the list of efficacy indicator descriptions. Note that the regarding the
|
||||
name, unit and description of each indicator, the database stores the i18n
|
||||
unique name of the string, not the translated version.
|
||||
|
||||
A new table should also be created in order to store the specification of each
|
||||
efficacy indicator and reference the goal that uses it.
|
||||
|
||||
|
||||
REST API impact
|
||||
@ -140,12 +204,11 @@ There will be an impact on every REST resource URLs that starts with
|
||||
* PATCH /v1/action_plans
|
||||
* GET /v1/action_plans/detail
|
||||
|
||||
The type **ActionPlan** will contain a new **efficacy** object with a hashmap
|
||||
of indicators.
|
||||
The type **ActionPlan** will contain a new **efficacy** object with a
|
||||
dictionary of indicators.
|
||||
|
||||
Here is a sample of the new JSON payload for an action plan which includes
|
||||
the **efficacy** object composed of one global efficacy indicator and three
|
||||
detailed indicators:
|
||||
one global efficacy indicator and three detailed indicators:
|
||||
|
||||
::
|
||||
|
||||
@ -167,11 +230,8 @@ detailed indicators:
|
||||
"state": "ONGOING",
|
||||
"updated_at": "2016-02-08T12:59:33.445869",
|
||||
"uuid": "9ef4d84c-41e8-4418-9220-ce55be0436af",
|
||||
"efficacy" : [
|
||||
{
|
||||
"name" : "Global efficacy score",
|
||||
"value" : "89"
|
||||
},
|
||||
"global_efficacy": 89,
|
||||
"efficacy_indicators" : [
|
||||
{
|
||||
"name" : "Average CPU load",
|
||||
"value" : "22"
|
||||
@ -196,18 +256,20 @@ There will also be an impact on every REST resource URLs that starts with
|
||||
* GET /v1/goals/(goal_uuid)
|
||||
* GET /v1/goals/detail
|
||||
|
||||
The type **Goal** will contain a new **efficacy_specification** object with a
|
||||
hashmap of indicator descriptions.
|
||||
Each **Goal** object will be a dictionary with a **goal_name** and an
|
||||
**efficacy_specification** which contains an array of indicator
|
||||
descriptions.
|
||||
|
||||
Here is a sample of the new JSON payload for a goal which includes
|
||||
the **efficacy_specification** object composed of 4 indicators:
|
||||
the **efficacy_specification** object composed of four indicators:
|
||||
|
||||
::
|
||||
|
||||
{
|
||||
"goals": [
|
||||
{
|
||||
"name": "Reduce Energy Consumption",
|
||||
"goal_name": "REDUCE_ENERGY",
|
||||
"goal_display_name": "Reduce Energy Consumption",
|
||||
"efficacy_specification" : [
|
||||
{
|
||||
"name" : "Relative energy gain",
|
||||
@ -244,6 +306,11 @@ the **efficacy_specification** object composed of 4 indicators:
|
||||
]
|
||||
}
|
||||
|
||||
Note that, from the user point of view, the link between indicators returned in
|
||||
the **Goal** object and in the **ActionPlan** object can be done through the
|
||||
name of the indicator (which is unique for a given `goal`_).
|
||||
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
@ -268,7 +335,7 @@ Performance Impact
|
||||
|
||||
The calculation of efficacy indicators will be done by the targeted
|
||||
strategy, we should keep in mind that this calculation must not add delay to
|
||||
the building of the actions plan.
|
||||
the building of the `solution`_.
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
@ -298,26 +365,73 @@ Work Items
|
||||
|
||||
Here is the list of foreseen work items:
|
||||
|
||||
* add a field to action-plan object to store the list of efficacy indicators
|
||||
* add a new **Goal** object in **/watcher/db/sqlalchemy/models.py**
|
||||
* add a new **EfficacyIndicatorSpecification** object in
|
||||
*/watcher/db/sqlalchemy/models.py**
|
||||
* add a new **goal.py** class in the **/watcher/objects/** package
|
||||
* add a dictionary to **BaseSolution** and **ActionPlan** classes to store the
|
||||
list of efficacy indicator values.
|
||||
* the **ActionPlan** class should be able to read/write the efficacy indicator
|
||||
values from/to the `Watcher database`_ (see **/db/api.py** and
|
||||
**/db/sqlalchemy/api.py** classes).
|
||||
* the **BaseSolution** class should have a new method named
|
||||
**set_efficacy_values()** which enables a `Strategy`_ to set the list of
|
||||
efficacy indicator values in the `solution`_ that it builds.
|
||||
* implement an example of efficacy specification class which contains a
|
||||
description of the expected efficacy indicators for a given goal and the
|
||||
needed methods:
|
||||
|
||||
- **checkEfficacyIndicators(ActionPlan)**
|
||||
- **getGlobalEfficacy(ActionPlan)**
|
||||
- **check_efficacy_indicators(Solution)** which returns a boolean. The
|
||||
developer may use the `Voluptuous Python data validation library`_ to
|
||||
define the constraints on the efficacy indicators in this method.
|
||||
- **get_global_efficacy(Solution)** which returns the computed global
|
||||
efficacy value.
|
||||
* implement a base class dedicated to a goal that will hold a list of possible
|
||||
strategies and an efficacy specification.
|
||||
`Strategies`_ and an efficacy specification.
|
||||
* when the `Watcher Decision Engine`_ service is started, Watcher should
|
||||
browse the list of available goal unique names, get for each goal the list of
|
||||
efficacy indicator descriptions and store them in the `Watcher database`_.
|
||||
* make sure that when a new version of a goal is deployed, when the
|
||||
`Watcher Decision Engine`_ service is restarted, all recommended and pending
|
||||
action_plan and audit are changed to a **CANCELLED** state.
|
||||
* update all existing strategies so that they define three private methods
|
||||
which are automatically called from the public **execute()** method (i.e. use
|
||||
a `Template method design pattern`_):
|
||||
|
||||
- **_pre_execute()**: which prepares the data needed by the optimization
|
||||
algorithm.
|
||||
- **_do_execute()**: which computes the `solution`_. This is where the main
|
||||
optimization algorithm is executed and where the efficacy indicators are
|
||||
computed.
|
||||
- **_post_execute()**: which can do some cleaning task but, more importantly,
|
||||
**this is where efficacy indicators are set**.
|
||||
|
||||
* update the Watcher devstack plugin setup to adapt it (remove the auto
|
||||
watcher_goals config setup).
|
||||
* update **python-watcherclient** to provide extra information regarding
|
||||
efficacy indicators when requesting action plans or goals.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
There are some dependencies with the following blueprint:
|
||||
There are some dependencies with the following blueprints:
|
||||
|
||||
* https://blueprints.launchpad.net/watcher/+spec/get-goal-from-strategy : if
|
||||
there is a base class **XYZBaseStrategy** for all strategies associated to
|
||||
the same XYZ goal, the **XYZEfficacySpecification** should be associated to
|
||||
this base class.
|
||||
* https://blueprints.launchpad.net/watcher/+spec/get-goal-from-strategy :
|
||||
there should be a common base class **XYZBaseStrategy** for all `Strategies`_
|
||||
associated to the same **XYZGoal**.
|
||||
* https://blueprints.launchpad.net/watcher/+spec/optimization-threshold : this
|
||||
blueprint deals with input parameters provided to `Strategies`_ and there
|
||||
may be some common impacts with this blueprint which deals with output
|
||||
parameters generated by `Strategies`_.
|
||||
* https://blueprints.launchpad.net/watcher/+spec/persistent-audit-parameters :
|
||||
this blueprint must be implemented so that it is possible to get from the
|
||||
database the information regarding the goal and strategy to which the
|
||||
indicators of the action plan were related (hence the specification of these
|
||||
indicators) when the audit was launched.
|
||||
* `Strategy Selector blueprint`_ : this component will probably use the
|
||||
efficacy indicators to automatically select the best strategy.
|
||||
* `Planner Selector blueprint`_ : this component will probably use the
|
||||
efficacy indicators to automatically select the best scheduler.
|
||||
|
||||
There is also a dependency with the following bug:
|
||||
|
||||
@ -340,7 +454,8 @@ Testing
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
None
|
||||
The developer guide should explain how to specify efficacy indicators for a
|
||||
given goal.
|
||||
|
||||
|
||||
References
|
||||
@ -359,6 +474,14 @@ None
|
||||
.. _goal: https://factory.b-com.com/www/watcher/doc/watcher/glossary.html#goal
|
||||
.. _audit template: http://factory.b-com.com/www/watcher/doc/watcher/glossary.html#audit-template
|
||||
.. _action plan: https://factory.b-com.com/www/watcher/doc/watcher/glossary.html#action-plan
|
||||
.. _solution: https://factory.b-com.com/www/watcher/doc/watcher/glossary.html#solution
|
||||
.. _Strategy: https://factory.b-com.com/www/watcher/doc/watcher/glossary.html#strategy
|
||||
.. _Strategies: https://factory.b-com.com/www/watcher/doc/watcher/glossary.html#strategy
|
||||
.. _Watcher Decision Engine: https://factory.b-com.com/www/watcher/doc/watcher/architecture.html#watcher-decision-engine
|
||||
.. _Watcher database: https://factory.b-com.com/www/watcher/doc/watcher/architecture.html#watcher-database
|
||||
.. _DDD Specification Pattern: http://martinfowler.com/apsupp/spec.pdf
|
||||
.. _Voluptuous Python data validation library: https://github.com/alecthomas/voluptuous
|
||||
.. _Template method design pattern: https://en.wikipedia.org/wiki/Template_method_pattern
|
||||
.. _Planner Selector blueprint: https://blueprints.launchpad.net/watcher/+spec/watcher-planner-selector
|
||||
.. _Strategy Selector blueprint: https://blueprints.launchpad.net/watcher/+spec/watcher-strategy-selector
|
||||
.. _State machine diagrams: https://factory.b-com.com/www/watcher/doc/watcher/architecture.html#state-machine-diagrams
|
||||
Loading…
x
Reference in New Issue
Block a user