Initial class structure definitions

Change-Id: I2c140d38ebdd5fe4739629692b90fc1ed12ed985
This commit is contained in:
Geir Horn 2023-12-25 19:03:51 +01:00
parent 2af114e479
commit 5ee0dbcad6
6 changed files with 604 additions and 0 deletions

33
AMPLSolver.hpp Normal file
View File

@ -0,0 +1,33 @@
/*==============================================================================
AMPL Solver
This instantiates the purely virtual Solver providing an interface to the
mathematical domain specific language "A Mathematical Programming Language"
(AMPL) [1] allowing to use the wide range of solvers, free or commercial, that
support AMPL descriptions of the optimisation problem.
The AMPL problem description and associated data files are received by handlers
and stored locally as proper files to ensure that the problem is always solved
for the lates problem descriptions received. When the actor receives an
Application Execution Context message, the AMPL description and data files will
be loaded and the appropriate solver called from the AMPL library. When the
solution is returned from AMPL, the solution message is returned to the sender
of the application execution context message, typically the Solution Manager
actor for publishing the solution to external subscribers.
References:
[1] https://ampl.com/
Author and Copyright: Geir Horn, University of Oslo
Contact: Geir.Horn@mn.uio.no
License: MPL2.0 (https://www.mozilla.org/en-US/MPL/2.0/)
==============================================================================*/
#ifndef NEBULOUS_AMPL_SOLVER
#define NEBULOUS_AMPL_SOLVER
namespace NebulOuS
{
} // namespace NebulOuS
#endif // NEBULOUS_AMPL_SOLVER

69
MetricUpdater.cpp Normal file
View File

@ -0,0 +1,69 @@
/*==============================================================================
Metric Updater
This file implements the methods of the Metric Updater class subscribing to
the relevant metric values of the application and publishes a data file with
the metric values when a new solution is requested.
Author and Copyright: Geir Horn, University of Oslo
Contact: Geir.Horn@mn.uio.no
License: MPL2.0 (https://www.mozilla.org/en-US/MPL/2.0/)
==============================================================================*/
#include "ranges" // Better containers
#include <source_location> // Making informative error messages
#include <sstream> // To format error messages
#include <stdexcept> // standard exceptions
#include "Communication/AMQ/AMQEndpoint.hpp" // For Topic subscriptions
#include "MetricUpdater.hpp"
namespace NebulOuS
{
// --------------------------------------------------------------------------
// Subscribing to metric prediction values
// --------------------------------------------------------------------------
//
// The received message must be a JSON object with metric names as
// attribute (keys) and the topic name as the value. Multiple metrics maby be
// included in the same message and and the andler will iterate and set up a
// subcription for each of the provided metrics. It should be noted that
// initially the metric has no value, and it is a prerequisite that all
// metric values must be updated before the complete set of metrics will be
// used for finding a better configuration for the application's execution
// context given by the metric values.
void MetricUpdater::AddMetricSubscription( const MetricTopic & TheMetrics,
const Address OptimiserController )
{
if( TheMetrics.is_object() )
for( const auto & [MetricName, TopicName] : TheMetrics.items() )
{
auto [ MetricRecord, NewMetric ] = MetricValues.try_emplace( TopicName,
MetricName, JSON() );
if( NewMetric )
Send( Theron::AMQ::NetworkLayer::TopicSubscription(
Theron::AMQ::NetworkLayer::TopicSubscription::Action::Subscription,
TopicName ),
Theron::AMQ::Network::GetAddress( Theron::Network::Layer::Session) );
}
else
{
std::source_location Location = std::source_location::current();
std::ostringstream ErrorMessage;
ErrorMessage << "[" << Location.file_name() << " at line " << Location.line()
<< "in function " << Location.function_name() <<"] "
<< "The message to define a new metric subscription is given as "
<< std::endl << TheMetrics.dump(2) << std::endl
<< "this is not a JSON object!";
throw std::invalid_argument( ErrorMessage.str() );
}
}
} // End name space NebulOuS

251
MetricUpdater.hpp Normal file
View File

@ -0,0 +1,251 @@
/*==============================================================================
Metric Updater
This is an actor that gets the metrics and the corresponding AMQ topics where
predictions for the metrics will be published, and subscribes to updates for
these metric predictions. When a new prediction is provided it is initially
just recorded. When the SLO Violation Detector decides that a new application
configuration is necessary to maintain the application within the feasible
region, it will issue a message to indicate that a new solution is necessary.
The Metric Updater will respond to this message by generating a data file
containing the parameters and their values and publish this data file. This
will then be used by the solver to find the optimal configuration for the
the given application execution context defined by the recorded set of metric
value predictions.
The metrics of the application execution context can either be sent as one
JSON message where the attributes are the metric names to be used in the
optimisation data file and the values are the metric topic paths, or as a
sequence of messages, one per metric.
The metric values are sent using messages as defined by the Event Management
System (EMS) [1], and the format of the data file generated for the solver
follows the AMPL data file format [2]. The message from the SLO Violation
Detector is supposed to be a message for Event V since all the predicted
metric values are already collected [3]. In future versions it may be possible
to also use the computed erro bounds for the metric predictions of the other
SLO Violation events calculated.
References:
[1] https://openproject.nebulouscloud.eu/projects/nebulous-collaboration-hub/wiki/monitoringdata-interface
[2] https://ampl.com/wp-content/uploads/Chapter-9-Specifying-Data-AMPL-Book.pdf
[3] https://openproject.nebulouscloud.eu/projects/nebulous-collaboration-hub/wiki/slo-severity-based-violation-detector
Author and Copyright: Geir Horn, University of Oslo
Contact: Geir.Horn@mn.uio.no
License: MPL2.0 (https://www.mozilla.org/en-US/MPL/2.0/)
==============================================================================*/
#ifndef NEBULOUS_METRIC_UPDATE
#define NEBULOUS_METRIC_UPDATE
// Standard headers
#include <string_view> // Constant strings
#include <unordered_map> // To store metric-value maps
// Other packages
#include <nlohmann/json.hpp> // JSON object definition
using JSON = nlohmann::json; // Short form name space
// Theron++ files
#include "Actor.hpp" // Actor base class
#include "Utility/StandardFallbackHandler.hpp" // Exception unhanded messages
#include "Communication/NetworkingActor.hpp" // Actor to receive messages
#include "Communication/PolymorphicMessage.hpp" // The network message type
// AMQ communication files
#include "Communication/AMQ/AMQjson.hpp" // For JSON metric messages
#include "Communication/AMQ/AMQEndpoint.hpp" // AMQ endpoint
#include "Communication/AMQ/AMQSessionLayer.hpp" // For topic subscriptions
namespace NebulOuS
{
// Definitions for the terminology to facilitate changing the lables of the
// various message labels without changing the code. The definitions are
// compile time constants and as such should not lead to any run-time overhead.
constexpr std::string_view ValueLabel{ "metricValue" };
constexpr std::string_view TimePoint { "predictionTime" };
// The topic used for receiving the message(s) defining the metrics of the
// application execution context as published by the Optimiser Controller is
// defined next.
constexpr std::string_view MetricSubscriptions{ "ApplicationContext" };
// The metric value messages will be published on different topics and to
// check if an inbound message is from a metric value topic, it is necessary
// to test against the base string for the metric value topics according to
// the Wiki-page at
// https://openproject.nebulouscloud.eu/projects/nebulous-collaboration-hub/wiki/monitoringdata-interface
constexpr std::string_view MetricValueRootString{
"eu.nebulouscloud.monitoring.predicted"
};
/*==============================================================================
Metric Updater
==============================================================================*/
//
// The Metric Updater actor is an Networking Actor supporting the AMQ message
// exchange.
class MetricUpdater
: virtual public Theron::Actor,
virtual public Theron::StandardFallbackHandler,
virtual public Theron::NetworkingActor<
typename Theron::AMQ::Message::PayloadType >
{
private:
using ProtocolPayload = Theron::PolymorphicMessage<
typename Theron::AMQ::Message::PayloadType >::PayloadType;
// --------------------------------------------------------------------------
// Metric value registry
// --------------------------------------------------------------------------
//
// The metric values are stored essentially as a JSON values where the
// attributes are the metric names and the values are JSON values because
// they are polymorphic with respect to different variable types, and as
// they arrive as JSON values this avoids converting the values on input and
// output. The metric optimisation name is just a string.
private:
class MetricValueRecord
{
public:
const std::string OptimisationName;
JSON Value;
MetricValueRecord( const std::string & TheName, JSON InitialValue )
: OptimisationName( TheName ), Value( InitialValue )
{}
MetricValueRecord( const MetricValueRecord & Other )
: OptimisationName( Other.OptimisationName ), Value( Other.Value )
{}
MetricValueRecord() = delete;
~MetricValueRecord() = default;
};
// This value record is used in the map where the subscribed topic name is
// the key so that values can quickly be updated when messages arrives.
std::unordered_map< Theron::AMQ::TopicName, MetricValueRecord > MetricValues;
// --------------------------------------------------------------------------
// JSON messages: Type by topic
// --------------------------------------------------------------------------
//
// The JSON message initialiser assumes that the content_type field of
// the message contains an unique label for the JSON message type to
// cover the situation where an actor may subscribe to multiple different
// messages all encoded as JSON messages. However, for this actor the type
// of the message will be decided by the topic on which the message is
// received. It is therefore necessary to set the message content type equal
// to the AMQ sender prior to decoding the AMQ message to the correct JSON
// object.
//
// The issue with the metric subscriptions is that the same type of message
// can come from any of the topics publishing metric values, and as such any
// topic name not being from the metric subscription command topic or the
// SLO Violation Event topic will be understood as a metric value update
// event The initialiser will check if the sender (topic) starts with the
// message identifier. This will allow the wildcard matching for metric
// values as well as an exact match for topic whose reply to address
// equals the message identifer.
class TypeByTopic
: public Theron::AMQ::JSONMessage
{
protected:
virtual bool
Initialize( const ProtocolPayload & ThePayload ) noexcept override
{
if( ThePayload->reply_to().starts_with( GetMessageIdentifier() ) )
{
ThePayload->content_type( GetMessageIdentifier() );
return JSONMessage::Initialize( ThePayload );
}
else return false;
}
public:
TypeByTopic( const std::string & TopicIdentifier )
: JSONMessage( TopicIdentifier )
{}
TypeByTopic( const TypeByTopic & Other )
: JSONMessage( Other.GetMessageIdentifier(), Other )
{}
virtual ~TypeByTopic() = default;
};
// --------------------------------------------------------------------------
// Subscribing to metric prediction values
// --------------------------------------------------------------------------
//
// Initially, the Optimiser Controller will pass a message containing all
// optimiser metric names and the AMQ topic on which their values will be
// published. Essentially, these messages arrives as a JSON message with
// one attribute per metric, and where the value is the topic string for
// the value publisher.
class MetricTopic
: public TypeByTopic
{
public:
MetricTopic( void )
: TypeByTopic( MetricSubscriptions.data() )
{}
};
// The handler for this message will check each attribute value of the
// received JSON struct, and those not already existing in the metric
// value map be added and a subscription made for the published
// prediction values.
void AddMetricSubscription( const MetricTopic & TheMetrics,
const Address OptimiserController );
// --------------------------------------------------------------------------
// Metric values
// --------------------------------------------------------------------------
//
class MetricValueUpdate
: public TypeByTopic
{
public:
MetricValueUpdate( void )
: TypeByTopic( MetricValueRootString.data() )
{}
};
// The handler function will check the sender address against the subscribed
// topics and if a match is found it will update the value of the metric.
// if no subscribed metric corresponds to the received message, the message
// will just be discarded.
void UpdateMetricValue( const MetricValueUpdate & TheMetricValue,
const Address TheMetricTopic );
}; // Class Metric Updater
} // Name space NebulOuS
#endif // NEBULOUS_METRIC_UPDATE

47
SolutionManager.hpp Normal file
View File

@ -0,0 +1,47 @@
/*==============================================================================
Solution Manager
This class handles the Execution Context mssage containing a time stamp and a
set of variable value assignments.It manages a time sorted queue and dispatches
the first application execution context to the solver when the solver is ready.
The solution returned for a given execution context will be published together
with the execution context and the maximal utility value found by the solver.
The solver actor class is given as a template argument to the solution manager,
and at least one solver actor is instantiated at start up. This to allow
multiple solvers to run in parallel should this be necessary to serve properly
the queue of waiting application execution contexts. If there are multiple
objects defined, they have to be optimised individualy, and for this purpose
it would also be useful to have multiple solvers running in parallel working
on the same problem, but for different objective functions. This will reduce
the time to find the Pareto front [1] for the multi-objective optimisation
problem.
The functionality of receiving and maintaining the work queue separately from
the solver is done to avoid blocking the reception of new execution contexts
while the solver searches for a solution in separate threads. This is done
for other entities to use the solver to find the optised configuration, i.e.
feasible value assignments to all propblem variables, maximising the givne
utiliy for a particular set of independent metric variables, i.e. the
application execution context. The idea is that other components may use
the solver in this way to produce training sets for machine learning methods
that aims to estimate the application's performance indicators or even the
change in utility as a function of the varying the metric values of the
application execution context.
References:
[1] https://en.wikipedia.org/wiki/Pareto_front
Author and Copyright: Geir Horn, University of Oslo
Contact: Geir.Horn@mn.uio.no
License: MPL2.0 (https://www.mozilla.org/en-US/MPL/2.0/)
==============================================================================*/
#ifndef NEBULOUS_SOLUTION_MANAGER
#define NEBULOUS_SOLUTION_MANAGER
namespace NebulOuS
{
} // namespace NebulOuS
#endif // NEBULOUS_SOLUTION_MANAGER

94
Solver.code-workspace Normal file
View File

@ -0,0 +1,94 @@
{
"folders": [
{
"path": "../../Theron++"
},
{
"path": "../../Theron++/Examples"
},
{
"path": "."
}
],
"settings": {
"files.associations": {
"cctype": "cpp",
"clocale": "cpp",
"cmath": "cpp",
"csignal": "cpp",
"cstdarg": "cpp",
"cstddef": "cpp",
"cstdio": "cpp",
"cstdlib": "cpp",
"cstring": "cpp",
"ctime": "cpp",
"cwchar": "cpp",
"cwctype": "cpp",
"array": "cpp",
"atomic": "cpp",
"strstream": "cpp",
"bit": "cpp",
"*.tcc": "cpp",
"cfenv": "cpp",
"charconv": "cpp",
"chrono": "cpp",
"codecvt": "cpp",
"compare": "cpp",
"complex": "cpp",
"concepts": "cpp",
"condition_variable": "cpp",
"cstdint": "cpp",
"deque": "cpp",
"list": "cpp",
"map": "cpp",
"set": "cpp",
"string": "cpp",
"unordered_map": "cpp",
"unordered_set": "cpp",
"vector": "cpp",
"exception": "cpp",
"algorithm": "cpp",
"functional": "cpp",
"iterator": "cpp",
"memory": "cpp",
"memory_resource": "cpp",
"numeric": "cpp",
"optional": "cpp",
"random": "cpp",
"ratio": "cpp",
"source_location": "cpp",
"string_view": "cpp",
"system_error": "cpp",
"tuple": "cpp",
"type_traits": "cpp",
"utility": "cpp",
"format": "cpp",
"fstream": "cpp",
"initializer_list": "cpp",
"iomanip": "cpp",
"iosfwd": "cpp",
"iostream": "cpp",
"istream": "cpp",
"limits": "cpp",
"mutex": "cpp",
"new": "cpp",
"numbers": "cpp",
"ostream": "cpp",
"ranges": "cpp",
"semaphore": "cpp",
"span": "cpp",
"sstream": "cpp",
"stdexcept": "cpp",
"stop_token": "cpp",
"streambuf": "cpp",
"thread": "cpp",
"cinttypes": "cpp",
"typeindex": "cpp",
"typeinfo": "cpp",
"variant": "cpp",
"any": "cpp",
"forward_list": "cpp",
"valarray": "cpp"
}
}
}

110
Solver.hpp Normal file
View File

@ -0,0 +1,110 @@
/*==============================================================================
Solver
The solver is a generic base class for all solvers defining the interface with
the Solution Manager actor. The solver reacts to an Application Execution
Context messagedefined in the class. The application execution context is
defined to be independent metric values that has no, or little, correlation
with the application configuration and that are involved in the utility
expression(s) or in the constraints of the optimisation problem.
Receiving this message triggers the search for an optimial solution to the
given named objective. Once the solution is found, the Solution message should
be returned to the actor making the request. The solution message will contain
the configuration being the feasible assignment to all variables of the
problem, all the objective values in this problem, and the identifier for the
application execution context.
The messages are essentially JSON objects defined using Niels Lohmann's
library [1] and in particular the AMQ extension for the Theron++ Actor
library [2] allowing JSON messages to be transmitted as Qpid Proton AMQ [3]
messages to renote requestors.
References:
[1] https://github.com/nlohmann/json
[2] https://github.com/GeirHo/TheronPlusPlus
[3] https://qpid.apache.org/proton/
Author and Copyright: Geir Horn, University of Oslo
Contact: Geir.Horn@mn.uio.no
License: MPL2.0 (https://www.mozilla.org/en-US/MPL/2.0/)
==============================================================================*/
#ifndef NEBULOUS_SOLVER
#define NEBULOUS_SOLVER
// Standard headers
#include <string_view> // Constant strings
// Other packages
#include <nlohmann/json.hpp> // JSON object definition
using JSON = nlohmann::json; // Short form name space
// Theron++ headers
#include "Actor.hpp" // Actor base class
#include "Utility/StandardFallbackHandler.hpp" // Exception unhanded messages
// AMQ communication headers
#include "Communication/AMQ/AMQjson.hpp" // For JSON metric messages
namespace NebulOuS
{
/*==============================================================================
Solver Actor
==============================================================================*/
//
class Solver
: virtual public Theron::Actor,
virtual public Theron::StandardFallbackHandler
{
private:
// --------------------------------------------------------------------------
// Application Execution Context
// --------------------------------------------------------------------------
//
// The message is defined as a JSON message representing an attribute-value
// object. The attributes expected are defined as constant strings so that
// the actual textual representation can be changed without changing the code
//
// "Identifier" It can be anything corresponding to the need of the sender
// and returned to the sender with the found solution
static constexpr std::string_view ContextIdentifier = "Identifier";
// "Timestamp" : This is the field giving the implicit order of the
// different application execution execution contexts waiting for being
// solved when there are more requests than there are solvers available to
// work on the different problems.
static constexpr std::string_view TimeStamp = "Timestamp";
// There is also a definition for the objective function label since a multi-
// objective optimisation problem can have multiple objective functions and
// the solution is found for only one of these functions at the time even
// though all objective function values will be returned with the solution,
// the solution will maximise only the objective function whose label is
// given in the application execution context request message.
static constexpr std::string_view
ObjectiveFunctionLabel = "ObjectiveFunction";
// Finally, there is another JSON object that defines all the metric name and
// value pairs that define the actual execution context. Note that there must
// be at least one metric-value pair for the request to be valid.
static constexpr std::string_view ExecutionContext = "ExecutionContext";
};
} // namespace NebulOuS
#endif // NEBULOUS_SOLVER