diff --git a/specs/mitaka/module-management.rst b/specs/mitaka/module-management.rst new file mode 100644 index 0000000..b9b27e8 --- /dev/null +++ b/specs/mitaka/module-management.rst @@ -0,0 +1,835 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + + Sections of this template were taken directly from the Nova spec + template at: + https://github.com/openstack/nova-specs/blob/master/specs/template.rst + +.. + This template should be in ReSTructured text. The filename in the git + repository should match the launchpad URL, for example a URL of + https://blueprints.launchpad.net/trove/+spec/awesome-thing should be named + awesome-thing.rst. + + Please do not delete any of the sections in this template. If you + have nothing to say for a whole section, just write: None + + Note: This comment may be removed if desired, however the license notice + above should remain. + + +================= +Module Management +================= + +.. If section numbers are desired, unindent this + .. sectnum:: + +.. If a TOC is desired, unindent this + .. contents:: + +Historically, Trove has supported open source databases. As more datastores +were added it was inevitable that this would eventually change to include +proprietary databases as well. Starting with the Liberty release this is the +case (support for Vertica and DB2 now being available) and with this comes the +issue of managing the licenses of said databases. In addition, operators may +find it useful to include other software on their images that may require +'activation' by end-users (for example New Relic's analytical suite). A method +of activating this software is also needed. + +The concept of applying a 'license' or 'activation' or 'configuration' of this +third-party software is what is referred to herein as 'module' management. + +Launchpad Blueprint: +https://blueprints.launchpad.net/trove/+spec/module-management + + +Problem Description +=================== + +Users of a particular cloud may be willing to purchase a license to use a +datastore from the cloud vendor on a pay-as-you-go based model, and this is the +model assumed at the moment (as both Vertica and DB2 redstack images include a +fully-functional and licensed database). It is also desirable, however, that +users be allowed to 'bring their own license.' In this scenario the user +provides a license file that the database requires, and as such a mechanism +needs to be in place to 'activate' and/or 'renew' the license through the use +of the user provided file. + +This same problem exists for any other proprietary software that an operator +may wish to include in their Trove images. These software packages also +typically require activation through the use of a license key or file (such as +New Relic [1]_) or configuration of some kind. + + +Proposed Change +=============== + +Trove's responsiblity towards module management will be restricted to the scope +of encrypting and storing the required data file (for example, a license file) +and providing a way to apply this data to a new or existing Trove instance or +cluster. A mechanism will be put in place to allow end users the ability to +manage adding, deleting, listing, viewing and updating these module data files. + +Methods to apply, remove, query and retrieve the actual 'module' data file on +the Trove instance will also be provided. + +A repeatable option (--module) will be added to the create and cluster-create +commands to allow adhoc module selection. In addition, modules can be set to +auto-apply, which will have the effect of the Guest Agent installing that +module on any instance created with the relevant datastore combination. + +Configuration +------------- + +The following configuration changes are anticipated. + +A way of specifying valid module 'types' will be needed for proper validation +on module create: + +.. code-block:: python + + cfg.StrOpt('module_types', default=None, + help='A list of module types supported.'), + +A key will be needed in order to be able to encrypt the module data file before +storing it in the database: + +.. code-block:: python + + cfg.StrOpt('module_aes_cbc_key', default='module_aes_cbc_key', + help='OpenSSL aes_cbc key for module encryption.'), + +Database +-------- + +A new table (modules) will be added to the Trove schema: + + ================= ============ =========== ============================== + Column Type Allow Nulls Description + ================= ============ =========== ============================== + id varchar(36) No ID of module (autogenerated) + type varchar(255) No Type of module. This will + correlate directly to the + required plugin (i.e. a + plugin must exist of this + 'type') + tenant_id varchar(36) No ID of tenant to apply + module to. 'all' means module + applies to all tenants + datastore varchar(36) No Name of datastore to apply + module to. 'all' means module + applies to all datastores + datastore_version varchar(36) No Name of datastore version to + apply module to. 'all' means + module applies to all + datastores + name varchar(255) No Name of module + description varchar(512) Yes Description of module + auto_apply tinyint(1) No Should this module be + automatically applied during + instance/cluster create. Will + default to 'no' if not + provided + visible tinyint(1) No Should this module be + visible to non-admin users. + Will default to 'yes' if not + provided + live_update tinyint(1) No Can this module be updated + while applied-to instances + still exist. If set to 'no' + all instances must have the + corresponding module removed + before it can be updated. + Defaults to 'no' + contents blob No Encrypted module contents + md5 varchar(32) No MD5 hash of module contents + created DateTime No Created date + updated DateTime No Updated date + deleted tinyint(1) Yes Deleted flag + deleted_at DateTime Yes Deleted date + ================= ============ =========== ============================== + +A unique index will be created from the (datastore, datastore_version, name) +fields, to allow easy determination of the correct module to apply to a +specific instance. + +An MD5 hash of the module contents will be stored in the module record as +well. This hash will be reported back when querying the module from a +running instance, as the original module record could have been modified with +new contents after it was initially applied. + +On installing the module contents on a given instance, a file will be created +in a know location using --.lic as a +pattern. + +Creating modules that apply to 'all' tenants or 'all' datastores and ones that +are auto-applied will require admin credentials. + +Setting a module to 'not' visible is also an admin-only option. This will +allow administrators to 'hide' modules from users if they so desire. Modules +that are marked visible=False will not be returned in commands such as list or +show unless requested by an admin user. Non-admin users won't be able to apply +a non-visible module, however they will still be auto-applied if so designated. + +A new table (instance_modules) will be added to the Trove schema to track which +modules have been applied to each instance: + + ================= ============ =========== ============================== + Column Type Allow Nulls Description + ================= ============ =========== ============================== + id varchar(36) No ID of association + (autogenerated) + instance_id varchar(36) No ID of instance + module_id varchar(36) No ID of module + md5 varchar(32) No MD5 hash of module contents + created DateTime No Created date + updated DateTime No Updated date + deleted tinyint(1) Yes Deleted flag + deleted_at DateTime Yes Deleted date + ================= ============ =========== ============================== + +Public API +---------- + +New ReST API calls will be added to the Trove infrastructure. These fall into +two categories - ones to manage the maintenance of the actual modules, and +ones to handle the instance interactions. + +In addition, the create and cluster-create calls will be enhanced. + +Module Maintenance +.................. + +To retrieve a list of all modules that can be applied, the following request +would be made: + +Request:: + + GET v1/modules + +Response:: + + { + 'modules' : [ + { + 'id': , + 'type': 'vertica_license', + 'tenant': , + 'datastore': 'vertica', + 'datastore_version': 'all', + 'name': '100GB', + 'description': 'Vertica license for 100GB', + 'auto_apply': False, + 'visible': True, # returned for admin only + 'live_update': False, + 'md5': , + 'created': , + 'updated': , + }, + { + 'id': , + 'type': 'new_relic_activation', + 'tenant': , + 'datastore': 'all', + 'datastore_version': 'all', + 'name': 'new_relic', + 'description': 'New Relic activation', + 'auto_apply': True, + 'visible': True, # returned for admin only + 'live_update': True, + 'md5': , + 'created': , + 'updated': , + }, + ] + } + +Response Codes:: + + 200 Success + +Note that an admin user will receive the modules for all tenants, whereas +regular users will see modules for their tenant only. + +To retrieve a list of valid modules that can be applied to a specific +datastore, the following request would be made: + +Request:: + + GET v1/datastores/{datastore_id}/modules + +Response:: + + { + 'modules' : [ + { + 'id': , + 'type': 'new_relic_activation', + 'tenant': , + 'datastore': 'all', + 'datastore_version': 'all', + 'name': 'new_relic', + 'description': 'New Relic activation', + 'auto_apply': True, + 'visible': True, # returned for admin only + 'live_update': True, + 'md5': , + 'updated': , + }, + ] + } + +Response Codes:: + + 200 Success + +To show the details of a particular module, the following request would be +made: + +Request:: + + GET v1/modules/ + +Response:: + + { + 'id': , + 'type': 'new_relic_activation', + 'tenant': , + 'datastore': 'all', + 'datastore_version': 'all', + 'name': 'new_relic', + 'description': 'New Relic activation', + 'auto_apply': True, + 'visible': True, # returned for admin only + 'live_update': True, + 'md5': , + 'created': , + 'updated': , + } + +Response Codes:: + + 200 Success + 404 Not Found + +To create a module, the following request would be made: + +Request:: + + POST /v1.0/modules + { + 'type': 'vertica_license', + 'tenant': , + 'datastore': 'vertica', + 'datastore_version': 'all', + 'name': '100GB', + 'description': 'Vertica license for 100GB', + 'auto_apply': False, + 'visible': False, # admin-only option + 'live_update': True, + 'contents': , + } + +Response:: + + { + "module": { + 'id': , + 'type': 'vertica_license', + 'tenant': , + 'datastore': 'vertica', + 'datastore_version': 'all', + 'name': '100GB', + 'description': 'Vertica license for 100GB', + 'auto_apply': False, + 'visible': False, # returned for admin only + 'live_update': True, + 'md5': , + 'created': , + 'updated': , + } + } + +Response Codes:: + + 200 Success + 400 Bad Request + +To update a module, the following request would be made: + +Request:: + + PATCH /v1.0/modules/{module_id} + { + 'type': 'new_type', + 'tenant': , + 'datastore': 'new_datastore', + 'datastore_version': 'new_datastore_version', + 'name': 'new_name', + 'description': 'new_description', + 'auto_apply': True, + 'visible': False, # admin-only option + 'live_update': True, + 'contents': , + } + +Response:: + + { + "module": { + 'id': , + 'type': 'new_type', + 'tenant': , + 'datastore': 'new_datastore', + 'datastore_version': 'new_datastore_version', + 'name': 'new_name', + 'description': 'new_description', + 'auto_apply': True, + 'visible': False, # returned for admin only + 'live_update': True, + 'md5': , + 'created': , + 'updated': , + } + } + +Response Codes:: + + 200 Success + 400 Bad Request + 404 Not Found + +To delete a module, the following request would be made: + +Request:: + + DELETE /v1.0/modules/{module_id} + { + } + +Response:: + + This operation has no response body + + +Response Codes:: + + 200 Success + 404 Not Found + +To query which instances have a particular module applied, the following +request would be made: + +Request:: + + GET v1/modules/{module_id}/instances + { + } + +Response:: + + { + 'instance': , + 'modules' : [ + { + 'name': '100GB', + 'id': , + 'md5': , + 'installed': , + }, + { + 'name': 'new_relic', + 'id': , + 'md5': , + 'installed': , + }, + ] + } + +Response Codes:: + + 200 Success + 404 Not Found + +Instance Interaction +.................... + +To apply modules to an instance, the following request would be made: + +Request:: + + POST v1/{tenant_id}/instances/{instance_id}/modules + { + 'modules' : [ + { + "id": , + }, + ] + } + +Response:: + + { + 'type': 'vertica_license', + 'datastore': 'vertica', + 'datastore_version': 'all', + 'name': '100GB', + 'md5': , + } + +Response Codes:: + + 202 Success + 400 Bad Request + 404 Not Found + +To query an instance about installed modules, the following request would be +made: + +Request:: + + GET v1/{tenant_id}/instances/{instance_id}/modules + { + } + +Response:: + + { + 'modules' : [ + { + 'type': 'vertica_license', + 'datastore': 'vertica', + 'datastore_version': 'all', + 'name': '100GB', + 'filename': 'vertica-all-100GB.lic', + 'md5': , + 'installed': , + 'status': 'OK', + 'error_message': None, + }, + { + 'type': 'new_relic_activation', + 'datastore': 'all', + 'datastore_version': 'all', + 'name': 'new_relic', + 'filename': 'all-all-new_relic.lic', + 'md5': , + 'installed': , + 'status': 'FAILED', + 'error_message': 'New Relic binaries not found', + }, + ] + } + +Response Codes:: + + 200 Success + 404 Not Found + +To retrieve a module from an instance, the following request would be made: + +Request:: + + GET v1/{tenant_id}/instances/{instance_id}/modules/{module_id} + { + } + +Response:: + + { + 'filename': 'vertica-all-100GB.lic', + 'contents': , + 'md5': , + } + +Response Codes:: + + 200 Success + 404 Not Found + +To delete a module from an instance, the following request would be made: + +Request:: + + DELETE v1/{tenant_id}/instances/{instance_id}/modules/{module_id} + { + } + +Response:: + + This operation has no response body + +Response Codes:: + + 202 Success + 404 Not Found + +Creation Enhancements +..................... + +The instance create API will be enhanced to include a module field, containing +a list of modules to apply. These will be sent down during the normal +'prepare' call and the appropriate plugin called once this instance has been +provisioned correctly. + +.. code-block:: python + + { + 'modules' : [ + { + "id": , + }, + ] + } + +In a similar manner, the cluster create API will also be enhanced to include +module information in the instances field, as is currently done with flavors, +AZs, etc. + + +Public API Security +------------------- + +Since the file will be transmitted clear text across the management +network, there is a chance that the module can be intercepted if the network +is compromised. + +It should be ensured that each plugin created does not 'execute' the contents +of the supplied module data file, as this would present the opportunity for a +security breach. This seems unlikely though (and will not be the case for the +proposed implementations) as most module data files will be passed to another +process for validation, and it is up to that process to ensure proper security +is maintained. Code reviews will be vital to make sure no plugin accidentally +executes this data. + +Python API +---------- + +New methods will be added to the Python API to facilitate the licensing. +A few existing methods will need to be extended as well. + +Module Maintenance +.................. + +.. code-block:: python + + def module_list(self, datastore=None): + """Get a list of all modules that can be applied. Return only + those that apply to the datastore if it is passed in. + """ + + def module_list_instances(self, module): + """Get a list of all instances that have a given module applied.""" + + def module_show(self, module): + """Show the details of the module.""" + + def module_create(self, module_type, name, description, contents, + datastore, datastore_version='all', auto_apply=False, + all_tenants=False, visible=True, live_update=False): + """Create a new module.""" + + def module_update(self, module, module_type=None, name=None, + description=None, contents=None, datastore=None, + datastore_version=None, auto_apply=None, + all_tenants=None, visible=None, live_update=None): + """Update an existing module.""" + + def module_delete(self, module): + """Delete a module.""" + +Instance Interaction +.................... + +.. code-block:: python + + def module_apply(self, instance, modules): + """Apply modules to an instance.""" + + def module_query(self, instance): + """Query an instance about installed modules.""" + + def module_retrieve(self, instance, module=None, filename=None): + """Retrieve the module data file from an instance and save it in + filename. If module is not supplied, retrieve all the modules. + If filename is not supplied, use the generated filename found + on the instance. + """ + + def module_remove(self, instance, module): + """Remove a module from an instance.""" + +Creation Enhancements +..................... + +For instance.create, the modules field will be added to the call: + +.. code-block:: python + + def create(self, name, flavor_id, volume=None, databases=None, users=None, + restorePoint=None, availability_zone=None, datastore=None, + datastore_version=None, nics=None, configuration=None, + replica_of=None, slave_of=None, replica_count=None, + modules=None): + """Create (boot) a new instance.""" + +For cluster.create, the modules field will be added to the +['cluster']['instances'] data structure that is already being passed in. + +CLI (python-troveclient) +------------------------ + +The following Trove CLI commands (upon completion) will be fully functional + +- module-list Displays all modules for the tenant. +- module-show Shows details for a particular module resource. +- module-create Creates a new module resource. +- module-update Updates module details for a particular module + resource. +- module-delete Delete a module resource. + +- module-apply Apply the given modules to a Trove instance. +- module-query Query the given Trove instance for any installed + modules. +- module-retrieve Retrieves the current modules from a Trove instance. +- module-remove Remove a module from a Trove instance. + +- create --module [--module] + Creates a new instance and applies the given modules. + +- cluster-create --instance=module=[,module=] + Creates a new cluster and applies the given modules to + each instance. + +Internal API +------------ + +Changes also need to be made to the internal API to include any module IDs as +a part of the message body that is sent to the task manager. + +The API server will need to make calls to the Guest Agent for the instance +interaction type commands. + +Guest Agent +----------- + +In the Guest Agent, the modules will be managed with a plugin style +architecture based on the stevedore.driver.DriverManager paradym. Each plugin +will need to implement 'apply', 'query' and 'remove' actions. The 'query' +action will need to report the status of the module 'apply' action. This +would report (at a minimum) 'OK' or 'FAILED' plus any other state that seems +reasonable for users of the relevant software. If possible, the +'error_message' field should be filled with useful information if an error +occurs. + +A simple plugin 'base class' that defines the contract will be provided. It +will also provide functionality such as placing the file contents into a +specified location and retrieving the file will be added. This can be used +as the basis for all other plugins. + +The Guest Agent code will use the module 'type' to determine if a plugin exists +for the given module. If no plugin can be found, then an error will be written +to the log and processing stopped. + +To provide a concrete, real-world plugin implementation, a Vertica license +module plugin will be created to allow licenses to be applied to a Vertica +datastore. A New Relic plugin will also be created to illustrate activation of +other third party software on a guest image. + +Alternatives +------------ + +None + + +Dashboard Impact (UX) +===================== + +A multi-dropdown will need to be added to the instance create dialog that +contains all modules for the selected datastore. These modules, along with +any auto-apply ones, will need to be sent along on the create call. The same +will be needed for the cluster create dialog. + +A module detail panel will need to be created. This panel will have fields +representing the attributes of a module (see module-create command). + +A 'modules' list panel will need to be created. This will have buttons for +'delete' and 'update' and will have a link to the detail page for each listed +module. This will be a high-level panel, similar to 'Instances.' + +The instance list panel will need to have a new action added: 'apply module.' +This will cause a pop-up where the available modules are displayed. The +selected module will then be passed in to the module-apply command. + +The instance detail panel will need to run 'module-query' and display the +results in a new section 'modules.' Alternately, a link could be placed here +that would open a module list panel with the results of the 'module-query' +call. Here, buttons for 'module-remove' and 'module-retrieve' would be +needed. + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + [peterstac] + +Milestones +---------- + +Mitaka + +Work Items +---------- + +The work will be undertaken with the following tasks: + + * Client (Python and CLI) changes + * Server (API) changes + * Guest Agent module plugin infrastructure + * Vertica/New Relic plugin implementation + + +Upgrade Implications +==================== + +Since this change is net-new, no upgrade issues are expected. + + +Dependencies +============ + +None. + + +Testing +======= + +Generic int-tests will be written, however these will not be run under MySQL +testing as it requires no module-based handling. + + +Documentation Impact +==================== + +This is a net-new feature, and as such will require documentation. + + +References +========== + +.. [1] nrsysmond-config --set license_key=. + + +Appendix +======== + +None