trove-specs/specs/mitaka/module-management.rst

Module Management

Historically, Trove has supported open source databases. As more datastores were added it was inevitable that this would eventually change to include proprietary databases as well. Starting with the Liberty release this is the case (support for Vertica and DB2 now being available) and with this comes the issue of managing the licenses of said databases. In addition, operators may find it useful to include other software on their images that may require 'activation' by end-users (for example New Relic's analytical suite). A method of activating this software is also needed.

The concept of applying a 'license' or 'activation' or 'configuration' of this third-party software is what is referred to herein as 'module' management.

Launchpad Blueprint: https://blueprints.launchpad.net/trove/+spec/module-management

Problem Description

Users of a particular cloud may be willing to purchase a license to use a datastore from the cloud vendor on a pay-as-you-go based model, and this is the model assumed at the moment (as both Vertica and DB2 redstack images include a fully-functional and licensed database). It is also desirable, however, that users be allowed to 'bring their own license.' In this scenario the user provides a license file that the database requires, and as such a mechanism needs to be in place to 'activate' and/or 'renew' the license through the use of the user provided file.

This same problem exists for any other proprietary software that an operator may wish to include in their Trove images. These software packages also typically require activation through the use of a license key or file (such as New Relic¹) or configuration of some kind.

Proposed Change

Trove's responsiblity towards module management will be restricted to the scope of encrypting and storing the required data file (for example, a license file) and providing a way to apply this data to a new or existing Trove instance or cluster. A mechanism will be put in place to allow end users the ability to manage adding, deleting, listing, viewing and updating these module data files.

Methods to apply, remove, query and retrieve the actual 'module' data file on the Trove instance will also be provided.

A repeatable option (--module) will be added to the create and cluster-create commands to allow adhoc module selection. In addition, modules can be set to auto-apply, which will have the effect of the Guest Agent installing that module on any instance created with the relevant datastore combination.

Configuration

The following configuration changes are anticipated.

A way of specifying valid module 'types' will be needed for proper validation on module create:

cfg.StrOpt('module_types', default=None,
           help='A list of module types supported.'),

A key will be needed in order to be able to encrypt the module data file before storing it in the database:

cfg.StrOpt('module_aes_cbc_key', default='module_aes_cbc_key',
           help='OpenSSL aes_cbc key for module encryption.'),

Database

A new table (modules) will be added to the Trove schema:

Column	Type	Allow Nulls	Description
id	varchar(36)	No	ID of module (autogenerated)
type	varchar(255)	No	Type of module. This will correlate directly to the required plugin (i.e. a plugin must exist of this 'type')
tenant_id	varchar(36)	No	ID of tenant to apply module to. 'all' means module applies to all tenants
datastore	varchar(36)	No	Name of datastore to apply module to. 'all' means module applies to all datastores
datastore_version	varchar(36)	No	Name of datastore version to apply module to. 'all' means module applies to all datastores
name	varchar(255)	No	Name of module
description	varchar(512)	Yes	Description of module
auto_apply	tinyint(1)	No	Should this module be automatically applied during instance/cluster create. Will default to 'no' if not provided
visible	tinyint(1)	No	Should this module be visible to non-admin users. Will default to 'yes' if not provided
live_update	tinyint(1)	No	Can this module be updated while applied-to instances still exist. If set to 'no' all instances must have the corresponding module removed before it can be updated. Defaults to 'no'
contents	blob	No	Encrypted module contents
md5	varchar(32)	No	MD5 hash of module contents
created	DateTime	No	Created date
updated	DateTime	No	Updated date
deleted	tinyint(1)	Yes	Deleted flag
deleted_at	DateTime	Yes	Deleted date

A unique index will be created from the (datastore, datastore_version, name) fields, to allow easy determination of the correct module to apply to a specific instance.

An MD5 hash of the module contents will be stored in the module record as well. This hash will be reported back when querying the module from a running instance, as the original module record could have been modified with new contents after it was initially applied.

On installing the module contents on a given instance, a file will be created in a know location using <datastore>-<datastore_version>-<name>.lic as a pattern.

Creating modules that apply to 'all' tenants or 'all' datastores and ones that are auto-applied will require admin credentials.

Setting a module to 'not' visible is also an admin-only option. This will allow administrators to 'hide' modules from users if they so desire. Modules that are marked visible=False will not be returned in commands such as list or show unless requested by an admin user. Non-admin users won't be able to apply a non-visible module, however they will still be auto-applied if so designated.

A new table (instance_modules) will be added to the Trove schema to track which modules have been applied to each instance:

Column	Type	Allow Nulls	Description
id	varchar(36)	No	ID of association (autogenerated)
instance_id	varchar(36)	No	ID of instance
module_id	varchar(36)	No	ID of module
md5	varchar(32)	No	MD5 hash of module contents
created	DateTime	No	Created date
updated	DateTime	No	Updated date
deleted	tinyint(1)	Yes	Deleted flag
deleted_at	DateTime	Yes	Deleted date

Public API

New ReST API calls will be added to the Trove infrastructure. These fall into two categories - ones to manage the maintenance of the actual modules, and ones to handle the instance interactions.

In addition, the create and cluster-create calls will be enhanced.

Module Maintenance

To retrieve a list of all modules that can be applied, the following request would be made:

Request:

GET v1/modules

Response:

{
    'modules' : [
        {
            'id': <id>,
            'type': 'vertica_license',
            'tenant': <id>,
            'datastore': 'vertica',
            'datastore_version': 'all',
            'name': '100GB',
            'description': 'Vertica license for 100GB',
            'auto_apply': False,
            'visible': True,  # returned for admin only
            'live_update': False,
            'md5': <md5>,
            'created': <date>,
            'updated': <date>,
        },
        {
            'id': <id>,
            'type': 'new_relic_activation',
            'tenant': <id>,
            'datastore': 'all',
            'datastore_version': 'all',
            'name': 'new_relic',
            'description': 'New Relic activation',
            'auto_apply': True,
            'visible': True,  # returned for admin only
            'live_update': True,
            'md5': <md5>,
            'created': <date>,
            'updated': <date>,
        },
    ]
}

Response Codes:

200  Success

Note that an admin user will receive the modules for all tenants, whereas regular users will see modules for their tenant only.

To retrieve a list of valid modules that can be applied to a specific datastore, the following request would be made:

Request:

GET v1/datastores/{datastore_id}/modules

Response:

{
    'modules' : [
        {
            'id': <id>,
            'type': 'new_relic_activation',
            'tenant': <id>,
            'datastore': 'all',
            'datastore_version': 'all',
            'name': 'new_relic',
            'description': 'New Relic activation',
            'auto_apply': True,
            'visible': True,  # returned for admin only
            'live_update': True,
            'md5': <md5>,
            'updated': <date>,
        },
    ]
}

Response Codes:

200  Success

To show the details of a particular module, the following request would be made:

Request:

GET v1/modules/<id>

Response:

{
    'id': <id>,
    'type': 'new_relic_activation',
    'tenant': <id>,
    'datastore': 'all',
    'datastore_version': 'all',
    'name': 'new_relic',
    'description': 'New Relic activation',
    'auto_apply': True,
    'visible': True,  # returned for admin only
    'live_update': True,
    'md5': <md5>,
    'created': <date>,
    'updated': <date>,
}

Response Codes:

200  Success
404  Not Found

To create a module, the following request would be made:

Request:

POST /v1.0/modules
{
    'type': 'vertica_license',
    'tenant': <id>,
    'datastore': 'vertica',
    'datastore_version': 'all',
    'name': '100GB',
    'description': 'Vertica license for 100GB',
    'auto_apply': False,
    'visible': False,  # admin-only option
    'live_update': True,
    'contents': <module_contents>,
}

Response:

{
    "module": {
        'id': <id>,
        'type': 'vertica_license',
        'tenant': <id>,
        'datastore': 'vertica',
        'datastore_version': 'all',
        'name': '100GB',
        'description': 'Vertica license for 100GB',
        'auto_apply': False,
        'visible': False,  # returned for admin only
        'live_update': True,
        'md5': <md5>,
        'created': <date>,
        'updated': <date>,
    }
}

Response Codes:

200  Success
400  Bad Request

To update a module, the following request would be made:

Request:

PATCH /v1.0/modules/{module_id}
{
    'type': 'new_type',
    'tenant': <id>,
    'datastore': 'new_datastore',
    'datastore_version': 'new_datastore_version',
    'name': 'new_name',
    'description': 'new_description',
    'auto_apply': True,
    'visible': False,  # admin-only option
    'live_update': True,
    'contents': <module_contents>,
}

Response:

{
    "module": {
        'id': <id>,
        'type': 'new_type',
        'tenant': <id>,
        'datastore': 'new_datastore',
        'datastore_version': 'new_datastore_version',
        'name': 'new_name',
        'description': 'new_description',
        'auto_apply': True,
        'visible': False,  # returned for admin only
        'live_update': True,
        'md5': <new_md5>,
        'created': <date>,
        'updated': <date>,
    }
}

Response Codes:

200  Success
400  Bad Request
404  Not Found

To delete a module, the following request would be made:

Request:

DELETE /v1.0/modules/{module_id}
{
}

Response:

This operation has no response body

Response Codes:

200  Success
404  Not Found

To query which instances have a particular module applied, the following request would be made:

Request:

GET v1/modules/{module_id}/instances
{
}

Response:

{
    'instance': <id>,
    'modules' : [
        {
            'name': '100GB',
            'id': <id>,
            'md5': <md5>,
            'installed': <date>,
        },
        {
            'name': 'new_relic',
            'id': <id>,
            'md5': <md5>,
            'installed': <date>,
        },
    ]
}

Response Codes:

200  Success
404  Not Found

Instance Interaction

To apply modules to an instance, the following request would be made:

Request:

POST v1/{tenant_id}/instances/{instance_id}/modules
{
    'modules' : [
        {
            "id": <id>,
        },
    ]
}

Response:

{
    'type': 'vertica_license',
    'datastore': 'vertica',
    'datastore_version': 'all',
    'name': '100GB',
    'md5': <md5>,
}

Response Codes:

202  Success
400  Bad Request
404  Not Found

To query an instance about installed modules, the following request would be made:

Request:

GET v1/{tenant_id}/instances/{instance_id}/modules
{
}

Response:

{
    'modules' : [
        {
            'type': 'vertica_license',
            'datastore': 'vertica',
            'datastore_version': 'all',
            'name': '100GB',
            'filename': 'vertica-all-100GB.lic',
            'md5': <md5>,
            'installed': <date>,
            'status': 'OK',
            'error_message': None,
        },
        {
            'type': 'new_relic_activation',
            'datastore': 'all',
            'datastore_version': 'all',
            'name': 'new_relic',
            'filename': 'all-all-new_relic.lic',
            'md5': <md5>,
            'installed': <date>,
            'status': 'FAILED',
            'error_message': 'New Relic binaries not found',
        },
    ]
}

Response Codes:

200  Success
404  Not Found

To retrieve a module from an instance, the following request would be made:

Request:

GET v1/{tenant_id}/instances/{instance_id}/modules/{module_id}
{
}

Response:

{
    'filename': 'vertica-all-100GB.lic',
    'contents': <module_contents>,
    'md5': <md5>,
}

Response Codes:

200  Success
404  Not Found

To delete a module from an instance, the following request would be made:

Request:

DELETE v1/{tenant_id}/instances/{instance_id}/modules/{module_id}
{
}

Response:

This operation has no response body

Response Codes:

202  Success
404  Not Found

Creation Enhancements

The instance create API will be enhanced to include a module field, containing a list of modules to apply. These will be sent down during the normal 'prepare' call and the appropriate plugin called once this instance has been provisioned correctly.

{
    'modules' : [
        {
            "id": <id>,
        },
    ]
}

In a similar manner, the cluster create API will also be enhanced to include module information in the instances field, as is currently done with flavors, AZs, etc.

Public API Security

Since the file will be transmitted clear text across the management network, there is a chance that the module can be intercepted if the network is compromised.

It should be ensured that each plugin created does not 'execute' the contents of the supplied module data file, as this would present the opportunity for a security breach. This seems unlikely though (and will not be the case for the proposed implementations) as most module data files will be passed to another process for validation, and it is up to that process to ensure proper security is maintained. Code reviews will be vital to make sure no plugin accidentally executes this data.

Python API

New methods will be added to the Python API to facilitate the licensing. A few existing methods will need to be extended as well.

Module Maintenance

def module_list(self, datastore=None):
    """Get a list of all modules that can be applied. Return only
    those that apply to the datastore if it is passed in.
    """

def module_list_instances(self, module):
    """Get a list of all instances that have a given module applied."""

def module_show(self, module):
    """Show the details of the module."""

def module_create(self, module_type, name, description, contents,
                  datastore, datastore_version='all', auto_apply=False,
                  all_tenants=False, visible=True, live_update=False):
    """Create a new module."""

def module_update(self, module, module_type=None, name=None,
                  description=None, contents=None, datastore=None,
                  datastore_version=None, auto_apply=None,
                  all_tenants=None, visible=None, live_update=None):
    """Update an existing module."""

def module_delete(self, module):
    """Delete a module."""

Instance Interaction

def module_apply(self, instance, modules):
    """Apply modules to an instance."""

def module_query(self, instance):
    """Query an instance about installed modules."""

def module_retrieve(self, instance, module=None, filename=None):
    """Retrieve the module data file from an instance and save it in
    filename.  If module is not supplied, retrieve all the modules.
    If filename is not supplied, use the generated filename found
    on the instance.
    """

def module_remove(self, instance, module):
    """Remove a module from an instance."""

Creation Enhancements

For instance.create, the modules field will be added to the call:

def create(self, name, flavor_id, volume=None, databases=None, users=None,
           restorePoint=None, availability_zone=None, datastore=None,
           datastore_version=None, nics=None, configuration=None,
           replica_of=None, slave_of=None, replica_count=None,
           modules=None):
    """Create (boot) a new instance."""

For cluster.create, the modules field will be added to the ['cluster']['instances'] data structure that is already being passed in.

CLI (python-troveclient)

The following Trove CLI commands (upon completion) will be fully functional

• module-list Displays all modules for the tenant.
• module-show Shows details for a particular module resource.
• module-create Creates a new module resource.

• module-update Updates module details for a particular module

  resource.

• module-delete Delete a module resource.
• module-apply Apply the given modules to a Trove instance.

• module-query Query the given Trove instance for any installed

  modules.

• module-retrieve Retrieves the current modules from a Trove instance.
• module-remove Remove a module from a Trove instance.

• create --module [--module]

  Creates a new instance and applies the given modules.

• cluster-create --instance=module=<id>[,module=<id>]

  Creates a new cluster and applies the given modules to each instance.

Internal API

Changes also need to be made to the internal API to include any module IDs as a part of the message body that is sent to the task manager.

The API server will need to make calls to the Guest Agent for the instance interaction type commands.

Guest Agent

In the Guest Agent, the modules will be managed with a plugin style architecture based on the stevedore.driver.DriverManager paradym. Each plugin will need to implement 'apply', 'query' and 'remove' actions. The 'query' action will need to report the status of the module 'apply' action. This would report (at a minimum) 'OK' or 'FAILED' plus any other state that seems reasonable for users of the relevant software. If possible, the 'error_message' field should be filled with useful information if an error occurs.

A simple plugin 'base class' that defines the contract will be provided. It will also provide functionality such as placing the file contents into a specified location and retrieving the file will be added. This can be used as the basis for all other plugins.

The Guest Agent code will use the module 'type' to determine if a plugin exists for the given module. If no plugin can be found, then an error will be written to the log and processing stopped.

To provide a concrete, real-world plugin implementation, a Vertica license module plugin will be created to allow licenses to be applied to a Vertica datastore. A New Relic plugin will also be created to illustrate activation of other third party software on a guest image.

Alternatives

None

Dashboard Impact (UX)

A multi-dropdown will need to be added to the instance create dialog that contains all modules for the selected datastore. These modules, along with any auto-apply ones, will need to be sent along on the create call. The same will be needed for the cluster create dialog.

A module detail panel will need to be created. This panel will have fields representing the attributes of a module (see module-create command).

A 'modules' list panel will need to be created. This will have buttons for 'delete' and 'update' and will have a link to the detail page for each listed module. This will be a high-level panel, similar to 'Instances.'

The instance list panel will need to have a new action added: 'apply module.' This will cause a pop-up where the available modules are displayed. The selected module will then be passed in to the module-apply command.

The instance detail panel will need to run 'module-query' and display the results in a new section 'modules.' Alternately, a link could be placed here that would open a module list panel with the results of the 'module-query' call. Here, buttons for 'module-remove' and 'module-retrieve' would be needed.

Implementation

Assignee(s)

Primary assignee:

[peterstac]

Milestones

Mitaka

Work Items

The work will be undertaken with the following tasks:

• Client (Python and CLI) changes
• Server (API) changes
• Guest Agent module plugin infrastructure
• Vertica/New Relic plugin implementation

Upgrade Implications

Since this change is net-new, no upgrade issues are expected.

Dependencies

None.

Testing

Generic int-tests will be written, however these will not be run under MySQL testing as it requires no module-based handling.

Documentation Impact

This is a net-new feature, and as such will require documentation.

References

Appendix

None

---

1. nrsysmond-config --set license_key=<new_relic_key>.↩︎