From 5a06e19ce0fb8426e200b9cf0c3f6e5008daeb7a Mon Sep 17 00:00:00 2001 From: "James E. Blair" Date: Fri, 10 Oct 2014 15:56:09 -0700 Subject: [PATCH] Add nodepool-builder spec Change-Id: I2979351b8c3941eca4e42925e27bef68e73abf91 --- specs/nodepool-workers.rst | 163 +++++++++++++++++++++++++++++++++++++ 1 file changed, 163 insertions(+) create mode 100644 specs/nodepool-workers.rst diff --git a/specs/nodepool-workers.rst b/specs/nodepool-workers.rst new file mode 100644 index 0000000..91de5fa --- /dev/null +++ b/specs/nodepool-workers.rst @@ -0,0 +1,163 @@ +:: + + Copyright (c) 2014 Hewlett-Packard Development Company, L.P. + + This work is licensed under a Creative Commons Attribution 3.0 + Unported License. + http://creativecommons.org/licenses/by/3.0/legalcode + +======================================= +Nodepool image build and upload workers +======================================= + +Story: https://storyboard.openstack.org/#!/story/317 + +Split the image build and upload features of nodepool into separate +workers for scalability and flexibility. + +Problem Description +=================== + +In moving to using diskimage-builder (DIB) with nodepool we have +encountered some problems with building images on different platforms. +DIB should be able to cross-build for most GNU/Linux systems, but we +have seen some failures related to this. + +Further, updates to the DIB mechanism in nodepool require a full +nodepool server restart, which can be somewhat disruptive. + +Performing the image building on the same server as nodepool is not +currently a resource issue, however, parallel builds are constrained +by resources and distributing builds to multiple machines could +improve scalability. + +Finally, there are some platforms we will never be able to cross-build +images on, such as Microsoft Windows. This is not an immediate +concern for the OpenStack infrastructure program, but would be a +useful feature for downstream consumers of nodepool. + +Proposed Change +=============== + +Create a new worker (independent process which may run on either the +main nodepool host or one or more new servers) which perform image +build related tasks called "nodepool-builder". + +The nodepool-builder worker would read a configuration file to +determine which kinds of images it can produce (determined by the +operator according to criteria such as platform or operating system, +access to needed external resources, and scaleout plan). It will then +register a function with gearman for each kind of image it can build, +with the job name of the form "image-build:", e.g., +"image-build:centos7". A unique identifier for the image will be +provided as a parameter to the job. It will then use DIB to build the +image and store it in a local repository of image files. + +Additional image builders may be created to operate in another manner, +such as using the existing nodepool on-instance build methodology, or +other image building methods yet to be explored. This way the image +building mechanism may be substituted more easily in the future. + +The nodepool-builder worker will also register functions with gearman +with the form "image-upload:" where the image id is the image +identifier provided in the image-build function. The worker will +register such functions each time it completes an image build (it will +also register such functions on startup based on the images it has +available locally). This is so that the build and upload steps may be +separated into separate functions, yet can be routed to the correct +worker (which has the image file on a local filesystem). This +function will accept as an argument the name of a provider to which +the image should be uploaded. It will perform the necessary format +conversions on the image file prior to upload. It will return the +cloud image id with the WORK_COMPLETE packet. + +The builder will also register functions with the form +"image-delete:" (registered along with the image-upload +function as above) which will clean up local data related to the +image. When an image is deleted, the corresponding image-delete and +image-upload functions will be de-registered from gearman. + +The implementation of the worker will be performed in such a way that +it will be simple to create a subclass with a different method of +performing the image creation. For example, there may be a base class +that performs all of the actions described except for the actual image +creation, and the actual worker may be a subclass of that which uses +DIB as the build implementation. + +Alternatives +------------ + +The build and upload processes could be separate workers. However, +the two processes are somewhat tightly coupled since they need to run +on the same host and generally in immediate succession. Therefore the +current method where they share a process but use modular code to +enable different builders to share as much as possible is chosen. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: unknown + +Work Items +---------- + +* Create base class for nodepool-builder that implements above +* Create nodepool-builder that implements dib-based image builds +* Alter nodepool to use gearman functions when using dib images only + (continuing to use on-instance builds in the main server for other + image types during the migration phase) +* Optional: create nodepool-builder that implements nodepool on-image + builds +* Remove non-dib image builds from the main nodepool server +* Rename bits of the nodepool server and config that refer to dib + specifics (since that is now a build worker implementation detail) + +Repositories +------------ + +This affects the existing nodepool, system-config, and potentially +project-config repos. + + +Servers +------- + +Affects the existing nodepool server and may necessitate the creation +of new servers for specific nodepool image builders (optional). + +DNS Entries +----------- + +If any image build servers are created, they will need DNS entries. + +Documentation +------------- + +The infra/system-config nodepool documentation should be updated to +describe the new system. + +Security +-------- + +The gearman protocol is cleartext and unauthenticated. IP based +access control is currently used, and certificate support along with +authentication is planned and work is in progress. No sensitive +information will be sent over the wire (workers will read cloud +provider credentials from a local file). + +Testing +------- + +This should be testable privately and locally for most image types. +Any image types currently supplied by nodepool using dib will need to +switch over to the new system immediately. Others may migrate on an +image-by-image basis. + + +Dependencies +============ + +N/A.