diff --git a/specs/ocata/trove-integration-redstack.rst b/specs/ocata/trove-integration-redstack.rst new file mode 100644 index 0000000..cff5238 --- /dev/null +++ b/specs/ocata/trove-integration-redstack.rst @@ -0,0 +1,407 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + + Sections of this template were taken directly from the Nova spec + template at: + https://github.com/openstack/nova-specs/blob/master/specs/juno-template.rst + +.. + This template should be in ReSTructured text. The filename in the git + repository should match the launchpad URL, for example a URL of + https://blueprints.launchpad.net/trove/+spec/awesome-thing should be named + awesome-thing.rst. + + Please do not delete any of the sections in this template. If you + have nothing to say for a whole section, just write: None + + Note: This comment may be removed if desired, however the license notice + above should remain. + + +======================================== +Eliminate trove-integration and redstack +======================================== + +.. sectnum:: +.. contents:: + +In the very beginning, when Trove was integrated into OpenStack, one +of the subjects of some contention was the redstack tool, and the fact +that Trove had a somewhat non-standard structure in the form of a +trove-integration repository. + +It was, I understand, one of the stipulations of the Technical +Committee at the time that this be normalized. + +For a number of reasons, (which we will describe in the Problem +Description section) this structure is proving to be quite a nusiance +and it would be beneficial to the project to do this now. + +The goal of this project is to eliminate the trove-integration project +and the redstack tool through a normal process of deprecation. That is +to say that existing releases, through Newton, will continue to be +driven and may (I stress, may) use the tool, but new versions from +Ocata will not. + +Launchpad Blueprint: +https://blueprints.launchpad.net/trove/+spec/eliminate-trove-integration-and-redstack + + +Problem Description +=================== + +The trove-integration project was created as a place to store scripts +to create, manipulate and test DBaaS vm's, and to house the scripts +used to manage the development environment. The redstack tool was the +swiss army knife, it could provision your development environment, +make guest images, load them, and later run tests as well. + +On the whole, a very useful thing, there is no doubt about that. + +It did introduce several serious problems. + +1. trove-integration is not versioned + +It has only a master branch. Therefore all code that is version +sensitive has to accomplish that through crafty branching constructs. + +2. trove-integration is a repository outside the trove repository + +This brings with it dependencies on the trove repository, such as when +a new datastore needs to be added, or a new version of a datastore is +to be added. Trove 'elements' go into trove-integration but get used +by trove. + +3. The CI is very painful + +Since trove-integration isn't branched, we have to test each +trove-integration branch against not just master, but also stable +branches. + +4. The versioning just got more complex + +We now have to deal with multiple operating systems (ubuntu and +fedora), multiple versions (liberty, mitaka, newton, ocata, ...), +different versions of operating systems (trusty, and now xenial). The +code is now convoluted enough that attempting to add support for +xenial (as I had to do for SQL Server) became a nightmare. + +5. There is no easy way to build images + +This is the #1 problem faced by users of Trove; how do I build +images. A solution was proposed +(https://review.openstack.org/#/c/312806/) that would address this +through the creation of a trove-image-builder repository. This +solution has some merits and demerits, we choose here to propose a +different solution (see second bullet in proposed end state) that will +address that problem as well. + +6. redstack is 'non-standard' + +The current mode/phase approach provided by devstack is standardized +across all projects, and redstack is one of the few (if not the only) +non-standard approach. We already support the devstack plugin and +redstack (in the provisioning a stack use-case) is nothing more than a +thin veneer on devstack. + +Proposed Change +=============== + +The proposed change is in multiple parts, and several of the sections +don't match the typical headings one expects in a blueprint that is +aimed at a new feature or function in Trove. Therefore the sections +may be marked as Not Applicable. Additional sections have been added +where required. + +One thing that is important to highlight is that while this change is +highly disruptive, and moves a whole lot of code around, we are +largely talking about moving around a couple of hundred files +(elements) from one place to another, and leveraging a devstack plugin +that we already use extensively. There will be an impact on the tests +that are in trove-integration but we already do most of our testing +out of the trove repository. + +Changes that have been proposed in the trove-integration repository; +which are pending review and merge will be negatively impacted. But +that will be the case no matter when we choose to make a change like +this. Most of the elements are highly parameterized and the changes +actually required to make this all work are minimal. + +I feel that the best way to describe this proposed change is to begin +with a description of the proposed 'end-state'. + +Proposed End State +------------------ + +* The elements from trove-integration will get folded into the trove + repository + +* The elements will now be versioned along with the server, they are + in the same repository. Tagging the Ocata release will also cause + the release of a set of elements for the release and similarly the + tagging of the Pike release will cause the release of a set of + elements for that release. + + A set of tools will be provided that will make this easier, more + description of that below. + +* A set of sample configuration files will be provided that will cause + devstack to install and configure Trove. A user can copy these into + their own localrc, or use the commands to set the right environment + variables before invoking devstack/stack.sh + +* The trove devstack plugin will have additional code in the + stack/extra step that will build and load a set of guest images of + the users choosing. This image or images will be built using the + elements from the trove repository. + + This will cause the standard demo user to be used (not alt_demo). + + In running devstack, one must configure a configuration file for + devstack and one specifies the datastore whose image is to be built + and loaded in that configuration file. I was initially contemplating + allowing for the specification of a comma-seperated list but a + single datastore may be a good place to start. This datastore would + also be set as the default. + + One devstack environment can operate with multiple datastores. Three + commands (trove-image-builder.bash, load-trove-image.bash and + activate-trove-image.bash) will be provided. The first will build + the image for you. The second will load it into glance and register + it with Trove, the third will set a specified datastore/version to + the the default. + +* The trove devstack plugin will have additional code in the + stack/post-config phase that will configure the Trove test + environment, add the right flavors, and do the other things that are + required for Trove testing. + +* Scenario tests that are already in the trove repository will be run + without the use of the redstack tool. + +* Releases Newton and before may continue to use trove-integration and + redstack, releases Ocata and forward shall not use trove-integration + and redstack. For example, new elements for Ocata and the future + will not go into trove-integration. This means that during this + transition (which may last for two releases) some changes may have + to go to both trove-integration, and trove. + +* The Ocata CI will have a new set of jobs that will leverage this new + state, no longer using redstack for its provisioning, and the cross + version testing that is going on in trove-integration will no longer + need to be supported. + +Configuration +------------- + +* Some sample configuration files will be provided and these will help + a user configure their localrc before running stack.sh. This will + cause devstack to completely provision Trove, and additionally build + and load guest images as instructed by the user. + +How we get there +---------------- + +Getting to the desired end state will be a multi-step exercise. + +Step 1 + +Take everything in the trove-integration repository at some specific +point in time and drop it into the trove repository. For convenience, +I'm going to drop it into a folder called integration. + +It would be nice to do this and preserve history but that is going to +be quite a pain to accomplish. An email has been sent to the infra +mailing list requesting help with this but in the intermin, the commit +https://review.openstack.org/#/c/384195/ does what we need (sans +histories). + +Step 2 + +Make changes to the stuff now in the trove repository and make it such +that the redstack command (which will be renamed to trovestack) can be +executed just as it used to be in the trove-integration branch. This +also includes some small amendments to the documentation. + +Commit https://review.openstack.org/#/c/384746/ does this. + +Step 3 + +Refactor the CI jobs to use trovestack in the trove tree instead of +redstack from the trove-integration tree. This means all the CI jobs +(or maybe most) will become legacy jobs and new jobs will be created +for Ocata and higher. + +Step 4 + +There is a huge amount of cruft in trovestack that handled the fact +that redstack in trove-integration was not versioned and released with +each OpenStack version. We should do this cleanup because sooner +rather than later as we will soon have to handle the Ubuntu 16.04 +transition and having that dimension in addition to the OpenStack +version dimension will be very difficult to manage. + +Step 5 + +Take the image build aspect of trovestack (formerly redstack) and make +it an independent tool. https://review.openstack.org/#/c/374952/ began +this process and took the elements out of the place where they were +copied in from trove-integration and starts to make it a standalone +tool. We will update trovestack to use this tool and avoid the +duplication of the image building logic. + +Take the image loading logic (including the setting of the default +datastore) and make that a standalone tool and modify trovestack to +use that tool. + +Step 6 + +Refactor the CI to automatically build images for different datastores +and verify that the image build process is proper. This will be a new job. + +Step 7 + +Refactor the integration tests into the tests directory. + + +Database +-------- + +Not applicable + +Public API +---------- + +Not applicable + +Public API Security +------------------- + +Not applicable + +Python API +---------- + +Not applicable + +CLI (python-troveclient) +------------------------ + +Not applicable + +Internal API +------------ + +Not applicable + +Guest Agent +----------- + +The guest agent is already part of the trove repository and will +remain there. It will now be in the same versioned repository as the +server code which has dependencies on it. + +Alternatives +------------ + +There are several options that have been considered, some at great +length. + +1. Do nothing + +Well, we've done that for three years and it hasn't been fun. I think +we can safely say that there are strong reasons to disqualify this +option. + +2. Just get the image build out of redstack, leave the rest + +This was part (at least) of the approach behind setting up the +trove-image-builder repository. It would be an improvement over the +present state but still would have cross repository +dependencies. Considering that there is no clear justification for +releasing trove-image-builder at a different cadence than trove, I see +no justification in having an independent repository. + +We know that there is a tight coupling, and there likely will be one +for a long time longer, between the server side of Trove and the guest +agent/images. Having two repositories just makes handling that a pain. + +3. Use the trove-image-builder approach that was proposed earlier. + +We could use the approach that was proposed earlier +(https://review.openstack.org/#/c/312806/) but this solution has the +same kind(s) of challenges that one faces with trove-integration. Two +repositories, releasing them in sync, etc., + +Having elements in the trove repository is a better alternative. + +Dashboard Impact (UX) +===================== + +No dashboard impact is expected. + +The dashboard will be enabled (if the user enables the plugin). Code +already exists for that in the trove plugin. + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + amrith + +Dashboard assignee: + + +Milestones +---------- + +Target Milestone for completion: + Ocata-1 + +Work Items +---------- + +* Move elements from trove-integration to trove +* Write wrapper (trove-image-builder) script for simplicity of use +* Create sample configuration file settings +* Modify CI jobs + +Upgrade Implications +==================== + +This change has no upgrade implications as we are talking largely +about refactoring a development infrastructure. + +Dependencies +============ + +None + +Testing +======= + +We will test the creation of each of the images, and putting them +through their paces in the CI with our existing scenario tests. + +Documentation Impact +==================== + +Internal (developer) documentation will need to be updated. + + +References +========== + +None + +Appendix +======== + +None