diff --git a/specs/cinder-integration.rst b/specs/cinder-integration.rst index c2534040d..10fb33721 100644 --- a/specs/cinder-integration.rst +++ b/specs/cinder-integration.rst @@ -16,72 +16,169 @@ such as IP addresses, security groups, ports, volumes, etc. Zun containers should be able to use volumes which has multiple storage vendors or plugins support. -As zun is project belongs to OpenStack ecosystem and zun has integration +As Zun is project belongs to OpenStack ecosystem and Zun has integration with Cinder which is block storage service. -Fuxi is new OpenStack project which aims to integrate Cinder to docker -volumes. With Fuxi, Cinder can be used as the unified persistence storage -provider for virtual machine, baremetal and Docker container. - -The implementation of Cinder is enabled using fuxi driver from zun. We need -to implement Cinder driver in Zun which manages volumes, let Fuxi control the -mount/unmount volume from Docker container. - Problem description =================== -To store some large amount data in container is not possible. In case of -of containers, all the storage resides in host storage which is not good -solution. If host goes down, storage wont be accessible. -We need to somehow use cloud storage, which is reliable and we can attach -it to container so that people use containers for workloads also. +Data persistence in container is infeasible. The root file system of a Docker +container is a Union File System. Data resides in Union File System is +ephemeral because the data will lose whenever the container is deleted or +the host goes down. In addition, the performance of persisting a large amount +of data into Union File System is suboptimal. + +To address the use cases that require persisting a large amount of data, +a common solution is to leverage the Docker data volume. A data volume is a +specially-designated directory within one or more containers that bypasses +the Union File System [1]. It is designed for storing data and share the data +across containers. Data volume can be provisioned by directly mounting a host +directory, or by a volume plugin that interfaces with a cloud storage backend. Proposed change =============== -There are two approaches docker provides to add volume to Container. +This spec proposes the following changes. -1. Using Docker run +1. Enhance existing API to support bind-mounting Cinder volumes to a container + as data volumes. -:: +2. Define a pluggable interface that can be implemented by different volume + drivers. A volume driver is a module that is responsible for managing Cinder + volumes for containers. Initially, we are going to provide two drivers: + a Cinder driver and a Fuxi driver. - docker run -d --volume-driver= Fuxi -v my-named-volume --name web_app +Cinder driver +============= +This driver is responsible to manage the bind-mounting of Cinder volumes. +If users want to create a container with a volume, they are required to +pre-create the volume in Cinder. Zun will perform the necessary steps to make +the Cinder volume available to the container, which typically includes +retrieving volume information from Cinder, connecting to the volume by using +os-brick library, mounting the connected volume into a directory in the +host's filesystem, and calling Docker API to bind-mount the specific directory +into the container. -2. Create volume first & then add it to Container +The typical workflow to create a container with a Cinder volume will be as +following: -:: +1. A user calls Zun APIs to create a container with a volume:: - docker volume create --driver fuxi - --name test_vol - -o size=1 - -o fstype=ext4 - -o multiattach=true + $ zun run --volume-driver=cinder -v my-cinder-volume:/data cirros -:: +2. After receiving this request, Zun will make an API call to Cinder to + reserve the volume. This step will update the status of the volume to + "attaching" in Cinder to ensure it cannot be used by other users:: - docker run -d -v my-named-volume --name web_app + cinderclient.volumes.reserve(volume) -I think, we can support both +3. Zun makes an API call to Cinder to retrieve the connection information:: -1. To Implement first approach, we need following changes + conn_info = cinderclient.volumes.initialize_connection(volume, ...) + +4. Zun uses os-brick library with the returned connection to do the connect. + A successful connection should return the device information that will be + used for mounting:: + + device_info = brick_connector.connect_volume(conn_info) + +5. Zun makes an API call to Cinder to finalize the volume connection. + This will update the status of the volume from "attaching" to "attached" + in Cinder:: + + cinderclient.volumes.attach(volume) + +6. Zun mounts the storage device (provided by step 4) into a directory in the + host's filesystem, and calls Docker API to create a container and use + that directory as a data volume:: + + $ docker run -d -v /opt/stack/data/zun/mnt/:/data cirros + +The typical workflow to delete a container with a Cinder volume will be as +following: + +1. A user calls Zun APIs to delete a container:: + + $ zun delete my-container + +2. After receiving this request, Zun will make an API call to Cinder to + begin detaching the volume. This will update the status of the volume to + "detaching" state:: + + cinderclient.volumes.begin_detaching(volume) + +3. Zun uses os-brick library to disconnect the volume:: + + device_info = brick_connector.disconnect_volume(conn_info) + +4. Zun makes an API call to Cinder to terminate the connection:: + + conn_info = cinderclient.volumes.terminate_connection(volume, ...) + +5. Zun makes an API call to Cinder to finalize the volume disconnection. + This will update the status of the volume from "detaching" to "available" + in Cinder:: + + cinderclient.volumes.detach(volume) + + +Fuxi driver +=========== +Fuxi is new OpenStack project which aims to integrate Cinder to Docker +volumes. Fuxi can be used as the unified persistence storage provider for +various storage services such as Cinder and Manila. + +The implementation of Cinder is enabled using Fuxi driver from Zun. We need +to implement Cinder driver in Zun which manages volumes, let Fuxi control the +mount/unmount volume from Docker container. + +There are two approaches Docker provides to add volume to Container. + +1. Using Docker run:: + + $ docker run -d --volume-driver=fuxi -v my-named-volume:/data --name web_app + +2. Create volume first & then add it to Container:: + + $ docker volume create --driver fuxi \ + --name my-named-volume \ + -o size=1 \ + -o fstype=ext4 \ + -o multiattach=true + + $ docker run -d --name web_app -v my-named-volume:/data + +I think we can support both. + +1. To implement the first approach, we need following changes - Introduce fields in Container API - volume-driver, vol-name, vol-size. - We pass call to Volume Driver to create volume. - Volume driver connects to Cinder & handles volume creation. -- Once volume is created in Cinder, then we finally go add volume-driver as Fuxi & add volume name which created in cinder. +- Once volume is created in Cinder, then we finally go add volume-driver + as Fuxi & add volume name which created in Cinder. - Fuxi should be installed in Docker host and configured with Cinder engine. -2. To Implement Second approach, we need following changes +2. To implement the second approach, we need following changes -- Introduce Volume API in Zun which has fields volume-driver, volume-name, volume-size etc. -- Volume API will connect to volume driver which will sit under /zun/volume/driver.py. +- Introduce Volume API in Zun which has fields volume-driver, volume-name, + volume-size etc. +- Volume API will connect to volume driver which will sit under + /zun/volume/driver.py. - Volume Driver connects to Cinder and handles volume creation in Cinder. -- Once the volume is created in Cinder, it communicates to Docker Volume API to attach the created volume in Docker. -- Docker Volume API use --driver=Fuxi which goes talks to Cinder and attach created Volume in Docker. -- Prerequisite here is, Fuxi should be installed on docker host & configured with Cinder. If not, it returns the 500 response. -- Also we need to introduce new Volume table which contains field vol-driver, vol-name, vol-size fields. -- We need to add storage section in conf file, where we can specify some default attributes like storage engine Cinder, Cinder endpoint etc. +- Once the volume is created in Cinder, it communicates to Docker Volume API + to attach the created volume in Docker. +- Docker Volume API use --driver=Fuxi which goes talks to Cinder and attach + created Volume in Docker. +- Prerequisite here is, Fuxi should be installed on Docker host & configured + with Cinder. If not, it returns the 500 response. +- Also we need to introduce new Volume table which contains field vol-driver, + vol-name, vol-size fields. +- We need to add storage section in conf file, where we can specify some + default attributes like storage engine Cinder, Cinder endpoint etc. - We also need to configure Cinder endpoint in Fuxi conf file. - We can use same implementation for Flocker also as it supports Cinder. -- I think if we can create separate CinderDriver which calls from Volume volume driver. This approach enables way to implement multiple storages supports in the future and we can plug-in multiple storage implementation. +- I think if we can create separate CinderDriver which calls from Volume + volume driver. This approach enables way to implement multiple storages + supports in the future and we can plug-in multiple storage implementation. The diagram below offers an overview of the system architecture. The Zun service may communicate with Fuxi and fuxi talks to Cinder for volumes. @@ -116,13 +213,14 @@ service may communicate with Fuxi and fuxi talks to Cinder for volumes. Design Principles ----------------- 1. Similar user experience between VMs and containers. In particular, the ways - to configure volumes of a container should be similar as the VM equivalent. + to configure volumes of a container should be similar to the VM equivalent. + 2. Full-featured container APIs. Alternatives ------------ -1. We can use rexray for storage support, its again third party tool which +1. We can use rexray [2] for storage support, its again third party tool which increases the dependency. Data model impact @@ -136,8 +234,11 @@ REST API impact We need to add below APIs 1. Create a volume - POST /v1/volumes + 2. List volumes - GET /v1/volumes + 3. Inspect volume - GET /v1/volumes/ + 4. Delete Volume - DELETE /v1/volumes/ Security impact @@ -175,7 +276,6 @@ Implementation Assignee(s) ----------- - Primary assignee: Digambar @@ -184,13 +284,18 @@ Other contributors: Work Items ---------- - 1. We need to introduce new Volume API. -2. Implement volume driver in zun. + +2. Implement volume driver in Zun. + 3. Implement Cinder calls under the volume driver. + 4. Implement Docker volume support in Zun. + 5. Add volume section in zun.conf. + 6. Add volume-driver support in CLI. + 7. Implement unit/integration test. @@ -207,3 +312,9 @@ Each patch will have unit tests, and Tempest functional tests covered. Documentation Impact ==================== A set of documentation for this new feature will be required. + +References +========== +[1] https://docs.docker.com/engine/tutorials/dockervolumes/ + +[2] https://github.com/codedellemc/rexray