Support modern compression algorithms in cinder backup

Change-Id: I866bf7af44f975842271fdd70e19102c16f8f671
Implements: blueprint support-modern-compression-algorithms-in-cinder-backup

doc/source/index.rst

@@ -7,16 +7,11 @@ Block Storage Service Specifications (cinder)
 Victoria approved specs
 =======================
 
-.. any patch adding a spec should remove the following line:
-
-* No specs have yet been approved.
-
 .. toctree::
    :glob:
    :maxdepth: 1
 
-.. any patch adding a spec should uncomment the following line:
+   specs/victoria/*
-..   specs/victoria/*
 
 Ussuri approved specs
 =====================

specs/victoria/support-modern-compression-algorithms-in-cinder-backup.rst

@@ -0,0 +1,163 @@
+..
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
+ License.
+
+ http://creativecommons.org/licenses/by/3.0/legalcode
+
+======================================================
+Support modern compression algorithms in cinder backup
+======================================================
+
+https://blueprints.launchpad.net/cinder/+spec/support-modern-compression-algorithms-in-cinder-backup
+
+This blueprint proposes to add support of modern compression algorithms like
+zstd to cinder backup.
+
+Problem description
+===================
+
+Cinder backup currently supports only gzip and bzip2 to compress/decompress the
+volume data during backup and restore operation.
+
+It should support other compression algorithms as well because efficiency of
+compression entirely depends on cloud provider's environment and choice. Some
+may want fast compression compromising the compression ratio, so to provide more
+choices to cloud provider, it will be good to add support of other algorithm
+like zstd.
+
+Zstandard(zstd) gives high compression ratio(comparable to zlib) with very much
+fast speed. It's now used as package compression algorithm for arch linux.
+
+https://facebook.github.io/zstd/
+
+Some general benchmark result is available on the website introduced above, or
+lzbench website.
+
+https://github.com/inikep/lzbench/
+
+Use Cases
+=========
+
+* Cinder backup with high-speed compression algorithm
+
+Proposed change
+===============
+
+* add zstd algorithm package details in requirements.txt.
+
+* add the support of zstd algorithm in get compressor routine of cinder backup.
+
+Alternatives
+------------
+
+None.
+
+Data model impact
+-----------------
+
+None.
+
+REST API impact
+---------------
+
+None.
+
+Security impact
+---------------
+
+None.
+
+Notifications impact
+--------------------
+
+None.
+
+Other end user impact
+---------------------
+
+None.
+
+Performance Impact
+------------------
+
+* Cinder backup and restore operation gets faster.
+* Size of backup data may be bigger than zlib or bz2 compression algorithm
+  (depends on content of data volume)
+
+
+* Simple performance test result I have done
+
+  * Server spec: 8core CPU, 16GB memory
+  * Backup store is NFS-backend on localhost (i.e. small network bottleneck)
+  * Backup/restore a 16GB volume created from a Windows image (qcow2
+    12001017856 bytes)
+  * Measured backup/restore time backup data size, and size ratio versus size of
+    qcow2 (considering as approximate amount of data in the volume)
+    The smaller the better for all values.
+  * CPU usage for algo other than zstd is about 100% to 110% (i.e. Using only
+    one core.) where max usage is 800% (= 8cores * 100%).
+  * CPU usage for zstd is about 80% to 400% (utilizes multi cores).
+  * Famous algorithm like gzip may performs better if offloaded to hardware.
+
+================== ================ ================= =========== =============
+Algorithm          backup time(sec) restore time(sec) size(bytes) v.s. qcow2(%)
+================== ================ ================= =========== =============
+none                          205.8             109.2 17218167818         143.5
+gzip                          743.2             141.5  6658990044          55.5
+bz2                          2102.5             742.2  6480248772          54.0
+zstd(to be added)             196.2             100.2  6286227236          52.4
+lz4(for reference)            211.0             113.0  7903490364          65.9
+================== ================ ================= =========== =============
+
+Other deployer impact
+---------------------
+
+The python module `zstd <https://pypi.org/project/zstd/>`_ is required to use
+zstd compression algorithm.
+
+Developer impact
+----------------
+
+None.
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+
+Primary assignee: Kazufumi Noto
+
+Other contributors: None
+
+Work Items
+----------
+
+* Add the package details to cinder/requirements.txt.
+* Implement the code to add support of zstd compression.
+* Documents need to be updated to reflect the added support.
+* Add test cases to create backup using zstd compression.
+
+Dependencies
+============
+
+None
+
+Testing
+=======
+
+* Unit tests will be added to create backup using zstd compression.
+
+Documentation Impact
+====================
+
+The following documents need to be updated to reflect the added support:
+
+* cinder sample configuration file in configuration reference.
+
+References
+==========
+
+* https://facebook.github.io/zstd/
+
+* https://github.com/inikep/lzbench/