NVMe connector support MD replication spec.
Implements: blueprint nvme-of-add-client-raid-1 Change-Id: Ibeebc62ec649933747b30537d3a6d4d84641fad0
This commit is contained in:
Zohar Mamedov
parent
30466d896b
commit
d6e0be38df
250
specs/wallaby/nvme-connector-md-support.rst
Normal file
250
specs/wallaby/nvme-connector-md-support.rst
Normal file
@ -0,0 +1,250 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==========================================
|
||||
NVMe Connector Support MDRAID replication
|
||||
==========================================
|
||||
|
||||
https://blueprints.launchpad.net/cinder/+spec/nvme-of-add-client-raid-1
|
||||
|
||||
NVMe connector replicated volume support via MDRAID.
|
||||
Allow OpenStack to use replicated NVMe volumes that are distributed on
|
||||
scale-out storage.
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
When consuming block storage, resilience of the block volumes is required.
|
||||
This can be achieved on the storage backend with NVMe and iSCSI though the
|
||||
target will remain a single point of failure. And with multipathing, the HA
|
||||
of paths to a target does not handle volume data replication.
|
||||
|
||||
A storage solution exposing high performance NVMeoF storage would need a
|
||||
way to handle volume replication. We propose achieving this by using RAID1
|
||||
on the host, since the relative performance impact will be less significant
|
||||
in such an environment and will be greatly outweighed by the benefits.
|
||||
|
||||
|
||||
Use Cases
|
||||
=========
|
||||
|
||||
When resiliency is needed for NVMeoF storage.
|
||||
|
||||
Taking advantage of NVMe over a high performance fabric, gain benefits of
|
||||
replication on the host while maintaining good performance.
|
||||
|
||||
In this case volume replica failures will have no impact on the consumer,
|
||||
and will allow for self healing to take place seamlessly.
|
||||
|
||||
For developers of volume drivers, this will allow support for OpenStack
|
||||
to use replicated volumes that they would expose in their storage backend.
|
||||
|
||||
End users operating in this mode will benefit from performance of NVMe with
|
||||
added resilience due to volume replication.
|
||||
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
Expand the NVMeoF connector in os-brick to be able to take in connection
|
||||
information for replicated volumes.
|
||||
|
||||
When `connect_volume` is called with replicated volume information, NVMe
|
||||
connect to all replica targets and create a RAID1 over the devices.
|
||||
|
||||
.. image:: https://wiki.openstack.org/w/images/a/ab/Nvme-of-add-client-raid1-detail.png
|
||||
:width: 448
|
||||
:alt: NVMe client RAID diagram
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
Replication can also be actively maintained on the backend side, which is
|
||||
the way it is commonly done. However, with NVMe on a high performance fabric,
|
||||
the benefits of handling replication on the consumer host can outweigh the
|
||||
performance costs.
|
||||
|
||||
Adding support for this also increases the type of backends we fully support,
|
||||
as not all vendors will chose supporting replication on the backend side.
|
||||
|
||||
And we can support both without any kind of impact on each one.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
NVMe Connector's `connect_volume` and `disconnect_volume` param
|
||||
`connection_properties` can now also hold a `volume_replicas` list, which will
|
||||
contain necessary info for connecting to and identifying the NVMe subsystems
|
||||
for then doing MDRAID replication over them.
|
||||
|
||||
Each replica dict in `volume_replicas` list must include the normal flat
|
||||
connection properties.
|
||||
|
||||
If `volume_replicas` has only one replica, treat it as non-replicated volume.
|
||||
|
||||
If `volume_replicas` is ommitted, use the normal flat connection properties
|
||||
for NVMe as in the existing version of this NVMe connector.
|
||||
|
||||
non-replicated volume connection properties::
|
||||
|
||||
{'uuid': '96e25fb4-9f91-4c88-ab59-275fd354777e',
|
||||
'nqn': 'nqn.2014-08.org.nvmexpress:uuid:...'
|
||||
'portals': [{
|
||||
'address': '10.0.0.101',
|
||||
'port': '4420',
|
||||
'transport': 'tcp'
|
||||
}],
|
||||
'volume_replicas': None}
|
||||
|
||||
replicated volume connection properties::
|
||||
|
||||
{'volume_replicas': [
|
||||
{'uuid': '96e25fb4-9f91-4c88-ab59-275fd354777e',
|
||||
'nqn': 'nqn.2014-08.org.nvmexpress:uuid:...'
|
||||
'portals': [{
|
||||
'address': '10.0.0.101',
|
||||
'port': '4420',
|
||||
'transport': 'tcp'
|
||||
}]},
|
||||
{'uuid': '12345fb4-9f91-4c88-ab59-275fd3544321',
|
||||
'nqn': 'nqn.2014-08.org.nvmexpress:uuid:...'
|
||||
'portals': [{
|
||||
'address': '10.0.0.110',
|
||||
'port': '4420',
|
||||
'transport': 'tcp'
|
||||
}]},
|
||||
]}
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
Requires elevated priviliges for managing MDRAID.
|
||||
(Current NVMe connector already does sudo executions of nvme cli, so this
|
||||
change will just add execution of `mdadm`)
|
||||
|
||||
|
||||
Active/Active HA impact
|
||||
-----------------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
Working in this replicated mode will allow for special case scenarios where for
|
||||
example a MDRAID array with 4 replicas loses connection to two of the replicas,
|
||||
keeps writing data to two remaining ones. Then, after a re-attach from a reboot
|
||||
or a migration, for some reason now has access to only the two originally
|
||||
lost replicas, and not the two "good" ones, then the re-created MDRAID array
|
||||
will have old / bad data.
|
||||
|
||||
The above can be remedied by storage backend awareness of devices going faulty
|
||||
in the array. This is enabled by the NVMe monitoring agent, which can recognize
|
||||
replicas going faulty in an array and notify the storage backend, which will
|
||||
mark these replicas as faulty for the replicated volume.
|
||||
|
||||
Multi attach is not supported for NVMe MDRAID volumes.
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
Replicated volume attachments will be slower (need to build MDRAID array).
|
||||
It's a fair tradeoff, slower attachment for more resiliency.
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
NVMe and MDRAID and their CLI clients (`nvme` and `mdadm`) need to be
|
||||
available on the hosts for NVMe connections and RAID replication respectively.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
Gives option for storage vendors to support replicated NVMeoF volumes via
|
||||
their driver.
|
||||
|
||||
To use this feature, volume drivers will need to expose NVMe storage that is
|
||||
replicated and provide necessary connection information for it when using this
|
||||
feature of the connector.
|
||||
|
||||
This would not affect non-replicated volumes.
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Zohar Mamedov
|
||||
zoharm
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
All done in NVMe connector:
|
||||
|
||||
- In `connect_volume` parse connection information for replicated volumes.
|
||||
- Connect to NVMeoF targets and identify the devices.
|
||||
- Create MD RAID1 array over devices.
|
||||
- Return symlink to MDRAID device.
|
||||
- `disconnect_volume` destroy the MDRAID.
|
||||
- `extend_volume` grow the MDRAID.
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
NVMe and MDRAID and their CLI clients (`nvme` and `mdadm`) need to be
|
||||
available on the hosts for NVMe connections and RAID replication respectively.
|
||||
Fail gracefully if they are not found.
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
In order to properly test this in tempest, programmatic access will be needed
|
||||
to the storage backend. For example, to fail one of the drives of a replicated
|
||||
volume.
|
||||
|
||||
We could also slide by with just a check of connected NVMe subsystems
|
||||
(`nvme list`) and scan of MDRAID arrays (`mdadm -D scan`) to see that multiple
|
||||
NVMe devices were connected and a RAID was created.
|
||||
|
||||
In either case tempest will need to be aware that the storage backend is
|
||||
configured to use replicated NVMe volumes and only then do these checks.
|
||||
|
||||
Aside from that, running tempest with NVMe replicated volume backend
|
||||
will still fully test this functionality, it's just the specific assertions
|
||||
(nvme devices and RAID info) that would be different.
|
||||
|
||||
Finally, we will start with unit tests, and have functional tests in os-brick
|
||||
as a stretch goal.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
Document that NVMe connector will now support replicated volumes and that
|
||||
connection information of replicas is required from the volume driver to
|
||||
support it.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
Architectural diagram
|
||||
https://wiki.openstack.org/wiki/File:Nvme-of-add-client-raid1-detail.png
|
||||
Loading…
Reference in New Issue
Block a user