Over subscription in thin provisioning
This spec proposes to allow over subscription in thin provisioning. blueprint over-subscription-in-thin-provisioning Change-Id: Ibdde48c22ae9454d181d909f0897f5935dda4e68
This commit is contained in:
Xing Yang
parent
2f24768ac6
commit
54f988971c
407
specs/kilo/over-subscription-in-thin-provisioning.rst
Normal file
407
specs/kilo/over-subscription-in-thin-provisioning.rst
Normal file
@ -0,0 +1,407 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==========================================
|
||||
Over Subscription in Thin Provisioning
|
||||
==========================================
|
||||
|
||||
Include the URL of your launchpad blueprint:
|
||||
|
||||
https://blueprints.launchpad.net/cinder/+spec/over-subscription-in-thin-provisioning
|
||||
|
||||
This proposal is to introduce a mechanism to allow over subscription in thin
|
||||
provisioning and also introduce a used ratio to prevent over provisioning.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
There are a couple of problems in capacity reporting in thin provisioning:
|
||||
|
||||
* Some drivers are still using 'infinite' and 'unknown' to report capacity.
|
||||
This could lead to over provisioning.
|
||||
|
||||
* There's no mechanism to allow over subscription in thin provisioning.
|
||||
Over subscription allows flexibility in storage allocation and usage and
|
||||
it is an important concept in providing thin provisioning support.
|
||||
|
||||
Terminologies
|
||||
===============
|
||||
|
||||
The following terminologies will be used in this spec:
|
||||
|
||||
total_capacity: This is an existing parameter already reported by the driver.
|
||||
It is the total physical capacity.
|
||||
Example: Assume backend A has a total physical capacity of 100G.
|
||||
|
||||
available_capacity: This is an existing parameter already reported by the
|
||||
driver. It is the real physical capacity available to be used.
|
||||
Example: Assume backend A has a total physical capacity of 100G.
|
||||
There are 10G thick luns and 20G thin luns (10G out of the 20G thin luns
|
||||
are written). In this case, available_capacity = 100 - 10 -10 = 80G.
|
||||
|
||||
used_capacity: This parameter is calculated by the difference between
|
||||
total_capacity and available_capacity. It is used below for calculating
|
||||
used ratio.
|
||||
|
||||
volume_size: This is an existing parameter. It is the size of the volume to
|
||||
be provisioned.
|
||||
|
||||
provisioned_capacity: This is a new parameter. It is the apparent allocated
|
||||
space indicating how much capacity has been provisioned.
|
||||
Example: User A created 2x10G volumes in Cinder from backend A, and
|
||||
user B created 3x10G volumes from backend A directly, without using Cinder.
|
||||
Assume those are all the volumes provisioned on backend A. The total
|
||||
provisioned_capacity will be 50G and that is what the driver should be
|
||||
reporting.
|
||||
|
||||
allocated_capacity: This is an existing parameter. Cinder uses this to
|
||||
keep track of how much capacity has been allocated through Cinder.
|
||||
Example: Using the same example above for provisioned_capacity, the
|
||||
allocated_capacity will be 20G because that is what has been provisioned
|
||||
through Cinder. allocated_capacity is documented here to differentiate
|
||||
from the new parameter provisioned_capacity. If a driver can guarantee
|
||||
that the backend is used solely for a specific Cinder installation,
|
||||
then provisioned_capacity could be the same as allocated_capacity.
|
||||
Only driver has knowledge of this and therefore driver can make a
|
||||
decision on how to report provisioned_capacity.
|
||||
|
||||
max_over_subscription_ratio: This is a new parameter. It is the
|
||||
float representation of the over subscription ratio when thin provisioning
|
||||
is involved. It is a ratio of provisioned capacity over total capacity.
|
||||
Default ratio is 1.0, meaning provisioned capacity cannot exceed the
|
||||
total capacity. Note that this ratio is per backend or per pool depending
|
||||
on driver implementation.
|
||||
|
||||
max_used_ratio: This is a new parameter. It is the float representation
|
||||
of the maximum allowed usage. It is a ratio of used capacity over total
|
||||
capacity before the backend is no longer considered a valid choice by
|
||||
the scheduler. Default is 0.95. Note this refers to the real capacity.
|
||||
Note that this ratio is per backend or per pool depending on driver
|
||||
implementation.
|
||||
|
||||
apparent_available_capacity: This is a parameter calculated based on other
|
||||
parameters. It refers to how much space a user can still provision apparently.
|
||||
Note that this is different from available_capacity. available_capacity is
|
||||
about the real available physical capacity. apparent_available_capacity is
|
||||
about apparent available virtual capacity.
|
||||
Example: Assume the total capacity of backend A is 100G. The max over
|
||||
subscription ratio is 2.0. If no volumes have been provisioned yet,
|
||||
the apparent_available_capacity is 100 x 2.0 = 200. If 50G volumes have
|
||||
already been provisioned, the apparent_available_capacity is 200 - 50 = 150.
|
||||
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
New parameters in get_volume_stats
|
||||
----------------------------------
|
||||
Two new configuration options "max_over_subscription_ratio" and
|
||||
"max_used_ratio" will be added to cinder/volume/driver.py. These two
|
||||
configuration options will be added to cinder.conf. They can be configured
|
||||
for each backend when multiple-backend is enabled.
|
||||
|
||||
Note: These two configuration options are provided as a reference
|
||||
implementation and will be used by the LVM driver. However, it is not a
|
||||
requirement for a driver to use these two options from cinder.conf. Driver
|
||||
can choose its own way if that makes more sense.
|
||||
|
||||
In scheduler/host_manager.py, the following additional information sent by
|
||||
each backend will be saved and will be used later by the scheduler to make
|
||||
decisions. Note these are reported for a backend or a pool, depending on
|
||||
driver implementation. In the LVM driver, these will be reported for a
|
||||
backend (which is treated as one pool).
|
||||
|
||||
* provisioned_capacity
|
||||
* max_over_subscription_ratio
|
||||
* max_used_ratio
|
||||
|
||||
The two configuration options max_over_subscription_ratio and
|
||||
max_used_ratio added in cinder.conf are for configuring a backend.
|
||||
For a driver that supports multiple pools per backend, it can report
|
||||
these ratios for each pool in get_volume_stats.
|
||||
|
||||
* Driver can get these two options in cinder.conf and report the same ratios
|
||||
for the pools that belonging to the same backend.
|
||||
* Alternatively driver can choose to report different ratios for each pool
|
||||
in get_volume_stats, without using these two options in cinder.conf.
|
||||
|
||||
Capabilities
|
||||
------------
|
||||
Driver can report the following capabilities for a backend or a pool:
|
||||
|
||||
thin_provisioning_support = True (or False)
|
||||
thick_provisioning_support = True (or False)
|
||||
|
||||
Two capabilities are added here to allow a backend or pool to claim support
|
||||
for thin provisioning, or thick provisioning, or both.
|
||||
|
||||
Volume type extra specs
|
||||
-----------------------
|
||||
If volume type is provided as part of the volume creation request, it can
|
||||
have the following extra specs defined:
|
||||
|
||||
Key: "thin_provisioning_support", Value: True or False
|
||||
Key: "thick_provisioning_support", Value: True or False
|
||||
|
||||
The above extra specs are used by the scheduler to find a backend that
|
||||
supports thin provisioning, thick provisioning, or both to match the needs
|
||||
of a specific volume type.
|
||||
|
||||
If an extra spec scope key "provisioning:type" is defined, it can be used
|
||||
by the driver to detemine whether the lun to be provisioned is thin or thick.
|
||||
The value of this extra spec is either "thin" or "thick".
|
||||
|
||||
Capacity filter
|
||||
---------------
|
||||
In the capacity filter, the following will be evaluated in the decision making
|
||||
when choosing a backend that fits the criteria:
|
||||
|
||||
If (provisioned_capacity + volume_size) / total_capacity >=
|
||||
max_over_subscription_ratio, the backend will not be chosen to provision
|
||||
the volume.
|
||||
Note: This formula will be executed only if "thin_provisioning_support"
|
||||
is True.
|
||||
|
||||
If available_capacity < volume_size, the backend will not be chosen
|
||||
to provision the volume. Note this check is already in the capacity filter.
|
||||
|
||||
If (available_capacity * max_over_subscription_ratio) < volume_size,
|
||||
the backend will not be chosen to provision the volume.
|
||||
Note: This formula will be executed only if "thin_provisioning_support"
|
||||
is True.
|
||||
|
||||
Taking a conservative approach, we assume that the new volume will be
|
||||
fully written even for "thin" luns:
|
||||
used = (total_capacity - available_capacity - volume_size) / total_capacity
|
||||
If used > max_used_ratio, this backend will not be chosen to provision the
|
||||
volume.
|
||||
|
||||
Capacity weigher
|
||||
----------------
|
||||
In the capacity weigher, apparent available capacity should be used in
|
||||
addition to real free capacity for ranking.
|
||||
apparent_available_capacity = total_capacity * max_over_subscription_ratio -
|
||||
provisioned_capacity
|
||||
Note: apparent_available_capacity should only be used if
|
||||
"thin_provisioning_support" is True.
|
||||
|
||||
LVM driver
|
||||
----------
|
||||
In the default LVM driver, changes will be made in get_volume_stats which
|
||||
periodically reports capabilities and the information will be received by the
|
||||
scheduler.
|
||||
|
||||
* Changes will be made in the LVM driver to report provisioned_capacity.
|
||||
It makes calls to the LVM class in brick to retrieve volume information
|
||||
including capacities.
|
||||
|
||||
* The LVM driver will also report max_over_subscription_ratio and
|
||||
max_used_ratio. Those will be from the configuration parameters set in
|
||||
cinder.conf.
|
||||
|
||||
* While other drivers need to report max_over_subscription_ratio and
|
||||
max_used_ratio, they are not required to read those ratios from
|
||||
cinder.conf.
|
||||
|
||||
Changes will also be made in the following LVM driver functions to make sure
|
||||
over provisioning will not happen even when a request didn't go through the
|
||||
scheduler:
|
||||
|
||||
* create_volume
|
||||
* extend_volume
|
||||
|
||||
The following will be evaluated in the above LVM driver functions:
|
||||
|
||||
* If the ratio of the apparent provisioned capacity over real total capacity
|
||||
has exceeded the over subscription ratio, the operation will fail.
|
||||
|
||||
* If the ratio of real used capacity over total physical capacity has exceeded
|
||||
the used ratio, the operation will fail.
|
||||
|
||||
Use cases
|
||||
---------
|
||||
The design of this feature will support the following use cases.
|
||||
|
||||
Use case 1:
|
||||
Each volume type has a separate backend or pool. For example, Gold volume
|
||||
type uses pool gold, Silver volume type uses pool silver, and Bronze
|
||||
volume type uses pool bronze. Each pool can have a different max over
|
||||
subscription ratio.
|
||||
|
||||
Use case 2:
|
||||
One volume type is associated with multiple backends or pools. For example,
|
||||
Silver volume type uses pool 1 and pool 2. Both pools can have the same
|
||||
max over subscription ratio. Note that capacities for each pool can be
|
||||
different at any given time.
|
||||
|
||||
Use case 3:
|
||||
One backend or pool is used by multiple volume types. For example, pool 3
|
||||
is used by volume types Gold, Silver, and Bronze. Assume Gold volume type
|
||||
uses thick luns only, Silver volume type can have either thick or thin
|
||||
luns, and Bronze volume type has thin luns only. Because the over subscription
|
||||
ratio is calculated by the ratio of provisioned_capacity over total_capacity
|
||||
and all three volume types are sharing the same pool, the ratio will be
|
||||
the same for all volume types. Gold volume type can guarantee its space
|
||||
reservation by creating thick luns. The apparent size and the used size
|
||||
of a Gold volume will always be the same. For a thin lun created as
|
||||
Silver or Bronze volume type, the apparent size can be bigger than
|
||||
the real size. Some detailed examples are shown at this etherpad:
|
||||
https://etherpad.openstack.org/p/cinder-over-subscription-white-board
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
Without this, we cannot support over subscription in thin provisioning and
|
||||
there's also no upper limit that prevents over provisioning from happening.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
N/A
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
N/A
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
N/A
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
If the capacity usage has exceeded the used ratio or if the provisioned
|
||||
capacity has exceeded the over subscription ratio, a notification should be
|
||||
sent. The notification should report the name of the backend or pool and the
|
||||
capacity information from the backend or pool. The purpose of the
|
||||
notification is for the storage administrator to take notice and take actions
|
||||
to fix the problem.
|
||||
|
||||
Notification will also be sent periodically whenever the scheduler receives
|
||||
an update of capacities from the backend. This will be consumed by Ceilometer.
|
||||
This was discussed for the Capacity Headroom topic at the summit. The
|
||||
Ceilometer team will be responsible for the changes required on the Ceilometer
|
||||
side for this.
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
There are two new parameters in cinder.conf that end user
|
||||
needs to be aware of.
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
N/A
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
New parameters over_subscription_ratio and used_ratio will be
|
||||
added to cinder.conf.
|
||||
|
||||
Volume type extra specs for "provisioning:type" should be added
|
||||
for thin or thick.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
Drivers should report provisioning_capabilities (thin, thick, or both).
|
||||
|
||||
Drivers supporting thin provisioning should report provisioned capacity
|
||||
in addition to available capacity in get_volume_stats.
|
||||
|
||||
For drivers supporting thick provisioning only, available capacity will be
|
||||
used just as before.
|
||||
|
||||
For drivers supporting both thin and thick provisioning, provisioned capacity
|
||||
and available capacity should both be reported.
|
||||
|
||||
If there is a range regarding capacity and you are not sure how to report,
|
||||
please be conservative. For example, if the available capacity is in the
|
||||
range of 80 to 100 GB, be conservative and report the lower bound 80 GB.
|
||||
|
||||
Note: This work is also needed for Cinder to use ThinLVM as the default driver
|
||||
in Kilo.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
xing-yang
|
||||
|
||||
Other contributors:
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. Add max_over_subscription_ratio and max_used_ratio in driver.py.
|
||||
|
||||
2. Modify host_manager.py to update provisioned capacity, over
|
||||
subscription ratio, and used ratio reported by the backends.
|
||||
|
||||
3. Modify capacity filter to check whether over subscription ratio
|
||||
or used ratio has been exceeded in a backend.
|
||||
|
||||
4. New parameters max_over_subscription_ratio and max_used_ratio will be
|
||||
added to cinder.conf.
|
||||
|
||||
5. LVM driver will be changed to report virtual capacity, over
|
||||
subscrption ratio, and used ratio.
|
||||
|
||||
6. LVM class in brick will be updated to calculate provisioned capacity.
|
||||
|
||||
7. LVM driver functions will be changed to check whether over
|
||||
subscription ratio or used ratio has been exceeded.
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
N/A
|
||||
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
New unit tests will be added to test the changed code.
|
||||
Testing will be done using the LVM driver for thin provisioning.
|
||||
Testing will be done to cover the 3 use cases described above.
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
Documentation changes are needed for the following:
|
||||
New parameters max_over_subscription_ratio and max_used_ratio will be
|
||||
added to cinder.conf.
|
||||
Driver needs to add provisioning_capabilities (thick, thin, or both).
|
||||
Volume type extra specs need a key "provisioning:type" and value
|
||||
"thin" or "thick".
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
Examples:
|
||||
https://etherpad.openstack.org/p/cinder-over-subscription-white-board
|
||||
|
||||
Virtual capacity "provisioned_capacity_gb" was discussed in Winston's spec
|
||||
https://review.openstack.org/#/c/105190/6/specs/juno/volume-statistics-reporting.rst
|
||||
|
||||
Kilo design summit session on this topic:
|
||||
https://etherpad.openstack.org/p/kilo-cinder-over-subscription
|
||||
https://etherpad.openstack.org/p/kilo-cinder-capacity-headroom
|
||||
|
||||
Documentation on the filter scheduler:
|
||||
http://docs.openstack.org/developer/nova/devref/filter_scheduler.html
|
||||
Note: This is a document on Nova filter scheduler, but it is very similar to
|
||||
the Cinder filter scheduler.
|
||||
Loading…
Reference in New Issue
Block a user