Redfish driver firmware update
This patch adds support for performing firmware updates using the redfish and idrac hardware types. Co-Authored-By: Aija Jaunteva <aija.jaunteva@dell.com> Change-Id: Ie8f3f68c4a771121ec0ee13ce9349c7cd2b1e567 Depends-On: https://review.opendev.org/#/c/745950 Story: 2008140 Task: 40872
This commit is contained in:
Christopher Dearborn
parent
e32e5f27a4
commit
f0e0ef634e
@ -17,7 +17,7 @@ hardware types, though with smaller feature sets.
|
|||||||
Key features of the Dell iDRAC driver include:
|
Key features of the Dell iDRAC driver include:
|
||||||
|
|
||||||
* Out-of-band node inspection
|
* Out-of-band node inspection
|
||||||
* Boot device management
|
* Boot device management and firmware management
|
||||||
* Power management
|
* Power management
|
||||||
* RAID controller management and RAID volume configuration
|
* RAID controller management and RAID volume configuration
|
||||||
* BIOS settings configuration
|
* BIOS settings configuration
|
||||||
@ -29,7 +29,7 @@ The ``idrac`` hardware type supports the following Ironic interfaces:
|
|||||||
|
|
||||||
* `BIOS Interface`_: BIOS management
|
* `BIOS Interface`_: BIOS management
|
||||||
* `Inspect Interface`_: Hardware inspection
|
* `Inspect Interface`_: Hardware inspection
|
||||||
* Management Interface: Boot device management
|
* `Management Interface`_: Boot device and firmware management
|
||||||
* Power Interface: Power management
|
* Power Interface: Power management
|
||||||
* `RAID Interface`_: RAID controller and disk management
|
* `RAID Interface`_: RAID controller and disk management
|
||||||
* `Vendor Interface`_: BIOS management
|
* `Vendor Interface`_: BIOS management
|
||||||
@ -265,6 +265,13 @@ The ``idrac-redfish`` inspect interface does not currently set ``pxe_enabled``
|
|||||||
on the ports. The user should ensure that ``pxe_enabled`` is set correctly on
|
on the ports. The user should ensure that ``pxe_enabled`` is set correctly on
|
||||||
the ports following inspection with the ``idrac-redfish`` inspect interface.
|
the ports following inspection with the ``idrac-redfish`` inspect interface.
|
||||||
|
|
||||||
|
Management Interface
|
||||||
|
====================
|
||||||
|
|
||||||
|
The management interface for ``idrac-redfish`` supports updating firmware on
|
||||||
|
nodes using a manual cleaning step.
|
||||||
|
|
||||||
|
See :doc:`/admin/drivers/redfish` for more information on firmware update support.
|
||||||
|
|
||||||
RAID Interface
|
RAID Interface
|
||||||
==============
|
==============
|
||||||
|
|||||||
@ -249,6 +249,136 @@ scenario.
|
|||||||
|
|
||||||
Make sure to use add the simple-init_ element when building the IPA ramdisk.
|
Make sure to use add the simple-init_ element when building the IPA ramdisk.
|
||||||
|
|
||||||
|
Firmware update using manual cleaning step
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
The ``redfish`` hardware type supports updating the firmware on nodes using a
|
||||||
|
manual cleaning step.
|
||||||
|
|
||||||
|
The firmware update cleaning step allows one or more firmware updates to be
|
||||||
|
applied to a node. If multiple updates are specified, then they are applied
|
||||||
|
sequentially in the order given. The server is rebooted once per update.
|
||||||
|
If a failure occurs, the cleaning step immediately fails which may result
|
||||||
|
in some updates not being applied. If the node is placed into maintenance
|
||||||
|
mode while a firmware update cleaning step is running that is performing
|
||||||
|
multiple firmware updates, the update in progress will complete, and processing
|
||||||
|
of the remaining updates will pause. When the node is taken out of maintenance
|
||||||
|
mode, processing of the remaining updates will continue.
|
||||||
|
|
||||||
|
When updating the BMC firmware, the BMC may become unavailable for a period of
|
||||||
|
time as it resets. In this case, it may be desireable to have the cleaning step
|
||||||
|
wait after the update has been applied before indicating that the
|
||||||
|
update was successful. This allows the BMC time to fully reset before further
|
||||||
|
operations are carried out against it. To cause the cleaning step to wait after
|
||||||
|
applying an update, an optional ``wait`` argument may be specified in the
|
||||||
|
firmware image dictionary. The value of this argument indicates the number of
|
||||||
|
seconds to wait following the update. If the ``wait`` argument is not
|
||||||
|
specified, then this is equivalent to ``wait 0``, meaning that it will not
|
||||||
|
wait and immediately proceed with the next firmware update if there is one,
|
||||||
|
or complete the cleaning step if not.
|
||||||
|
|
||||||
|
The ``update_firmware`` cleaning step accepts JSON in the following format::
|
||||||
|
|
||||||
|
[{
|
||||||
|
"interface": "management",
|
||||||
|
"step": "update_firmware",
|
||||||
|
"args": {
|
||||||
|
"firmware_images":[
|
||||||
|
{
|
||||||
|
"url": "<url_to_firmware_image1>",
|
||||||
|
"wait": <number_of_seconds_to_wait>
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"url": "<url_to_firmware_image2>"
|
||||||
|
},
|
||||||
|
...
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}]
|
||||||
|
|
||||||
|
The different attributes of the ``update_firmware`` cleaning step are as follows:
|
||||||
|
|
||||||
|
.. csv-table::
|
||||||
|
:header: "Attribute", "Description"
|
||||||
|
:widths: 30, 120
|
||||||
|
|
||||||
|
"``interface``", "Interface of the cleaning step. Must be ``management`` for firmware update"
|
||||||
|
"``step``", "Name of cleaning step. Must be ``update_firmware`` for firmware update"
|
||||||
|
"``args``", "Keyword-argument entry (<name>: <value>) being passed to cleaning step"
|
||||||
|
"``args.firmware_images``", "Ordered list of dictionaries of firmware images to be applied"
|
||||||
|
|
||||||
|
Each firmware image dictionary, is of the form::
|
||||||
|
|
||||||
|
{
|
||||||
|
"url": "<URL of firmware image file>",
|
||||||
|
"wait": <Optional time in seconds to wait after applying update>
|
||||||
|
}
|
||||||
|
|
||||||
|
The ``url`` argument in the firmware image dictionary is mandatory, while the
|
||||||
|
``wait`` argument is optional.
|
||||||
|
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
Only ``http`` and ``https`` URLs are currently supported in the ``url``
|
||||||
|
argument.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
At the present time, targets for the firmware update cannot be specified.
|
||||||
|
In testing, the BMC applied the update to all applicable targets on the
|
||||||
|
node. It is assumed that the BMC knows what components a given firmware
|
||||||
|
image is applicable to.
|
||||||
|
|
||||||
|
To perform a firmware update, first download the firmware to a web server that
|
||||||
|
the BMC has network access to. This could be the ironic conductor web server
|
||||||
|
or another web server on the BMC network. Using a web browser, curl, or similar
|
||||||
|
tool on a server that has network access to the BMC, try downloading
|
||||||
|
the firmware to verify that the URLs are correct and that the web server is
|
||||||
|
configured properly.
|
||||||
|
|
||||||
|
Next, construct the JSON for the firmware update cleaning step to be executed.
|
||||||
|
When launching the firmware update, the JSON may be specified on the command
|
||||||
|
line directly or in a file. The following
|
||||||
|
example shows one cleaning step that installs two firmware updates. The first
|
||||||
|
updates the BMC firmware followed by a five minute wait to allow the BMC time
|
||||||
|
to start back up. The second updates the firmware on all applicable NICs.::
|
||||||
|
|
||||||
|
[{
|
||||||
|
"interface": "management",
|
||||||
|
"step": "update_firmware",
|
||||||
|
"args": {
|
||||||
|
"firmware_images":[
|
||||||
|
{
|
||||||
|
"url": "http://192.0.2.10/BMC_4_22_00_00.EXE",
|
||||||
|
"wait": 300
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"url": "https://192.0.2.10/NIC_19.0.12_A00.EXE"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}]
|
||||||
|
|
||||||
|
Finally, launch the firmware update cleaning step against the node. The
|
||||||
|
following example assumes the above JSON is in a file named
|
||||||
|
``firmware_update.json``::
|
||||||
|
|
||||||
|
openstack baremetal node clean <ironic_node_uuid> --clean-steps firmware_update.json
|
||||||
|
|
||||||
|
In the following example, the JSON is specified directly on the command line::
|
||||||
|
|
||||||
|
openstack baremetal node clean <ironic_node_uuid> --clean-steps '[{"interface": "management", "step": "update_firmware", "args": {"firmware_images":[{"url": "http://192.0.2.10/BMC_4_22_00_00.EXE", "wait": 300}, {"url": "https://192.0.2.10/NIC_19.0.12_A00.EXE"}]}}]'
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
Firmware updates may take some time to complete. If a firmware update
|
||||||
|
cleaning step consistently times out, then consider performing fewer
|
||||||
|
firmware updates in the cleaning step or increasing
|
||||||
|
``clean_callback_timeout`` in ironic.conf to increase the timeout value.
|
||||||
|
|
||||||
|
.. warning::
|
||||||
|
Warning: Removing power from a server while it is in the process of updating
|
||||||
|
firmware may result in devices in the server, or the server itself becoming
|
||||||
|
inoperable.
|
||||||
|
|
||||||
.. _Redfish: http://redfish.dmtf.org/
|
.. _Redfish: http://redfish.dmtf.org/
|
||||||
.. _Sushy: https://opendev.org/openstack/sushy
|
.. _Sushy: https://opendev.org/openstack/sushy
|
||||||
.. _TLS: https://en.wikipedia.org/wiki/Transport_Layer_Security
|
.. _TLS: https://en.wikipedia.org/wiki/Transport_Layer_Security
|
||||||
|
|||||||
@ -80,6 +80,16 @@ opts = [
|
|||||||
'or as the octal number ``0o644`` in Python. '
|
'or as the octal number ``0o644`` in Python. '
|
||||||
'This setting must be set to the octal number '
|
'This setting must be set to the octal number '
|
||||||
'representation, meaning starting with ``0o``.')),
|
'representation, meaning starting with ``0o``.')),
|
||||||
|
cfg.IntOpt('firmware_update_status_interval',
|
||||||
|
min=0,
|
||||||
|
default=60,
|
||||||
|
help=_('Number of seconds to wait between checking for '
|
||||||
|
'completed firmware update tasks')),
|
||||||
|
cfg.IntOpt('firmware_update_fail_interval',
|
||||||
|
min=0,
|
||||||
|
default=60,
|
||||||
|
help=_('Number of seconds to wait between checking for '
|
||||||
|
'failed firmware update tasks')),
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@ -15,8 +15,11 @@
|
|||||||
|
|
||||||
import collections
|
import collections
|
||||||
|
|
||||||
|
from futurist import periodics
|
||||||
|
from ironic_lib import metrics_utils
|
||||||
from oslo_log import log
|
from oslo_log import log
|
||||||
from oslo_utils import importutils
|
from oslo_utils import importutils
|
||||||
|
from oslo_utils import timeutils
|
||||||
|
|
||||||
from ironic.common import boot_devices
|
from ironic.common import boot_devices
|
||||||
from ironic.common import boot_modes
|
from ironic.common import boot_modes
|
||||||
@ -24,12 +27,17 @@ from ironic.common import components
|
|||||||
from ironic.common import exception
|
from ironic.common import exception
|
||||||
from ironic.common.i18n import _
|
from ironic.common.i18n import _
|
||||||
from ironic.common import indicator_states
|
from ironic.common import indicator_states
|
||||||
|
from ironic.common import states
|
||||||
from ironic.common import utils
|
from ironic.common import utils
|
||||||
from ironic.conductor import task_manager
|
from ironic.conductor import task_manager
|
||||||
|
from ironic.conductor import utils as manager_utils
|
||||||
|
from ironic.conf import CONF
|
||||||
from ironic.drivers import base
|
from ironic.drivers import base
|
||||||
|
from ironic.drivers.modules import deploy_utils
|
||||||
from ironic.drivers.modules.redfish import utils as redfish_utils
|
from ironic.drivers.modules.redfish import utils as redfish_utils
|
||||||
|
|
||||||
LOG = log.getLogger(__name__)
|
LOG = log.getLogger(__name__)
|
||||||
|
METRICS = metrics_utils.get_metrics_logger(__name__)
|
||||||
|
|
||||||
sushy = importutils.try_import('sushy')
|
sushy = importutils.try_import('sushy')
|
||||||
|
|
||||||
@ -663,3 +671,332 @@ class RedfishManagement(base.ManagementInterface):
|
|||||||
"node %(uuid)s") % {'indicator': indicator,
|
"node %(uuid)s") % {'indicator': indicator,
|
||||||
'component': component,
|
'component': component,
|
||||||
'uuid': task.node.uuid})
|
'uuid': task.node.uuid})
|
||||||
|
|
||||||
|
@METRICS.timer('RedfishManagement.update_firmware')
|
||||||
|
@base.clean_step(priority=0, abortable=False, argsinfo={
|
||||||
|
'firmware_images': {
|
||||||
|
'description': (
|
||||||
|
'A list of firmware images to apply.'
|
||||||
|
),
|
||||||
|
'required': True
|
||||||
|
}})
|
||||||
|
def update_firmware(self, task, firmware_images):
|
||||||
|
"""Updates the firmware on the node.
|
||||||
|
|
||||||
|
:param task: a TaskManager instance containing the node to act on.
|
||||||
|
:param firmware_images: A list of firmware images are to apply.
|
||||||
|
:returns: None if it is completed.
|
||||||
|
:raises: RedfishError on an error from the Sushy library.
|
||||||
|
"""
|
||||||
|
node = task.node
|
||||||
|
|
||||||
|
LOG.debug('Updating firmware on node %(node_uuid)s with firmware '
|
||||||
|
'%(firmware_images)s',
|
||||||
|
{'node_uuid': node.uuid,
|
||||||
|
'firmware_images': firmware_images})
|
||||||
|
|
||||||
|
update_service = redfish_utils.get_update_service(task.node)
|
||||||
|
|
||||||
|
# The cleaning infrastructure has an exclusive lock on the node, so
|
||||||
|
# there is no need to get one here.
|
||||||
|
self._apply_firmware_update(node, update_service, firmware_images)
|
||||||
|
|
||||||
|
# set_async_step_flags calls node.save()
|
||||||
|
deploy_utils.set_async_step_flags(
|
||||||
|
node,
|
||||||
|
reboot=True,
|
||||||
|
skip_current_step=True,
|
||||||
|
polling=True)
|
||||||
|
|
||||||
|
manager_utils.node_power_action(task, states.REBOOT)
|
||||||
|
|
||||||
|
return deploy_utils.get_async_step_return_state(task.node)
|
||||||
|
|
||||||
|
def _apply_firmware_update(self, node, update_service, firmware_updates):
|
||||||
|
"""Applies the next firmware update to the node
|
||||||
|
|
||||||
|
Applies the first firmware update in the firmware_updates list to
|
||||||
|
the node.
|
||||||
|
|
||||||
|
Note that the caller must have an exclusive lock on the node and
|
||||||
|
the caller must ensure node.save() is called after making this
|
||||||
|
call.
|
||||||
|
|
||||||
|
:param node: the node to apply the next update to
|
||||||
|
:param update_service: the sushy firmware update service
|
||||||
|
:param firmware_updates: the remaining firmware updates to apply
|
||||||
|
"""
|
||||||
|
|
||||||
|
firmware_update = firmware_updates[0]
|
||||||
|
firmware_url = firmware_update['url']
|
||||||
|
|
||||||
|
LOG.debug('Applying firmware %(firmware_image)s to node '
|
||||||
|
'%(node_uuid)s',
|
||||||
|
{'firmware_image': firmware_url,
|
||||||
|
'node_uuid': node.uuid})
|
||||||
|
|
||||||
|
task_monitor = update_service.simple_update(firmware_url)
|
||||||
|
|
||||||
|
driver_internal_info = node.driver_internal_info
|
||||||
|
firmware_update['task_monitor'] = task_monitor.task_monitor
|
||||||
|
driver_internal_info['firmware_updates'] = firmware_updates
|
||||||
|
node.driver_internal_info = driver_internal_info
|
||||||
|
|
||||||
|
def _continue_firmware_updates(self, task, update_service,
|
||||||
|
firmware_updates):
|
||||||
|
"""Continues processing the firmware updates
|
||||||
|
|
||||||
|
Continues to process the firmware updates on the node.
|
||||||
|
|
||||||
|
Note that the caller must have an exclusive lock on the node.
|
||||||
|
|
||||||
|
:param task: a TaskManager instance containing the node to act on.
|
||||||
|
:param update_service: the sushy firmware update service
|
||||||
|
:param firmware_updates: the remaining firmware updates to apply
|
||||||
|
"""
|
||||||
|
|
||||||
|
node = task.node
|
||||||
|
firmware_update = firmware_updates[0]
|
||||||
|
wait_interval = firmware_update.get('wait')
|
||||||
|
if wait_interval:
|
||||||
|
time_now = str(timeutils.utcnow().isoformat())
|
||||||
|
firmware_update['wait_start_time'] = time_now
|
||||||
|
|
||||||
|
LOG.debug('Waiting at %(time)s for %(seconds)s seconds after '
|
||||||
|
'firmware update %(firmware_image)s on node %(node)s',
|
||||||
|
{'time': time_now,
|
||||||
|
'seconds': wait_interval,
|
||||||
|
'firmware_image': firmware_update['url'],
|
||||||
|
'node': node.uuid})
|
||||||
|
|
||||||
|
driver_internal_info = node.driver_internal_info
|
||||||
|
driver_internal_info['firmware_updates'] = firmware_updates
|
||||||
|
node.driver_internal_info = driver_internal_info
|
||||||
|
node.save()
|
||||||
|
return
|
||||||
|
|
||||||
|
if len(firmware_updates) == 1:
|
||||||
|
self._clear_firmware_updates(node)
|
||||||
|
|
||||||
|
LOG.info('Firmware updates completed for node %(node)s',
|
||||||
|
{'node': node.uuid})
|
||||||
|
|
||||||
|
manager_utils.notify_conductor_resume_clean(task)
|
||||||
|
else:
|
||||||
|
firmware_updates.pop(0)
|
||||||
|
self._apply_firmware_update(node,
|
||||||
|
update_service,
|
||||||
|
firmware_updates)
|
||||||
|
node.save()
|
||||||
|
manager_utils.node_power_action(task, states.REBOOT)
|
||||||
|
|
||||||
|
def _clear_firmware_updates(self, node):
|
||||||
|
"""Clears firmware updates from driver_internal_info
|
||||||
|
|
||||||
|
Note that the caller must have an exclusive lock on the node.
|
||||||
|
|
||||||
|
:param node: the node to clear the firmware updates from
|
||||||
|
"""
|
||||||
|
driver_internal_info = node.driver_internal_info
|
||||||
|
driver_internal_info.pop('firmware_updates', None)
|
||||||
|
node.driver_internal_info = driver_internal_info
|
||||||
|
node.save()
|
||||||
|
|
||||||
|
@METRICS.timer('RedfishManagement._query_firmware_update_failed')
|
||||||
|
@periodics.periodic(
|
||||||
|
spacing=CONF.redfish.firmware_update_fail_interval,
|
||||||
|
enabled=CONF.redfish.firmware_update_fail_interval > 0)
|
||||||
|
def _query_firmware_update_failed(self, manager, context):
|
||||||
|
"""Periodic job to check for failed firmware updates."""
|
||||||
|
|
||||||
|
filters = {'reserved': False, 'provision_state': states.CLEANFAIL,
|
||||||
|
'maintenance': True}
|
||||||
|
|
||||||
|
fields = ['driver_internal_info']
|
||||||
|
|
||||||
|
node_list = manager.iter_nodes(fields=fields, filters=filters)
|
||||||
|
for (node_uuid, driver, conductor_group,
|
||||||
|
driver_internal_info) in node_list:
|
||||||
|
try:
|
||||||
|
lock_purpose = 'checking async firmware update failed.'
|
||||||
|
with task_manager.acquire(context, node_uuid,
|
||||||
|
purpose=lock_purpose,
|
||||||
|
shared=True) as task:
|
||||||
|
if not isinstance(task.driver.management,
|
||||||
|
RedfishManagement):
|
||||||
|
continue
|
||||||
|
|
||||||
|
firmware_updates = driver_internal_info.get(
|
||||||
|
'firmware_updates')
|
||||||
|
if not firmware_updates:
|
||||||
|
continue
|
||||||
|
|
||||||
|
node = task.node
|
||||||
|
|
||||||
|
# A firmware update failed. Discard any remaining firmware
|
||||||
|
# updates so when the user takes the node out of
|
||||||
|
# maintenance mode, pending firmware updates do not
|
||||||
|
# automatically continue.
|
||||||
|
LOG.warning('Firmware update failed for node %(node)s. '
|
||||||
|
'Discarding remaining firmware updates.',
|
||||||
|
{'node': node.uuid})
|
||||||
|
|
||||||
|
task.upgrade_lock()
|
||||||
|
self._clear_firmware_updates(node)
|
||||||
|
|
||||||
|
except exception.NodeNotFound:
|
||||||
|
LOG.info('During _query_firmware_update_failed, node '
|
||||||
|
'%(node)s was not found and presumed deleted by '
|
||||||
|
'another process.', {'node': node_uuid})
|
||||||
|
except exception.NodeLocked:
|
||||||
|
LOG.info('During _query_firmware_update_failed, node '
|
||||||
|
'%(node)s was already locked by another process. '
|
||||||
|
'Skip.', {'node': node_uuid})
|
||||||
|
|
||||||
|
@METRICS.timer('RedfishManagement._query_firmware_update_status')
|
||||||
|
@periodics.periodic(
|
||||||
|
spacing=CONF.redfish.firmware_update_status_interval,
|
||||||
|
enabled=CONF.redfish.firmware_update_status_interval > 0)
|
||||||
|
def _query_firmware_update_status(self, manager, context):
|
||||||
|
"""Periodic job to check firmware update tasks."""
|
||||||
|
|
||||||
|
filters = {'reserved': False, 'provision_state': states.CLEANWAIT}
|
||||||
|
fields = ['driver_internal_info']
|
||||||
|
|
||||||
|
node_list = manager.iter_nodes(fields=fields, filters=filters)
|
||||||
|
for (node_uuid, driver, conductor_group,
|
||||||
|
driver_internal_info) in node_list:
|
||||||
|
try:
|
||||||
|
lock_purpose = 'checking async firmware update tasks.'
|
||||||
|
with task_manager.acquire(context, node_uuid,
|
||||||
|
purpose=lock_purpose,
|
||||||
|
shared=True) as task:
|
||||||
|
if not isinstance(task.driver.management,
|
||||||
|
RedfishManagement):
|
||||||
|
continue
|
||||||
|
|
||||||
|
firmware_updates = driver_internal_info.get(
|
||||||
|
'firmware_updates')
|
||||||
|
if not firmware_updates:
|
||||||
|
continue
|
||||||
|
|
||||||
|
self._check_node_firmware_update(task)
|
||||||
|
|
||||||
|
except exception.NodeNotFound:
|
||||||
|
LOG.info('During _query_firmware_update_status, node '
|
||||||
|
'%(node)s was not found and presumed deleted by '
|
||||||
|
'another process.', {'node': node_uuid})
|
||||||
|
except exception.NodeLocked:
|
||||||
|
LOG.info('During _query_firmware_update_status, node '
|
||||||
|
'%(node)s was already locked by another process. '
|
||||||
|
'Skip.', {'node': node_uuid})
|
||||||
|
|
||||||
|
@METRICS.timer('RedfishManagement._check_node_firmware_update')
|
||||||
|
def _check_node_firmware_update(self, task):
|
||||||
|
"""Check the progress of running firmware update on a node."""
|
||||||
|
|
||||||
|
node = task.node
|
||||||
|
|
||||||
|
firmware_updates = node.driver_internal_info['firmware_updates']
|
||||||
|
current_update = firmware_updates[0]
|
||||||
|
|
||||||
|
try:
|
||||||
|
update_service = redfish_utils.get_update_service(node)
|
||||||
|
except exception.RedfishConnectionError as e:
|
||||||
|
# If the BMC firmware is being updated, the BMC will be
|
||||||
|
# unavailable for some amount of time.
|
||||||
|
LOG.warning('Unable to communicate with firmware update service '
|
||||||
|
'on node %(node)s. Will try again on the next poll. '
|
||||||
|
'Error: %(error)s',
|
||||||
|
{'node': node.uuid,
|
||||||
|
'error': e})
|
||||||
|
return
|
||||||
|
|
||||||
|
wait_start_time = current_update.get('wait_start_time')
|
||||||
|
if wait_start_time:
|
||||||
|
wait_start = timeutils.parse_isotime(wait_start_time)
|
||||||
|
|
||||||
|
elapsed_time = timeutils.utcnow(True) - wait_start
|
||||||
|
if elapsed_time.seconds >= current_update['wait']:
|
||||||
|
LOG.debug('Finished waiting after firmware update '
|
||||||
|
'%(firmware_image)s on node %(node)s. '
|
||||||
|
'Elapsed time: %(seconds)s seconds',
|
||||||
|
{'firmware_image': current_update['url'],
|
||||||
|
'node': node.uuid,
|
||||||
|
'seconds': elapsed_time.seconds})
|
||||||
|
current_update.pop('wait', None)
|
||||||
|
current_update.pop('wait_start_time', None)
|
||||||
|
|
||||||
|
task.upgrade_lock()
|
||||||
|
self._continue_firmware_updates(task,
|
||||||
|
update_service,
|
||||||
|
firmware_updates)
|
||||||
|
else:
|
||||||
|
LOG.debug('Continuing to wait after firmware update '
|
||||||
|
'%(firmware_image)s on node %(node)s. '
|
||||||
|
'Elapsed time: %(seconds)s seconds',
|
||||||
|
{'firmware_image': current_update['url'],
|
||||||
|
'node': node.uuid,
|
||||||
|
'seconds': elapsed_time.seconds})
|
||||||
|
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
task_monitor = update_service.get_task_monitor(
|
||||||
|
current_update['task_monitor'])
|
||||||
|
except sushy.exceptions.ResourceNotFoundError:
|
||||||
|
# The BMC deleted the Task before we could query it
|
||||||
|
LOG.warning('Firmware update completed for node %(node)s, '
|
||||||
|
'firmware %(firmware_image)s, but success of the '
|
||||||
|
'update is unknown. Assuming update was successful.',
|
||||||
|
{'node': node.uuid,
|
||||||
|
'firmware_image': current_update['url']})
|
||||||
|
task.upgrade_lock()
|
||||||
|
self._continue_firmware_updates(task,
|
||||||
|
update_service,
|
||||||
|
firmware_updates)
|
||||||
|
return
|
||||||
|
|
||||||
|
if not task_monitor.is_processing:
|
||||||
|
# The last response does not necessarily contain a Task,
|
||||||
|
# so get it
|
||||||
|
sushy_task = task_monitor.get_task()
|
||||||
|
|
||||||
|
# Only parse the messages if the BMC did not return parsed
|
||||||
|
# messages
|
||||||
|
messages = []
|
||||||
|
if not sushy_task.messages[0].message:
|
||||||
|
sushy_task.parse_messages()
|
||||||
|
|
||||||
|
messages = [m.message for m in sushy_task.messages]
|
||||||
|
|
||||||
|
if (sushy_task.task_state == sushy.TASK_STATE_COMPLETED
|
||||||
|
and sushy_task.task_status in
|
||||||
|
[sushy.HEALTH_OK, sushy.HEALTH_WARNING]):
|
||||||
|
LOG.info('Firmware update succeeded for node %(node)s, '
|
||||||
|
'firmware %(firmware_image)s: %(messages)s',
|
||||||
|
{'node': node.uuid,
|
||||||
|
'firmware_image': current_update['url'],
|
||||||
|
'messages': ", ".join(messages)})
|
||||||
|
|
||||||
|
task.upgrade_lock()
|
||||||
|
self._continue_firmware_updates(task,
|
||||||
|
update_service,
|
||||||
|
firmware_updates)
|
||||||
|
else:
|
||||||
|
error_msg = (_('Firmware update failed for node %(node)s, '
|
||||||
|
'firmware %(firmware_image)s. '
|
||||||
|
'Error: %(errors)s') %
|
||||||
|
{'node': node.uuid,
|
||||||
|
'firmware_image': current_update['url'],
|
||||||
|
'errors': ", ".join(messages)})
|
||||||
|
LOG.error(error_msg)
|
||||||
|
|
||||||
|
task.upgrade_lock()
|
||||||
|
self._clear_firmware_updates(node)
|
||||||
|
manager_utils.cleaning_error_handler(task, error_msg)
|
||||||
|
else:
|
||||||
|
LOG.debug('Firmware update in progress for node %(node)s, '
|
||||||
|
'firmware %(firmware_image)s.',
|
||||||
|
{'node': node.uuid,
|
||||||
|
'firmware_image': current_update['url']})
|
||||||
|
|||||||
@ -245,6 +245,23 @@ class SessionCache(object):
|
|||||||
cls._sessions.pop(session_key, None)
|
cls._sessions.pop(session_key, None)
|
||||||
|
|
||||||
|
|
||||||
|
def get_update_service(node):
|
||||||
|
"""Get a node's update service.
|
||||||
|
|
||||||
|
:param node: an Ironic node object
|
||||||
|
:raises: RedfishConnectionError when it fails to connect to Redfish
|
||||||
|
:raises: RedfishError when the UpdateService is not registered in Redfish
|
||||||
|
"""
|
||||||
|
|
||||||
|
try:
|
||||||
|
return _get_connection(node, lambda conn: conn.get_update_service())
|
||||||
|
except sushy.exceptions.MissingAttributeError as e:
|
||||||
|
LOG.error('The Redfish UpdateService was not found for '
|
||||||
|
'node %(node)s. Error %(error)s',
|
||||||
|
{'node': node.uuid, 'error': e})
|
||||||
|
raise exception.RedfishError(error=e)
|
||||||
|
|
||||||
|
|
||||||
def get_system(node):
|
def get_system(node):
|
||||||
"""Get a Redfish System that represents a node.
|
"""Get a Redfish System that represents a node.
|
||||||
|
|
||||||
@ -253,40 +270,60 @@ def get_system(node):
|
|||||||
:raises: RedfishError if the System is not registered in Redfish
|
:raises: RedfishError if the System is not registered in Redfish
|
||||||
"""
|
"""
|
||||||
driver_info = parse_driver_info(node)
|
driver_info = parse_driver_info(node)
|
||||||
system_id = driver_info.get('system_id')
|
system_id = driver_info['system_id']
|
||||||
|
|
||||||
|
try:
|
||||||
|
return _get_connection(
|
||||||
|
node,
|
||||||
|
lambda conn, system_id: conn.get_system(system_id),
|
||||||
|
system_id)
|
||||||
|
except sushy.exceptions.ResourceNotFoundError as e:
|
||||||
|
LOG.error('The Redfish System "%(system)s" was not found for '
|
||||||
|
'node %(node)s. Error %(error)s',
|
||||||
|
{'system': system_id or '<default>',
|
||||||
|
'node': node.uuid, 'error': e})
|
||||||
|
raise exception.RedfishError(error=e)
|
||||||
|
|
||||||
|
|
||||||
|
def _get_connection(node, lambda_fun, *args):
|
||||||
|
"""Get a Redfish connection to a node.
|
||||||
|
|
||||||
|
This method gets a Redfish connection to a node by calling the passed
|
||||||
|
lambda function, and returns the sushy object returned by the function.
|
||||||
|
|
||||||
|
:param node: an Ironic node object
|
||||||
|
:param lambda_fun: the function to call to retrieve the desired sushy
|
||||||
|
object
|
||||||
|
:param args: the arguments to pass to the function
|
||||||
|
:returns: the sushy object returned by the lambda function
|
||||||
|
:raises: RedfishConnectionError when it fails to connect to Redfish
|
||||||
|
"""
|
||||||
|
driver_info = parse_driver_info(node)
|
||||||
|
|
||||||
@retrying.retry(
|
@retrying.retry(
|
||||||
retry_on_exception=(
|
retry_on_exception=(
|
||||||
lambda e: isinstance(e, exception.RedfishConnectionError)),
|
lambda e: isinstance(e, exception.RedfishConnectionError)),
|
||||||
stop_max_attempt_number=CONF.redfish.connection_attempts,
|
stop_max_attempt_number=CONF.redfish.connection_attempts,
|
||||||
wait_fixed=CONF.redfish.connection_retry_interval * 1000)
|
wait_fixed=CONF.redfish.connection_retry_interval * 1000)
|
||||||
def _get_system():
|
def _get_cached_connection(lambda_fun, *args):
|
||||||
try:
|
try:
|
||||||
with SessionCache(driver_info) as conn:
|
with SessionCache(driver_info) as conn:
|
||||||
return conn.get_system(system_id)
|
return lambda_fun(conn, *args)
|
||||||
|
|
||||||
except sushy.exceptions.ResourceNotFoundError as e:
|
|
||||||
LOG.error('The Redfish System "%(system)s" was not found for '
|
|
||||||
'node %(node)s. Error %(error)s',
|
|
||||||
{'system': system_id or '<default>',
|
|
||||||
'node': node.uuid, 'error': e})
|
|
||||||
raise exception.RedfishError(error=e)
|
|
||||||
# TODO(lucasagomes): We should look at other types of
|
# TODO(lucasagomes): We should look at other types of
|
||||||
# ConnectionError such as AuthenticationError or SSLError and stop
|
# ConnectionError such as AuthenticationError or SSLError and stop
|
||||||
# retrying on them
|
# retrying on them
|
||||||
except sushy.exceptions.ConnectionError as e:
|
except sushy.exceptions.ConnectionError as e:
|
||||||
LOG.warning('For node %(node)s, got a connection error from '
|
LOG.warning('For node %(node)s, got a connection error from '
|
||||||
'Redfish at address "%(address)s" using auth type '
|
'Redfish at address "%(address)s" using auth type '
|
||||||
'"%(auth_type)s" when fetching System "%(system)s". '
|
'"%(auth_type)s". Error: %(error)s',
|
||||||
'Error: %(error)s',
|
{'address': driver_info['address'],
|
||||||
{'system': system_id or '<default>',
|
|
||||||
'address': driver_info['address'],
|
|
||||||
'auth_type': driver_info['auth_type'],
|
'auth_type': driver_info['auth_type'],
|
||||||
'node': node.uuid, 'error': e})
|
'node': node.uuid, 'error': e})
|
||||||
raise exception.RedfishConnectionError(node=node.uuid, error=e)
|
raise exception.RedfishConnectionError(node=node.uuid, error=e)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
return _get_system()
|
return _get_cached_connection(lambda_fun, *args)
|
||||||
except exception.RedfishConnectionError as e:
|
except exception.RedfishConnectionError as e:
|
||||||
with excutils.save_and_reraise_exception():
|
with excutils.save_and_reraise_exception():
|
||||||
LOG.error('Failed to connect to Redfish at %(address)s for '
|
LOG.error('Failed to connect to Redfish at %(address)s for '
|
||||||
|
|||||||
@ -13,6 +13,7 @@
|
|||||||
# License for the specific language governing permissions and limitations
|
# License for the specific language governing permissions and limitations
|
||||||
# under the License.
|
# under the License.
|
||||||
|
|
||||||
|
import datetime
|
||||||
from unittest import mock
|
from unittest import mock
|
||||||
|
|
||||||
from oslo_utils import importutils
|
from oslo_utils import importutils
|
||||||
@ -22,7 +23,10 @@ from ironic.common import boot_modes
|
|||||||
from ironic.common import components
|
from ironic.common import components
|
||||||
from ironic.common import exception
|
from ironic.common import exception
|
||||||
from ironic.common import indicator_states
|
from ironic.common import indicator_states
|
||||||
|
from ironic.common import states
|
||||||
from ironic.conductor import task_manager
|
from ironic.conductor import task_manager
|
||||||
|
from ironic.conductor import utils as manager_utils
|
||||||
|
from ironic.drivers.modules import deploy_utils
|
||||||
from ironic.drivers.modules.redfish import management as redfish_mgmt
|
from ironic.drivers.modules.redfish import management as redfish_mgmt
|
||||||
from ironic.drivers.modules.redfish import utils as redfish_utils
|
from ironic.drivers.modules.redfish import utils as redfish_utils
|
||||||
from ironic.tests.unit.db import base as db_base
|
from ironic.tests.unit.db import base as db_base
|
||||||
@ -691,3 +695,563 @@ class RedfishManagementTestCase(db_base.DbTestCase):
|
|||||||
mock_get_system.assert_called_once_with(task.node)
|
mock_get_system.assert_called_once_with(task.node)
|
||||||
|
|
||||||
self.assertEqual(indicator_states.ON, state)
|
self.assertEqual(indicator_states.ON, state)
|
||||||
|
|
||||||
|
@mock.patch.object(manager_utils, 'node_power_action', autospec=True)
|
||||||
|
@mock.patch.object(deploy_utils, 'get_async_step_return_state',
|
||||||
|
autospec=True)
|
||||||
|
@mock.patch.object(deploy_utils, 'set_async_step_flags', autospec=True)
|
||||||
|
@mock.patch.object(redfish_utils, 'get_update_service', autospec=True)
|
||||||
|
def test_update_firmware(self, mock_get_update_service,
|
||||||
|
mock_set_async_step_flags,
|
||||||
|
mock_get_async_step_return_state,
|
||||||
|
mock_node_power_action):
|
||||||
|
mock_task_monitor = mock.Mock()
|
||||||
|
mock_task_monitor.task_monitor = '/task/123'
|
||||||
|
mock_update_service = mock.Mock()
|
||||||
|
mock_update_service.simple_update.return_value = mock_task_monitor
|
||||||
|
mock_get_update_service.return_value = mock_update_service
|
||||||
|
with task_manager.acquire(self.context, self.node.uuid,
|
||||||
|
shared=False) as task:
|
||||||
|
task.node.save = mock.Mock()
|
||||||
|
|
||||||
|
task.driver.management.update_firmware(task,
|
||||||
|
[{'url': 'test1'},
|
||||||
|
{'url': 'test2'}])
|
||||||
|
|
||||||
|
mock_get_update_service.assert_called_once_with(task.node)
|
||||||
|
mock_update_service.simple_update.assert_called_once_with('test1')
|
||||||
|
self.assertIsNotNone(task.node
|
||||||
|
.driver_internal_info['firmware_updates'])
|
||||||
|
self.assertEqual(
|
||||||
|
[{'task_monitor': '/task/123', 'url': 'test1'},
|
||||||
|
{'url': 'test2'}],
|
||||||
|
task.node.driver_internal_info['firmware_updates'])
|
||||||
|
mock_set_async_step_flags.assert_called_once_with(
|
||||||
|
task.node, reboot=True, skip_current_step=True, polling=True)
|
||||||
|
mock_get_async_step_return_state.assert_called_once_with(
|
||||||
|
task.node)
|
||||||
|
mock_node_power_action.assert_called_once_with(task, states.REBOOT)
|
||||||
|
|
||||||
|
@mock.patch.object(task_manager, 'acquire', autospec=True)
|
||||||
|
def test__query_firmware_update_failed(self, mock_acquire):
|
||||||
|
driver_internal_info = {
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
mock_manager = mock.Mock()
|
||||||
|
node_list = [(self.node.uuid, 'redfish', '', driver_internal_info)]
|
||||||
|
mock_manager.iter_nodes.return_value = node_list
|
||||||
|
task = mock.Mock(node=self.node,
|
||||||
|
driver=mock.Mock(management=management))
|
||||||
|
mock_acquire.return_value = mock.MagicMock(
|
||||||
|
__enter__=mock.MagicMock(return_value=task))
|
||||||
|
management._clear_firmware_updates = mock.Mock()
|
||||||
|
|
||||||
|
management._query_firmware_update_failed(mock_manager,
|
||||||
|
self.context)
|
||||||
|
|
||||||
|
management._clear_firmware_updates.assert_called_once_with(self.node)
|
||||||
|
|
||||||
|
@mock.patch.object(task_manager, 'acquire', autospec=True)
|
||||||
|
def test__query_firmware_update_failed_not_redfish(self, mock_acquire):
|
||||||
|
driver_internal_info = {
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
mock_manager = mock.Mock()
|
||||||
|
node_list = [(self.node.uuid, 'not-redfish', '', driver_internal_info)]
|
||||||
|
mock_manager.iter_nodes.return_value = node_list
|
||||||
|
task = mock.Mock(node=self.node,
|
||||||
|
driver=mock.Mock(management=mock.Mock()))
|
||||||
|
mock_acquire.return_value = mock.MagicMock(
|
||||||
|
__enter__=mock.MagicMock(return_value=task))
|
||||||
|
management._clear_firmware_updates = mock.Mock()
|
||||||
|
|
||||||
|
management._query_firmware_update_failed(mock_manager,
|
||||||
|
self.context)
|
||||||
|
|
||||||
|
management._clear_firmware_updates.assert_not_called()
|
||||||
|
|
||||||
|
@mock.patch.object(task_manager, 'acquire', autospec=True)
|
||||||
|
def test__query_firmware_update_failed_no_firmware_upd(self, mock_acquire):
|
||||||
|
driver_internal_info = {'something': 'else'}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
mock_manager = mock.Mock()
|
||||||
|
node_list = [(self.node.uuid, 'redfish', '', driver_internal_info)]
|
||||||
|
mock_manager.iter_nodes.return_value = node_list
|
||||||
|
task = mock.Mock(node=self.node,
|
||||||
|
driver=mock.Mock(management=management))
|
||||||
|
mock_acquire.return_value = mock.MagicMock(
|
||||||
|
__enter__=mock.MagicMock(return_value=task))
|
||||||
|
management._clear_firmware_updates = mock.Mock()
|
||||||
|
|
||||||
|
management._query_firmware_update_failed(mock_manager,
|
||||||
|
self.context)
|
||||||
|
|
||||||
|
management._clear_firmware_updates.assert_not_called()
|
||||||
|
|
||||||
|
@mock.patch.object(redfish_mgmt.LOG, 'info', autospec=True)
|
||||||
|
@mock.patch.object(task_manager, 'acquire', autospec=True)
|
||||||
|
def test__query_firmware_update_failed_node_notfound(self, mock_acquire,
|
||||||
|
mock_log):
|
||||||
|
driver_internal_info = {
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
mock_manager = mock.Mock()
|
||||||
|
node_list = [(self.node.uuid, 'redfish', '', driver_internal_info)]
|
||||||
|
mock_manager.iter_nodes.return_value = node_list
|
||||||
|
mock_acquire.side_effect = exception.NodeNotFound
|
||||||
|
management._clear_firmware_updates = mock.Mock()
|
||||||
|
|
||||||
|
management._query_firmware_update_failed(mock_manager,
|
||||||
|
self.context)
|
||||||
|
|
||||||
|
management._clear_firmware_updates.assert_not_called()
|
||||||
|
self.assertTrue(mock_log.called)
|
||||||
|
|
||||||
|
@mock.patch.object(redfish_mgmt.LOG, 'info', autospec=True)
|
||||||
|
@mock.patch.object(task_manager, 'acquire', autospec=True)
|
||||||
|
def test__query_firmware_update_failed_node_locked(
|
||||||
|
self, mock_acquire, mock_log):
|
||||||
|
driver_internal_info = {
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
mock_manager = mock.Mock()
|
||||||
|
node_list = [(self.node.uuid, 'redfish', '', driver_internal_info)]
|
||||||
|
mock_manager.iter_nodes.return_value = node_list
|
||||||
|
mock_acquire.side_effect = exception.NodeLocked
|
||||||
|
management._clear_firmware_updates = mock.Mock()
|
||||||
|
|
||||||
|
management._query_firmware_update_failed(mock_manager,
|
||||||
|
self.context)
|
||||||
|
|
||||||
|
management._clear_firmware_updates.assert_not_called()
|
||||||
|
self.assertTrue(mock_log.called)
|
||||||
|
|
||||||
|
@mock.patch.object(task_manager, 'acquire', autospec=True)
|
||||||
|
def test__query_firmware_update_status(self, mock_acquire):
|
||||||
|
driver_internal_info = {
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
mock_manager = mock.Mock()
|
||||||
|
node_list = [(self.node.uuid, 'redfish', '', driver_internal_info)]
|
||||||
|
mock_manager.iter_nodes.return_value = node_list
|
||||||
|
task = mock.Mock(node=self.node,
|
||||||
|
driver=mock.Mock(management=management))
|
||||||
|
mock_acquire.return_value = mock.MagicMock(
|
||||||
|
__enter__=mock.MagicMock(return_value=task))
|
||||||
|
management._check_node_firmware_update = mock.Mock()
|
||||||
|
|
||||||
|
management._query_firmware_update_status(mock_manager,
|
||||||
|
self.context)
|
||||||
|
|
||||||
|
management._check_node_firmware_update.assert_called_once_with(task)
|
||||||
|
|
||||||
|
@mock.patch.object(task_manager, 'acquire', autospec=True)
|
||||||
|
def test__query_firmware_update_status_not_redfish(self, mock_acquire):
|
||||||
|
driver_internal_info = {
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
mock_manager = mock.Mock()
|
||||||
|
node_list = [(self.node.uuid, 'not-redfish', '', driver_internal_info)]
|
||||||
|
mock_manager.iter_nodes.return_value = node_list
|
||||||
|
task = mock.Mock(node=self.node,
|
||||||
|
driver=mock.Mock(management=mock.Mock()))
|
||||||
|
mock_acquire.return_value = mock.MagicMock(
|
||||||
|
__enter__=mock.MagicMock(return_value=task))
|
||||||
|
management._check_node_firmware_update = mock.Mock()
|
||||||
|
|
||||||
|
management._query_firmware_update_status(mock_manager,
|
||||||
|
self.context)
|
||||||
|
|
||||||
|
management._check_node_firmware_update.assert_not_called()
|
||||||
|
|
||||||
|
@mock.patch.object(task_manager, 'acquire', autospec=True)
|
||||||
|
def test__query_firmware_update_status_no_firmware_upd(self, mock_acquire):
|
||||||
|
driver_internal_info = {'something': 'else'}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
mock_manager = mock.Mock()
|
||||||
|
node_list = [(self.node.uuid, 'redfish', '', driver_internal_info)]
|
||||||
|
mock_manager.iter_nodes.return_value = node_list
|
||||||
|
task = mock.Mock(node=self.node,
|
||||||
|
driver=mock.Mock(management=management))
|
||||||
|
mock_acquire.return_value = mock.MagicMock(
|
||||||
|
__enter__=mock.MagicMock(return_value=task))
|
||||||
|
management._check_node_firmware_update = mock.Mock()
|
||||||
|
|
||||||
|
management._query_firmware_update_status(mock_manager,
|
||||||
|
self.context)
|
||||||
|
|
||||||
|
management._check_node_firmware_update.assert_not_called()
|
||||||
|
|
||||||
|
@mock.patch.object(redfish_mgmt.LOG, 'info', autospec=True)
|
||||||
|
@mock.patch.object(task_manager, 'acquire', autospec=True)
|
||||||
|
def test__query_firmware_update_status_node_notfound(self, mock_acquire,
|
||||||
|
mock_log):
|
||||||
|
driver_internal_info = {
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
mock_manager = mock.Mock()
|
||||||
|
node_list = [(self.node.uuid, 'redfish', '', driver_internal_info)]
|
||||||
|
mock_manager.iter_nodes.return_value = node_list
|
||||||
|
mock_acquire.side_effect = exception.NodeNotFound
|
||||||
|
management._check_node_firmware_update = mock.Mock()
|
||||||
|
|
||||||
|
management._query_firmware_update_status(mock_manager,
|
||||||
|
self.context)
|
||||||
|
|
||||||
|
management._check_node_firmware_update.assert_not_called()
|
||||||
|
self.assertTrue(mock_log.called)
|
||||||
|
|
||||||
|
@mock.patch.object(redfish_mgmt.LOG, 'info', autospec=True)
|
||||||
|
@mock.patch.object(task_manager, 'acquire', autospec=True)
|
||||||
|
def test__query_firmware_update_status_node_locked(
|
||||||
|
self, mock_acquire, mock_log):
|
||||||
|
driver_internal_info = {
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
mock_manager = mock.Mock()
|
||||||
|
node_list = [(self.node.uuid, 'redfish', '', driver_internal_info)]
|
||||||
|
mock_manager.iter_nodes.return_value = node_list
|
||||||
|
mock_acquire.side_effect = exception.NodeLocked
|
||||||
|
management._check_node_firmware_update = mock.Mock()
|
||||||
|
|
||||||
|
management._query_firmware_update_status(mock_manager,
|
||||||
|
self.context)
|
||||||
|
|
||||||
|
management._check_node_firmware_update.assert_not_called()
|
||||||
|
self.assertTrue(mock_log.called)
|
||||||
|
|
||||||
|
@mock.patch.object(redfish_mgmt.LOG, 'warning', autospec=True)
|
||||||
|
@mock.patch.object(redfish_utils, 'get_update_service', autospec=True)
|
||||||
|
def test__check_node_firmware_update_redfish_conn_error(
|
||||||
|
self, mock_get_update_services, mock_log):
|
||||||
|
mock_get_update_services.side_effect = exception.RedfishConnectionError
|
||||||
|
driver_internal_info = {
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
with task_manager.acquire(self.context, self.node.uuid,
|
||||||
|
shared=False) as task:
|
||||||
|
management._check_node_firmware_update(task)
|
||||||
|
|
||||||
|
self.assertTrue(mock_log.called)
|
||||||
|
|
||||||
|
@mock.patch.object(redfish_mgmt.LOG, 'debug', autospec=True)
|
||||||
|
@mock.patch.object(redfish_utils, 'get_update_service', autospec=True)
|
||||||
|
def test__check_node_firmware_update_wait_elapsed(
|
||||||
|
self, mock_get_update_service, mock_log):
|
||||||
|
mock_update_service = mock.Mock()
|
||||||
|
mock_get_update_service.return_value = mock_update_service
|
||||||
|
|
||||||
|
wait_start_time = datetime.datetime.utcnow() -\
|
||||||
|
datetime.timedelta(minutes=15)
|
||||||
|
driver_internal_info = {
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1',
|
||||||
|
'wait_start_time':
|
||||||
|
wait_start_time.isoformat(),
|
||||||
|
'wait': 1}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
management._continue_firmware_updates = mock.Mock()
|
||||||
|
|
||||||
|
with task_manager.acquire(self.context, self.node.uuid,
|
||||||
|
shared=False) as task:
|
||||||
|
management._check_node_firmware_update(task)
|
||||||
|
|
||||||
|
self.assertTrue(mock_log.called)
|
||||||
|
management._continue_firmware_updates.assert_called_once_with(
|
||||||
|
task,
|
||||||
|
mock_update_service,
|
||||||
|
[{'task_monitor': '/task/123', 'url': 'test1'}])
|
||||||
|
|
||||||
|
@mock.patch.object(redfish_mgmt.LOG, 'debug', autospec=True)
|
||||||
|
@mock.patch.object(redfish_utils, 'get_update_service', autospec=True)
|
||||||
|
def test__check_node_firmware_update_still_waiting(
|
||||||
|
self, mock_get_update_service, mock_log):
|
||||||
|
mock_update_service = mock.Mock()
|
||||||
|
mock_get_update_service.return_value = mock_update_service
|
||||||
|
|
||||||
|
wait_start_time = datetime.datetime.utcnow() -\
|
||||||
|
datetime.timedelta(minutes=1)
|
||||||
|
driver_internal_info = {
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1',
|
||||||
|
'wait_start_time':
|
||||||
|
wait_start_time.isoformat(),
|
||||||
|
'wait': 600}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
management._continue_firmware_updates = mock.Mock()
|
||||||
|
|
||||||
|
with task_manager.acquire(self.context, self.node.uuid,
|
||||||
|
shared=False) as task:
|
||||||
|
management._check_node_firmware_update(task)
|
||||||
|
|
||||||
|
self.assertTrue(mock_log.called)
|
||||||
|
management._continue_firmware_updates.assert_not_called()
|
||||||
|
|
||||||
|
@mock.patch.object(redfish_mgmt.LOG, 'warning', autospec=True)
|
||||||
|
@mock.patch.object(redfish_utils, 'get_update_service', autospec=True)
|
||||||
|
def test__check_node_firmware_update_task_monitor_not_found(
|
||||||
|
self, mock_get_update_service, mock_log):
|
||||||
|
mock_update_service = mock.Mock()
|
||||||
|
mock_update_service.get_task_monitor.side_effect =\
|
||||||
|
sushy.exceptions.ResourceNotFoundError(
|
||||||
|
method='GET', url='/task/123', response=mock.MagicMock())
|
||||||
|
mock_get_update_service.return_value = mock_update_service
|
||||||
|
driver_internal_info = {
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
management._continue_firmware_updates = mock.Mock()
|
||||||
|
|
||||||
|
with task_manager.acquire(self.context, self.node.uuid,
|
||||||
|
shared=False) as task:
|
||||||
|
management._check_node_firmware_update(task)
|
||||||
|
|
||||||
|
self.assertTrue(mock_log.called)
|
||||||
|
management._continue_firmware_updates.assert_called_once_with(
|
||||||
|
task,
|
||||||
|
mock_update_service,
|
||||||
|
[{'task_monitor': '/task/123', 'url': 'test1'}])
|
||||||
|
|
||||||
|
@mock.patch.object(redfish_mgmt.LOG, 'debug', autospec=True)
|
||||||
|
@mock.patch.object(redfish_utils, 'get_update_service', autospec=True)
|
||||||
|
def test__check_node_firmware_update_in_progress(self,
|
||||||
|
mock_get_update_service,
|
||||||
|
mock_log):
|
||||||
|
mock_task_monitor = mock.Mock()
|
||||||
|
mock_task_monitor.is_processing = True
|
||||||
|
mock_update_service = mock.Mock()
|
||||||
|
mock_update_service.get_task_monitor.return_value = mock_task_monitor
|
||||||
|
mock_get_update_service.return_value = mock_update_service
|
||||||
|
driver_internal_info = {
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
|
||||||
|
with task_manager.acquire(self.context, self.node.uuid,
|
||||||
|
shared=False) as task:
|
||||||
|
management._check_node_firmware_update(task)
|
||||||
|
|
||||||
|
self.assertTrue(mock_log.called)
|
||||||
|
|
||||||
|
@mock.patch.object(manager_utils, 'cleaning_error_handler', autospec=True)
|
||||||
|
@mock.patch.object(redfish_mgmt.LOG, 'error', autospec=True)
|
||||||
|
@mock.patch.object(redfish_utils, 'get_update_service', autospec=True)
|
||||||
|
def test__check_node_firmware_update_fail(self,
|
||||||
|
mock_get_update_service,
|
||||||
|
mock_log,
|
||||||
|
mock_cleaning_error_handler):
|
||||||
|
mock_sushy_task = mock.Mock()
|
||||||
|
mock_sushy_task.task_state = 'exception'
|
||||||
|
mock_message_unparsed = mock.Mock()
|
||||||
|
mock_message_unparsed.message = None
|
||||||
|
mock_message = mock.Mock()
|
||||||
|
mock_message.message = 'Firmware upgrade failed'
|
||||||
|
messages = mock.PropertyMock(side_effect=[[mock_message_unparsed],
|
||||||
|
[mock_message]])
|
||||||
|
type(mock_sushy_task).messages = messages
|
||||||
|
mock_task_monitor = mock.Mock()
|
||||||
|
mock_task_monitor.is_processing = False
|
||||||
|
mock_task_monitor.get_task.return_value = mock_sushy_task
|
||||||
|
mock_update_service = mock.Mock()
|
||||||
|
mock_update_service.get_task_monitor.return_value = mock_task_monitor
|
||||||
|
mock_get_update_service.return_value = mock_update_service
|
||||||
|
driver_internal_info = {'something': 'else',
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
management._continue_firmware_updates = mock.Mock()
|
||||||
|
|
||||||
|
with task_manager.acquire(self.context, self.node.uuid,
|
||||||
|
shared=False) as task:
|
||||||
|
task.upgrade_lock = mock.Mock()
|
||||||
|
task.process_event = mock.Mock()
|
||||||
|
|
||||||
|
management._check_node_firmware_update(task)
|
||||||
|
|
||||||
|
task.upgrade_lock.assert_called_once_with()
|
||||||
|
self.assertTrue(mock_log.called)
|
||||||
|
self.assertEqual({'something': 'else'},
|
||||||
|
task.node.driver_internal_info)
|
||||||
|
mock_cleaning_error_handler.assert_called_once()
|
||||||
|
management._continue_firmware_updates.assert_not_called()
|
||||||
|
|
||||||
|
@mock.patch.object(redfish_mgmt.LOG, 'info', autospec=True)
|
||||||
|
@mock.patch.object(redfish_utils, 'get_update_service', autospec=True)
|
||||||
|
def test__check_node_firmware_update_done(self,
|
||||||
|
mock_get_update_service,
|
||||||
|
mock_log):
|
||||||
|
mock_task = mock.Mock()
|
||||||
|
mock_task.task_state = sushy.TASK_STATE_COMPLETED
|
||||||
|
mock_task.task_status = sushy.HEALTH_OK
|
||||||
|
mock_message = mock.Mock()
|
||||||
|
mock_message.message = 'Firmware update done'
|
||||||
|
mock_task.messages = [mock_message]
|
||||||
|
mock_task_monitor = mock.Mock()
|
||||||
|
mock_task_monitor.is_processing = False
|
||||||
|
mock_task_monitor.get_task.return_value = mock_task
|
||||||
|
mock_update_service = mock.Mock()
|
||||||
|
mock_update_service.get_task_monitor.return_value = mock_task_monitor
|
||||||
|
mock_get_update_service.return_value = mock_update_service
|
||||||
|
driver_internal_info = {
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
management._continue_firmware_updates = mock.Mock()
|
||||||
|
|
||||||
|
with task_manager.acquire(self.context, self.node.uuid,
|
||||||
|
shared=False) as task:
|
||||||
|
management._check_node_firmware_update(task)
|
||||||
|
|
||||||
|
self.assertTrue(mock_log.called)
|
||||||
|
management._continue_firmware_updates.assert_called_once_with(
|
||||||
|
task,
|
||||||
|
mock_update_service,
|
||||||
|
[{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1'}])
|
||||||
|
|
||||||
|
@mock.patch.object(redfish_mgmt.LOG, 'debug', autospec=True)
|
||||||
|
def test__continue_firmware_updates_wait(self, mock_log):
|
||||||
|
mock_update_service = mock.Mock()
|
||||||
|
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
with task_manager.acquire(self.context, self.node.uuid,
|
||||||
|
shared=False) as task:
|
||||||
|
management._continue_firmware_updates(
|
||||||
|
task,
|
||||||
|
mock_update_service,
|
||||||
|
[{'task_monitor': '/task/123',
|
||||||
|
'url': 'test1',
|
||||||
|
'wait': 10,
|
||||||
|
'wait_start_time': '20200901123045'},
|
||||||
|
{'url': 'test2'}])
|
||||||
|
|
||||||
|
self.assertTrue(mock_log.called)
|
||||||
|
# Wait start time has changed
|
||||||
|
self.assertNotEqual(
|
||||||
|
'20200901123045',
|
||||||
|
task.node.driver_internal_info['firmware_updates']
|
||||||
|
[0]['wait_start_time'])
|
||||||
|
|
||||||
|
@mock.patch.object(redfish_mgmt.LOG, 'info', autospec=True)
|
||||||
|
@mock.patch.object(manager_utils, 'notify_conductor_resume_clean',
|
||||||
|
autospec=True)
|
||||||
|
def test__continue_firmware_updates_last_update(
|
||||||
|
self,
|
||||||
|
mock_notify_conductor_resume_clean,
|
||||||
|
mock_log):
|
||||||
|
mock_update_service = mock.Mock()
|
||||||
|
driver_internal_info = {
|
||||||
|
'something': 'else',
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123', 'url': 'test1'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
self.node.save()
|
||||||
|
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
with task_manager.acquire(self.context, self.node.uuid,
|
||||||
|
shared=False) as task:
|
||||||
|
management._continue_firmware_updates(
|
||||||
|
task,
|
||||||
|
mock_update_service,
|
||||||
|
[{'task_monitor': '/task/123', 'url': 'test1'}])
|
||||||
|
|
||||||
|
self.assertTrue(mock_log.called)
|
||||||
|
mock_notify_conductor_resume_clean.assert_called_once_with(task)
|
||||||
|
self.assertEqual({'something': 'else'},
|
||||||
|
task.node.driver_internal_info)
|
||||||
|
|
||||||
|
@mock.patch.object(redfish_mgmt.LOG, 'debug', autospec=True)
|
||||||
|
@mock.patch.object(manager_utils, 'node_power_action', autospec=True)
|
||||||
|
def test__continue_firmware_updates_more_updates(self,
|
||||||
|
mock_node_power_action,
|
||||||
|
mock_log):
|
||||||
|
mock_task_monitor = mock.Mock()
|
||||||
|
mock_task_monitor.task_monitor = '/task/987'
|
||||||
|
mock_update_service = mock.Mock()
|
||||||
|
mock_update_service.simple_update.return_value = mock_task_monitor
|
||||||
|
driver_internal_info = {
|
||||||
|
'something': 'else',
|
||||||
|
'firmware_updates': [
|
||||||
|
{'task_monitor': '/task/123', 'url': 'test1'},
|
||||||
|
{'url': 'test2'}]}
|
||||||
|
self.node.driver_internal_info = driver_internal_info
|
||||||
|
|
||||||
|
management = redfish_mgmt.RedfishManagement()
|
||||||
|
with task_manager.acquire(self.context, self.node.uuid,
|
||||||
|
shared=False) as task:
|
||||||
|
task.node.save = mock.Mock()
|
||||||
|
|
||||||
|
management._continue_firmware_updates(
|
||||||
|
task,
|
||||||
|
mock_update_service,
|
||||||
|
[{'task_monitor': '/task/123', 'url': 'test1'},
|
||||||
|
{'url': 'test2'}])
|
||||||
|
|
||||||
|
self.assertTrue(mock_log.called)
|
||||||
|
mock_update_service.simple_update.assert_called_once_with('test2')
|
||||||
|
self.assertIsNotNone(
|
||||||
|
task.node.driver_internal_info['firmware_updates'])
|
||||||
|
self.assertEqual(
|
||||||
|
[{'url': 'test2', 'task_monitor': '/task/987'}],
|
||||||
|
task.node.driver_internal_info['firmware_updates'])
|
||||||
|
task.node.save.assert_called_once_with()
|
||||||
|
mock_node_power_action.assert_called_once_with(task, states.REBOOT)
|
||||||
|
|||||||
@ -339,3 +339,20 @@ class RedfishUtilsTestCase(db_base.DbTestCase):
|
|||||||
mock.ANY, verify=mock.ANY,
|
mock.ANY, verify=mock.ANY,
|
||||||
auth=mock_basic_auth.return_value
|
auth=mock_basic_auth.return_value
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def test_get_update_service(self):
|
||||||
|
redfish_utils._get_connection = mock.Mock()
|
||||||
|
mock_update_service = mock.Mock()
|
||||||
|
redfish_utils._get_connection.return_value = mock_update_service
|
||||||
|
|
||||||
|
result = redfish_utils.get_update_service(self.node)
|
||||||
|
|
||||||
|
self.assertEqual(mock_update_service, result)
|
||||||
|
|
||||||
|
def test_get_update_service_error(self):
|
||||||
|
redfish_utils._get_connection = mock.Mock()
|
||||||
|
redfish_utils._get_connection.side_effect =\
|
||||||
|
sushy.exceptions.MissingAttributeError
|
||||||
|
|
||||||
|
self.assertRaises(exception.RedfishError,
|
||||||
|
redfish_utils.get_update_service, self.node)
|
||||||
|
|||||||
@ -156,6 +156,9 @@ SUSHY_SPEC = (
|
|||||||
'VIRTUAL_MEDIA_CD',
|
'VIRTUAL_MEDIA_CD',
|
||||||
'VIRTUAL_MEDIA_FLOPPY',
|
'VIRTUAL_MEDIA_FLOPPY',
|
||||||
'APPLY_TIME_ON_RESET',
|
'APPLY_TIME_ON_RESET',
|
||||||
|
'TASK_STATE_COMPLETED',
|
||||||
|
'HEALTH_OK',
|
||||||
|
'HEALTH_WARNING'
|
||||||
)
|
)
|
||||||
|
|
||||||
SUSHY_AUTH_SPEC = (
|
SUSHY_AUTH_SPEC = (
|
||||||
|
|||||||
@ -218,6 +218,9 @@ if not sushy:
|
|||||||
VIRTUAL_MEDIA_CD='cd',
|
VIRTUAL_MEDIA_CD='cd',
|
||||||
VIRTUAL_MEDIA_FLOPPY='floppy',
|
VIRTUAL_MEDIA_FLOPPY='floppy',
|
||||||
APPLY_TIME_ON_RESET='on reset',
|
APPLY_TIME_ON_RESET='on reset',
|
||||||
|
TASK_STATE_COMPLETED='completed',
|
||||||
|
HEALTH_OK='ok',
|
||||||
|
HEALTH_WARNING='warning'
|
||||||
)
|
)
|
||||||
|
|
||||||
sys.modules['sushy'] = sushy
|
sys.modules['sushy'] = sushy
|
||||||
|
|||||||
@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
features:
|
||||||
|
- |
|
||||||
|
Adds support for performing firmware updates using the ``redfish``
|
||||||
|
and ``idrac`` hardware types.
|
||||||
|
|
||||||
|
A new firmware update cleaning step has been added to the ``redfish``
|
||||||
|
hardware type. The ``idrac`` hardware type also automatically gains this
|
||||||
|
capability through inheritance.
|
||||||
Loading…
Reference in New Issue
Block a user