9169085db7
Our ramdisks have swelled, and are taking anywhere from 500-700
seconds to even reach the point where IPA is starting up.
This means, that a 900 second build timeout is cutting it close
and intermittent performance degredation in CI means that a job
may fail simply because it is colliding with the timeout.
One example I deconstruted today where a 900 second timout was
in effect:
* 08:21:41 Tempest job startes
* 08:21:46 Nova instance requested
* Compute service requests ironic to do the thing.
* Ironic downloads IPA and stages it - ~20-30 seconds
* VM boots and loads ipxe ~30 seconds.
* 08:23:22 - ipxe downloads kernel/ramdisk (time should be completion
unless apache has changed logging behavior for requests.)
* 08:26:28 - Kernel at 120 second marker and done decompressing
the ramdisk.
* ~08:34:30 - Kernel itself hit the six hundred second runtime
marker and hasn't even started IPA.
* 08:35:02 - Ironic declars the deploy failed due to wait timeout.
([conductor]deploy_callback_timeout hit at 700 seconds.)
* 08:35:32 - Nova fails the build saying it can't be scheduled.
(Note, I started adding times to figure out the window to myself, so
they are incomplete above.)
The time we can account for in the job is about 14 minutes or 840
seconds. As such, our existing defaults are just not enough to handle
the ramdisk size AND variance in cloud performance.
Change-Id: I4f9db300e792980059c401fce4c37a68c438d7c0
Ironic
Team and repository tags
Overview
Ironic consists of an API and plug-ins for managing and provisioning physical machines in a security-aware and fault-tolerant manner. It can be used with nova as a hypervisor driver, or standalone service using bifrost. By default, it will use PXE and IPMI to interact with bare metal machines. Ironic also supports vendor-specific plug-ins which may implement additional functionality.
Ironic is distributed under the terms of the Apache License, Version 2.0. The full terms and conditions of this license are detailed in the LICENSE file.
Project resources
- Documentation: https://docs.openstack.org/ironic/latest
- Source: https://opendev.org/openstack/ironic
- Bugs: https://storyboard.openstack.org/#!/project/943
- Wiki: https://wiki.openstack.org/wiki/Ironic
- APIs: https://docs.openstack.org/api-ref/baremetal/index.html
- Release Notes: https://docs.openstack.org/releasenotes/ironic/
- Design Specifications: https://specs.openstack.org/openstack/ironic-specs/
Project status, bugs, and requests for feature enhancements (RFEs) are tracked in StoryBoard: https://storyboard.openstack.org/#!/project/943
For information on how to contribute to ironic, see https://docs.openstack.org/ironic/latest/contributor