airship/drydock
Code Issues Proposed changes
drydock/python/drydock_provisioner/orchestrator
History
Mahmoudi, Ahmad (am495p) 5063769796 Added some validations loggs for site model 
During drydock node deployments, sometimes MaaS node deployment for
some nodes fails when the node tries to pull the node bootaction
files, using drydock api.
Drydock api call fails with `500 Internal Server Error`, when
drydock tries to create the booaction files for the node. The logs,
however do no provide any additional clues on what caused drydock to
fail. This issue does not happen always, and subsequent site updates
will most of the deploy the failed nodes.

The additional checks and logs are added to help pinpoint the root cause
of the 500 return code, if/when this issue heppens again.

This ps also, uplifted `MarkupSafe` pip library from 1.0 to 1.1.1 to
address the issue with MarkupSafe and latest version of setuptools
described here: https://github.com/pallets/markupsafe/issues/116

Change-Id: I08a088d9690d8d9dd1f771dc5e84d1eb02fbd39f
2020-07-24 18:57:21 +00:00
..
actions
omit the twice occured words in orchestrator.py
2018-11-14 17:57:43 +08:00
validations
Address PEP8 Failures in Drydock
2020-05-12 18:27:06 +00:00
__init__.py
Refactor build to include Go
2018-09-04 13:13:21 -05:00
orchestrator.py
Added some validations loggs for site model
2020-07-24 18:57:21 +00:00
readme.md
Remove the repetition words in readme.md
2019-01-04 20:17:03 +08:00
util.py
Fix: pep8 airship-drydock-omni-test fix
2018-11-05 18:35:36 +01:00

readme.md

Orchestrator

The orchestrator is the core of drydock and will manage the ordering of driver actions to implement the main Drydock actions. Each of these actions will be started by the external cLCP orchestrator with different parameters to control the scope of the action.

Orchestrator should persist the state of each task such that on failure the task can retried and only the steps needed will be executed.

Drydock Tasks

Bullet points listed below are not exhaustive and will change as we move through testing

ValidateDesign

Load design data from the statemgmt persistent store and validate that the current state of design data represents a valid site design. No claim is made that the design data is compatible with the physical state of the site.

Validations

  • Networking
    • No static IP assignments are duplicated
    • No static IP assignments are outside of the network they are targetted for
    • All IP assignments are within declared ranges on the network
    • No network is allowed on multiple network links
    • Network MTU is equal or less than NetworkLink MTU
    • MTU values are sane
    • NetworkLink bond mode is compatible with other bond options
    • NetworkLink with more than one allowed network supports trunking
  • Storage
    • Boot drive is above minimum size
    • Root drive is above minimum size
    • No physical device specifies a target VG and a partition list
    • No partition specifies a target VG and a filesystem
    • All defined VGs have at least one defined PV (partition or physical device)
    • Partition and LV sizing is sane
      • Percentages don't sum to above 100%
      • If percentages sum to 100%, no other partitions or LVs are defined
  • Node
    • Root filesystem is defined on a partition or LV
    • Networks assigned to each node's interface are within the set of the attached link's allowed_networks
    • Inter

VerifySite

Verify site-wide resources are in a useful state

  • Driver downstream resources are reachable (e.g. MaaS)
  • OS images needed for bootstrapping are available
  • Promenade or other next-step services are up and available
  • Verify credentials are available

PrepareSite

Begin preparing site-wide resources for bootstrapping. This action will lock site design data for changes.

  • Configure bootstrapper with site network configs
  • Shuffle images so they are correctly configured for bootstrapping

VerifyNode

Verification of per-node configurations within the context of the current node status

  • Status: Present
    • Basic hardware verification as available via OOB driver
      • BIOS firmware
      • PCI layout
      • Drives
      • Hardware alarms
    • IPMI connectivity
  • Status: Prepared
    • Full hardware manifest
    • Possibly network connectivity
    • Firmware versions

PrepareNode

Prepare a node for bootstrapping

  • Configure network port for PXE
  • Configure a node for PXE boot
  • Power-cycle the node
  • Setup commissioning configuration
    • Hardware drivers
    • Hardware configuration (e.g. RAID)
  • Configure node networking
  • Configure node storage
  • Interrogate node
    • lshw output
    • lldp output

DeployNode

Begin bootstrapping the node and monitor success

  • Initialize the Introspection service for the node
  • Bootstrap the node (i.e. Write persistent OS install)
  • Ensure network port is returned to production configuration
  • Reboot node from local disk
  • Monitor platform bootstrapping

DestroyNode

Destroy current node configuration and rebootstrap from scratch

RelabelNode

Relabel current Kubernetes cluster node through Kubernetes provisioner.

Integration with Drivers

Based on the requested task and the current known state of a node the orchestrator will call the enabled downstream drivers with one or more tasks. Each call will provide the driver with the desired state (the applied model) and current known state (the build model).