starlingx/docs
Code Issues Proposed changes
docs/doc/source/updates/kubernetes/handling-kubernetes-update-orchestration-failures.rst
Ron Stone 2b62f49a9d Fix symlinks 
Changed paths to avoid '..', which breaks symlinks in newer versions of sphinx.
Consolidated installation include files under /_includes. Prefixed r5 versions with 'r5_'
Moved files that are used up/down, but at different paths under /shared/_includes
and /shared/figures
Move two include files to /_includes
Moved addtional images to /shared/figures/... Required for DS platform builds.

Signed-off-by: Ron Stone <ronald.stone@windriver.com>
Change-Id: Ia38f4205c5803b3d1fc043e6c59617c34a4e5cbd
Signed-off-by: Ron Stone <ronald.stone@windriver.com>
2021-09-02 13:31:45 +00:00

3.7 KiB
Raw Blame History

Handle Kubernetes Version Upgrade Orchestration Failures

The creation or application of a strategy could fail for any of the listed reasons described in this section. Follow the suggested actions in each case to resolve the issue.

Strategy creation failure

  • Reason: build failed with no reason.
    • Action:
      • Verify that the --worker-apply-type was not set to ignore.
      • Check recent logs added to /var/log/nfv-vim.log.
  • Reason: alarms from platform are present.
    • Action:

      • Query for management affecting alarms and take actions to clear them.

        ~(keystone_admin)$ fm alarm-list --mgmt_affecting

      • If there are no management affecting alarms present, take actions to clear other reported alarms or try creating the strategy with the relaxed alarms restrictions option --alarm-restrictions relaxed.

  • Reason: no Kubernetes version upgrade required.
    • Action:

      • Verify that the Kubernetes patches have been uploaded and applied. Verify the version of Kubernetes on the hosts by executing "system kube-host-upgrade-list.

        Note

        If the strategy create failed, first you must resolve it. You must delete the failed strategy before you create another strategy.

Strategy Apply Failure

  • Reason: alarms from platform are present.

    • Action: suggests that an alarm has been raised since the creation of the strategy. Address the cause of the new alarm, delete the strategy and try creating and applying a new strategy.

  • Reason: unable to migrate instances.

    • Action: See Kubernetes Version Upgrade Operations Requiring Manual Migration <kubernetes-update-operations-requiring-manual-migration> for steps to resolve migration issues.

  • Reason: Kubernetes version upgrade failed. Suggests that the Kubernetes upgrade for the specified host has failed.

    starlingx

    partner

  • Reason: lock host failed.

    • Action:
      • Investigate the /var/log/sysinv.log, and /var/log/nfv-vim.log files.
      • Address the underlying issue.
      • Manually lock and unlock the host.
      • Try recreating and re-applying the Kubernetes version upgrade strategy to automatically finish the upgrade process.

  • Reason: unlock host failed.

    • Action:
      • Investigate /var/log/mtcAgent.log file for cause logs files.
      • Address the underlying issue.
      • Manually lock and unlock the host to recover.
      • Try recreating and re-applying the Kubernetes version upgrade strategy to automatically finish the upgrade process.

Note

If the strategy apply fails, you must resolve the strategyapply failure, and delete the failed strategy before trying to create and apply another strategy.