docs/doc/source/system_configuration/kubernetes/check-new-registry-and-reapply-application.rst

Check New Registry and Reapply Application

Perform these steps on both controllers. Use controller-0 first and then swact to controller-1 and perform the same steps.

1. Run this command to login:

   $ sudo docker login new-registry.domain.com:9001

2. Run this command to do a test pull of the image:

   $ sudo docker image pull new-registry.domain.com:9001/product-abc/starlingx/docker.io/alpine:latest
   $ crictl pull --creds docker:****** new-registry.domain.com:9001/product-abc/starlingx/docker.io/alpine:latest
   
   $ sudo docker image rm new-registry.domain.com:9001/product-abc/starlingx/docker.io/alpine:latest
   $ crictl rmi new-registry.domain.com:9001/product-abc/starlingx/docker.io/alpine:latest

3. Check if an application re-apply will now properly pull from the registries. First, remove the images for an application, such as nginx-ingress-controller, from the registry.local and the local image cache for all nodes (assuming AIO-SX), in order to force the next re-apply of the application to re-pull these images.

   ~(keystone_admin)]$ system registry-image-tags quay.io/kubernetes-ingress-controller/nginx-ingress-controller
   +-----------+
   | Image Tag |
   +-----------+
   | 0.23.0    |
   +-----------+
   
   system registry-image-delete quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
   system registry-garbage-collect
   crictl images ls | grep  quay.io/kubernetes-ingress-controller/nginx-ingress-controller
   registry.local:9001/quay.io/kubernetes-ingress-controller/nginx-ingress-controller   0.23.0                                                   42d47fe0c78f5       242MB
   
   crictl rmi registry.local:9001/quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
   # Note an error on this step means there is no image in the cache
   
   # SSH to controller-1 (or the standby controller)
   crictl rmi registry.local:9001/quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0

4. To reapply the application run the following command:

   ~(keystone_admin)]$ system application-apply nginx-ingress-controller

5. Then, debug tail -f /var/log/sysinv.log and look for the following information:

   sysinv 2020-09-09 23:42:23.476 14930 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0 is not available in local registry, download started from public/private registry sysinv 2020-09-09 23:42:23.526 14930 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/k8s.gcr.io/defaultbackend: download succeeded in 0 seconds sysinv 2020-09-09 23:43:10.226 14930 INFO sysinv.conductor.kube_app [-] Remove image <hostname>:5001/<quay.io path>/quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0 after push to local registry. sysinv 2020-09-09 23:43:10.595 14930 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0 download succeeded in 47 seconds sysinv 2020-09-09 23:43:10.596 14930 INFO sysinv.conductor.kube_app [-] All docker images for application nginx-ingress-controller were successfully downloaded in 50 seconds

6. Validate that the application is running:

   ~(keystone_admin)]$ system application-list
   +--------------------------+---------+-------------------------------------------+------------------+----------+----------------------+
   | application              | version | manifest name                             | manifest file    | status   | progress             |
   +--------------------------+---------+-------------------------------------------+------------------+----------+----------------------+
   | cert-manager             | 1.0-34  | cert-manager-fluxcd-manifests             | fluxcd-manifests | applied  | completed            |
   | nginx-ingress-controller | 1.1-35  | nginx-ingress-controller-fluxcd-manifests | fluxcd-manifests | applied  | completed            |
   | oidc-auth-apps           | 1.0-68  | oidc-auth-apps-fluxcd-manifests           | fluxcd-manifests | uploaded | completed            |
   | platform-integ-apps      | 1.0-52  | platform-integ-apps-fluxcd-manifests      | fluxcd-manifests | applied  | completed            |
   | rook-ceph-apps           | 1.0-17  | rook-ceph-manifest                        | manifest.yaml    | uploaded | completed            |
   +--------------------------+---------+-------------------------------------------+------------------+----------+----------------------+

7. Validate that the image is in the local registry:

   ~(keystone_admin)]$ system registry-image-tags quay.io/kubernetes-ingress-controller/nginx-ingress-controller
    +-----------+
    | Image Tag |
    +-----------+
    | 0.23.0    |
    +-----------+