Restrict the SRIOV device plugin mount path

The kube-sriov-device-plugin pod will mount HostPath:'/var/lib/kubelet'
If a pod mounting with rbd PVC is already running when
kube-sriov-device-plugin pod starts up, the kube-sriov-device-plugin
pod will refer to the rbd mountpoint which is under HostPath:
'/var/lib/kubelet'.
Even if the rbd is unmounted from the mountpoint on the host, the pod
will keep referring to it in its namespace.
So kubelet can't unmap the rbd and will fail to mount the volume when
pod with rbd PVC is recreated.
The kube-sriov-device-plugin doesn't need to use '/var/lib/kubelet' as
mountpath because its internal device socket is actually at
/var/lib/kubelet/device-plugins/.
Changing the kube-sriov-device-plugin mountpath to a less broad path
will preserve the rbd PVC mount point under /var/lib/kubelet/.


Test plan
PASS Installed AIO-SX
     create SRIOV interfaces
     create stateful pod with rbd PVC
     delete kube-sriov-device-plugin
     delete stateful pod
     A new stateful pod will automatically be created
     check if stateful pod was not stuck
     confirm if stateful pod could mount the volume
PASS Create a SRIOV NetworkAttachmentDefinition
     Launch a POD using the SRIOV interface
     check if POD is running and if POD has connectivity.
PASS Upgrades testing (partial) - verified controller-1 is upgraded and device plugin/pod working with new location.

Closes-Bug: #2007596

Signed-off-by: Fabiano Mercer <fabiano.correamercer@windriver.com>
Change-Id: I7ef43a1c0ac4f7f0af1a366c298b4c1029d3e915

playbookconfig/src/playbooks/roles/common/bringup-kubemaster/templates/k8s-v1.21.8/sriov-plugin.yaml.j2

@@ -83,7 +83,7 @@ spec:
             memory: "200Mi"
         volumeMounts:
         - name: devicesock
-          mountPath: /var/lib/kubelet/
+          mountPath: /var/lib/kubelet/device-plugins/
           readOnly: false
         - name: log
           mountPath: /var/log
@@ -95,7 +95,7 @@ spec:
       volumes:
         - name: devicesock
           hostPath:
-            path: /var/lib/kubelet/
+            path: /var/lib/kubelet/device-plugins/
         - name: log
           hostPath:
             path: /var/log

playbookconfig/src/playbooks/roles/common/bringup-kubemaster/templates/k8s-v1.22.5/sriov-plugin.yaml.j2

@@ -83,7 +83,7 @@ spec:
             memory: "200Mi"
         volumeMounts:
         - name: devicesock
-          mountPath: /var/lib/kubelet/
+          mountPath: /var/lib/kubelet/device-plugins/
           readOnly: false
         - name: log
           mountPath: /var/log
@@ -95,7 +95,7 @@ spec:
       volumes:
         - name: devicesock
           hostPath:
-            path: /var/lib/kubelet/
+            path: /var/lib/kubelet/device-plugins/
         - name: log
           hostPath:
             path: /var/log

playbookconfig/src/playbooks/roles/common/bringup-kubemaster/templates/k8s-v1.24.4/sriov-plugin.yaml.j2

@@ -85,7 +85,7 @@ spec:
             memory: "200Mi"
         volumeMounts:
         - name: devicesock
-          mountPath: /var/lib/kubelet/
+          mountPath: /var/lib/kubelet/device-plugins/
           readOnly: false
         - name: log
           mountPath: /var/log
@@ -97,7 +97,7 @@ spec:
       volumes:
         - name: devicesock
           hostPath:
-            path: /var/lib/kubelet/
+            path: /var/lib/kubelet/device-plugins/
         - name: log
           hostPath:
             path: /var/log