starlingx/docs
Code Issues Proposed changes

Update documentation for Kubevirt

Browse Source 
Add Usage Examples sections.
Create KubeVirt Architecture section.
Fix minor editorial issues.
Fix grammar and formatting issues.

Story: 2010931
Task: 50286

Change-Id: I6118d0af848d07f3764eeae5ea8467864c65fceb
Signed-off-by: Elisamara Aoki Goncalves <elisamaraaoki.goncalves@windriver.com>
This commit is contained in:
Elisamara Aoki Goncalves 2024-06-06 15:39:42 +00:00
parent 8dd6c00569
commit 1f91cd1ee0
24 changed files with 2520 additions and 53 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
doc/source/_vendor/rl-strings.txt
View File

@ -445,7 +445,7 @@
.. |applying-a-custom-branding-tarball-to-running-systems| replace:: :ref:`Apply a Custom Horizon Branding Tarball to Running Systems <applying-a-custom-branding-tarball-to-running-systems>`
.. |release-notes| replace:: :ref:`R10.0 Release Notes <release-notes>`
.. |create-an-ubuntu-vm-fafb82ec424b| replace:: :ref:`Create an Ubuntu VM <create-an-ubuntu-vm-fafb82ec424b>`
.. |removal-97cc897941bc| replace:: :ref:`Removal <removal-97cc897941bc>`
.. |kubevirt-removal-97cc897941bc| replace:: :ref:`Removal <kubevirt-removal-97cc897941bc>`
.. |hello-world-kubevirt-vm-05503659173c| replace:: :ref:`Hello World KubeVirt VM <hello-world-kubevirt-vm-05503659173c>`
.. |index-kubevirt-f1bfd2a21152| replace:: :ref:`KubeVirt <index-kubevirt-f1bfd2a21152>`
.. |set-up-cdi-proxy-ad165d884417| replace:: :ref:`Set up CDI Proxy <set-up-cdi-proxy-ad165d884417>`

102
doc/source/kube-virt/configuration-as-filesystem-0a5023c8386e.rst Normal file
View File

@ -0,0 +1,102 @@
.. WARNING: Add no lines of text between the label immediately following
.. and the title.
.. _configuration-as-filesystem-0a5023c8386e:
===========================
Configuration as Filesystem
===========================
By using filesystem, ``configMaps`` are shared through ``virtiofs``. In
contrast with using disks for sharing ``configMaps``, filesystem allows you to
dynamically propagate changes on ``configMaps`` to |VMIs| (i.e. the VM does not
need to be rebooted).
.. rubric:: Limitation
Currently, |VMIs| cannot be live migrated since ``virtiofs`` does not support
live migration.
Example of ``configMap``:
.. code-block:: none
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
data:
DATABASE: staging
USERNAME: admin
PASSWORD: secret
Example of VM using ``configMap`` as a filesystem:
.. code-block:: none
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
labels:
special: vmi-fedora
name: vmi-fedora
spec:
domain:
devices:
filesystems:
- name: config-fs
virtiofs: {}
disks:
- disk:
bus: virtio
name: containerdisk
machine:
type: ""
resources:
requests:
memory: 1024M
terminationGracePeriodSeconds: 0
volumes:
- name: containerdisk
containerDisk:
image: quay.io/containerdisks/fedora:latest
- cloudInitNoCloud:
userData: |-
#cloud-config
chpasswd:
expire: false
password: fedora
user: fedora
bootcmd:
# mount the ConfigMap
- "sudo mkdir /mnt/app-config"
- "sudo mount -t virtiofs config-fs /mnt/app-config"
name: cloudinitdisk
- configMap:
name: app-config
name: config-fs
To login and verify the VM run:
.. code-block:: none
[fedora@vmi-fedora ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda5 4.0G 461M 3.1G 13% /
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 450M 0 450M 0% /dev/shm
tmpfs 180M 720K 179M 1% /run
tmpfs 450M 0 450M 0% /tmp
/dev/vda2 966M 61M 840M 7% /boot
/dev/vda3 100M 12M 89M 12% /boot/efi
/dev/vda5 4.0G 461M 3.1G 13% /home
config-fs 9.8G 910M 8.4G 10% /mnt/app-config
tmpfs 90M 4.0K 90M 1% /run/user/1000
[fedora@vmi-fedora ~]$ ls -lrth /mnt/app-config
total 0
lrwxrwxrwx. 1 root 107 15 Jan 15 16:20 USERNAME -> ..data/USERNAME
lrwxrwxrwx. 1 root 107 15 Jan 15 16:20 PASSWORD -> ..data/PASSWORD
lrwxrwxrwx. 1 root 107 15 Jan 15 16:20 DATABASE -> ..data/DATABASE

BIN
doc/source/kube-virt/figures/kubevirt-architecture.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 35 KiB

BIN
doc/source/kube-virt/figures/kubrvirt-simplified-diagram.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 31 KiB

85
doc/source/kube-virt/host-device-assignment-a6feb2f0c3bc.rst Normal file
View File

@ -0,0 +1,85 @@
.. WARNING: Add no lines of text between the label immediately following
.. and the title.
.. _host-device-assignment-a6feb2f0c3bc:
======================
Host Device Assignment
======================
KubeVirt provides a mechanism for assigning host devices to a virtual machine.
This mechanism is generic and allows various types of |PCI| devices, such as
accelerators (including GPUs) or any other devices attached to a |PCI| bus,
to be assigned. It also allows Linux Mediated devices, such as pre-configured
virtual GPUs to be assigned using the same mechanism.
Preparation of Mediated Devices such as vGPU
--------------------------------------------
In general, configuration of Mediated devices (mdevs), such as |vGPUs|, should
be done according to vendor directions. KubeVirt can now facilitate the
creation of the Mediated devices/|vGPUs| on the cluster nodes. This assumes
that the required vendor driver is already installed on the nodes.
List Permitted Devices
----------------------
Administrators can control which host devices are exposed and permitted to be
used in the cluster. Permitted host devices in the cluster will need to be
listed in KubeVirt Custom Resource (CR) by its vendor:product selector for
|PCI| devices or Mediated device names. See the example yaml segment below,
that would be added to the KubeVirt CR with ``kubectl edit kubevirt -n
kubevirt``.
.. code-block:: none
configuration:
permittedHostDevices:
pciHostDevices:
- pciVendorSelector: "10DE:1EB8"
resourceName: "nvidia.com/TU104GL_Tesla_T4"
externalResourceProvider: true
- pciVendorSelector: "8086:6F54"
resourceName: "intel.com/qat"
mediatedDevices:
- mdevNameSelector: "GRID T4-1Q"
resourceName: "nvidia.com/GRID_T4-1Q"
.. note::
``pciVendorSelector`` is a |PCI| vendor ID and product ID tuple in the form
``vendor_id:product_id``. This tuple can identify specific types of devices
on a host. For example, the identifier ``10de:1eb8``, shown above, can be
found using ``lspci``.
.. code-block:: none
$ lspci -nnv|grep -i nvidia
65:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
Start VM Using vGPU
-------------------
Host devices can be assigned to VMs via the ``gpus`` and ``hostDevices``
fields. The ``deviceNames`` can reference both |PCI| and Mediated device
resource names.
.. code-block:: none
kind: VirtualMachineInstance
spec:
domain:
devices:
gpus:
- deviceName: nvidia.com/TU104GL_Tesla_T4
name: gpu1
- deviceName: nvidia.com/GRID_T4-1Q
name: gpu2
hostDevices:
- deviceName: intel.com/qat
NAME: QUICKACCESS1

17
doc/source/kube-virt/index-kubevirt-f1bfd2a21152.rst
View File

@ -10,8 +10,9 @@ KubeVirt
:maxdepth: 1
introduction-bb3a04279bf5
kubevirt-architecture-c92c8908e775
installation-66477d7646db
removal-97cc897941bc
kubevirt-removal-97cc897941bc
Usage Examples
==============
@ -21,6 +22,20 @@ Usage Examples
hello-world-kubevirt-vm-05503659173c
set-up-cdi-proxy-ad165d884417
virtual-machine-dd561f6db3fd
use-shared-resource-cpu-ad295227aa0c
live-migration-support-for-vms-bea635bacc50
interface-and-networks-7cadb7bdb80b
startup-scripts-6402154a6f37
persistent-storage-for-vms-8ddc8fa611aa
host-device-assignment-a6feb2f0c3bc
vm-snapshot-and-restore-21158b60cd56
virtualmachineinstancereplicaset-8518d55de52b
static-ip-assignment-via-cloudinit-configuration-d053375e78fa
configuration-as-filesystem-0a5023c8386e
vm-using-secret-as-startup-configuration-4a8255e26b1f
vm-using-service-account-as-filesystem-5fd4deb7339a
node-assignment-using-nodeselector-affinity-and-antiminusaffi-06ad222ceb13
create-an-ubuntu-vm-fafb82ec424b
set-up-remote-management-of-vms-a082461d660e
create-a-windows-vm-82957181df02

178
doc/source/kube-virt/installation-66477d7646db.rst
View File

@ -1,69 +1,161 @@
.. _installation-66477d7646db:
============
Installation
============
=====================
KubeVirt Installation
=====================
.. rubric:: |prereq|
- |prod| is up and running.
- System is Alarm free.
.. code-block:: none
$ fm alarm-list
.. rubric:: |proc|
Complete the following steps to install KubeVirt.
#. Upload the KubeVirt system application tarball and check the KubeVirt
application status:
.. only:: starlingx
.. code-block:: none
~(keystone_admin)$ system application-upload /usr/local/share/applications/helm/kubevirt-app-<version>.tgz
~(keystone_admin)$ system application-list
.. only:: partner
.. include:: /_includes/installation-66477d7646db.rest
:start-after: kubevirt-app-version
:end-before: kubevirt-app-version
#. Apply the KubeVirt system application and check the KubeVirt and |CDI|
status:
#. Upload and apply the KubeVirt package (Helm).
.. code-block:: none
~(keystone_admin)$ system application-apply kubevirt-app
# system application-upload /usr/local/share/applications/helm/kubevirt-app-1.0-17.tgz
# system application-apply kubevirt-app
# system application-show kubevirt-app
# system application-list
Wait for kubevirt-app status to complete.
#. Check all the objects are up and running.
.. code-block:: bash
.. code-block:: none
$ watch -n 5 system application-list
# kubectl get all -n cdi
# kubectl get all -n kubevirt
# Wait for all pods in kubevirt namespace to be Running
$ watch -n 5 kubectl get pods -n kubevirt
.. note::
# Wait for all pods in cdi namespace to be Running
$ watch -n 5 kubectl get pods -n cdi
When installing the containerized KubeVirt application, the virtctl
client is installed on the host. You can test the client by running
virtctl in an SSH / Local Console shell of the controller.
#. Setup 'virtctl' client executable to be accessible from sysadmin's PATH
#. Optionally, Install noVNC in order to access a VM's graphical console thru a
browser.
.. code-block:: bash
.. note::
# Create /home/sysadmin/bin directory, if it doesn't exist already
$ mkdir -p /home/sysadmin/bin
noVNC is used to access a VM's graphical console in a browser, where
virtctl provides access to the VM's serial console.
# Create symbolic link in /home/sysadmin/bin to virtctl client executable installed on host in step 2)
$ cd /home/sysadmin/bin
$ ln -s /var/opt/kubevirt/virtctl-v0.53.1-linux-amd64 virtctl
Create and apply ``vnc-install.yaml``.
# Logout and log back in to ensure that /home/sysadmin/bin gets added to your PATH variable.
$ exit
.. code-block:: yaml
login: sysadmin
password:
apiVersion: v1
kind: ServiceAccount
metadata:
name: virtvnc
namespace: kubevirt
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: virtvnc
subjects:
- kind: ServiceAccount
name: virtvnc
namespace: kubevirt
roleRef:
kind: ClusterRole
name: virtvnc
apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: virtvnc
rules:
- apiGroups:
- subresources.kubevirt.io
resources:
- virtualmachineinstances/console
- virtualmachineinstances/vnc
verbs:
- get
- apiGroups:
- kubevirt.io
resources:
- virtualmachines
- virtualmachineinstances
- virtualmachineinstancepresets
- virtualmachineinstancereplicasets
- virtualmachineinstancemigrations
verbs:
- get
- list
- watch
---
apiVersion: v1
kind: Service
metadata:
labels:
app: virtvnc
name: virtvnc
namespace: kubevirt
spec:
ports:
- port: 8001
protocol: TCP
targetPort: 8001
nodePort: 31002
selector:
app: virtvnc
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: virtvnc
namespace: kubevirt
spec:
replicas: 1
selector:
matchLabels:
app: virtvnc
template:
metadata:
labels:
app: virtvnc
spec:
serviceAccountName: virtvnc
nodeSelector:
node-role.kubernetes.io/master: ''
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Equal"
value: ""
effect: "NoSchedule"
containers:
- name: virtvnc
image: quay.io/samblade/virtvnc:latest
livenessProbe:
httpGet:
port: 8001
path: /
scheme: HTTP
failureThreshold: 30
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
$ which virtctl
/home/sysadmin/bin/virtctl
.. code-block:: none
kubectl apply -f vnc-install.yaml
To access the VNC: http://<OAM FLOATING IP>:31002
.. rubric:: |result|

321
doc/source/kube-virt/interface-and-networks-7cadb7bdb80b.rst Normal file
View File

@ -0,0 +1,321 @@
.. WARNING: Add no lines of text between the label immediately following
.. and the title.
.. _interface-and-networks-7cadb7bdb80b:
======================
Interface and Networks
======================
Connecting a |VM| to a network consists of two parts. First, networks are
specified in ``spec.networks``. Then, interfaces backed by the networks are
added to the VM by specifying them in ``spec.domain.devices.interfaces``.
Each interface must have a corresponding network with the same name.
An interface defines a virtual network interface of a virtual machine (also
called a frontend). A network specifies the backend of an interface and
declares which logical or physical device it is connected to (also called as
backend).
MacVTap
-------
In MacVTap mode, virtual machines are directly exposed to the Kubernetes nodes
L2 network. This is achieved by 'extending' an existing network interface with
a virtual device that has its own MAC address.
MacVTap interfaces are feature gated; to enable the feature, follow
instructions, in order to activate the MacVTap feature gate (case sensitive).
How to Activate a Feature Gate
------------------------------
.. code-block:: none
cat << END > enable-feature-gate.yaml
---
apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
name: kubevirt
namespace: kubevirt
spec:
configuration:
developerConfiguration:
featureGates:
- LiveMigration
- Macvtap
END
kubectl apply -f enable-feature-gate.yaml
.. note::
Make sure to add all existing feature gates in overrides file.
Alternatively, the existing kubevirt custom resources can be altered:
.. code-block:: none
kubectl edit kubevirt kubevirt -n kubevirt
...
spec:
configuration:
developerConfiguration:
featureGates:
- DataVolumes
- LiveMigration
- Macvtap
.. note::
The names of the feature gates are case sensitive.
Below is the usage example of MacVTap interface used by |VM|:
Create network attachment for the MacVTap network:
.. code-block:: none
kind: NetworkAttachmentDefinition
apiVersion: k8s.cni.cncf.io/v1
metadata:
name: macvtapnetwork
annotations:
k8s.v1.cni.cncf.io/resourceName: macvtap.network.kubevirt.io/dataplane1
spec:
config: '{
"cniVersion": "0.3.1",
"name": "macvtapnetwork",
"type": "macvtap",
"mtu": 1500
}'
.. note::
By running this yaml, the system will create 10
``macvtap.network.kubevirt.io/dataplane1`` network attachments or
interfaces.
Now you can create the |VM| using MacVTap network, for example:
.. code-block:: none
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
labels:
special: vmi-host-network
name: vmi-host-network
spec:
running: true
template:
metadata:
labels:
kubevirt.io/size: small
kubevirt.io/domain: fedora
spec:
domain:
cpu:
cores: 1
devices:
disks:
- name: containerdisk
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
interfaces:
- name: hostnetwork
macvtap: {}
resources:
requests:
memory: 1024M
networks:
- name: hostnetwork
multus:
networkName: macvtapnetwork
volumes:
- name: containerdisk
containerDisk:
image: docker.io/kubevirt/fedora-cloud-container-disk-demo:devel
- name: cloudinitdisk
cloudInitNoCloud:
userData: |-
#!/bin/bash
echo "fedora" |passwd fedora --stdin
Multus
------
It is also possible to connect VMIs to secondary networks using Multus. This
assumes that multus is installed across your cluster and a corresponding
``NetworkAttachmentDefinition`` |CRD| was created.
Example:
.. note::
First create the respective network attachment, for example if you want to
use MacVTap or |SRIOV| for a secondary interface.
.. code-block:: none
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
labels:
special: vmi-host-network
name: vmi-host-network-2
spec:
running: true
template:
metadata:
labels:
kubevirt.io/size: small
kubevirt.io/domain: fedora
spec:
domain:
cpu:
cores: 2
devices:
disks:
- name: containerdisk
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
interfaces:
- name: default
masquerade: {}
- name: hostnetwork
macvtap: {}
resources:
requests:
memory: 1024M
networks:
- name: default
pod: {}
- name: hostnetwork
multus:
networkName: macvtapnetwork
volumes:
- name: containerdisk
containerDisk:
image: docker.io/kubevirt/fedora-cloud-container-disk-demo:devel
- name: cloudinitdisk
cloudInitNoCloud:
userData: |-
#!/bin/bash
echo "fedora" |passwd fedora --stdin
In the example manifest above, the |VM| uses the first interface as default and
second interface is MacVTap which is mapped using multus.
SRIOV
-----
In |SRIOV| mode, virtual machines are directly exposed to an |SRIOV| |PCI|
device, usually allocated by Intel |SRIOV| device plugin. The device is passed
through into the guest operating system as a host device, using the |VFIO|
userspace interface, to maintain high networking performance.
.. note::
In |prod| |SRIOV| device plugin is part of the default platform
functionality.
.. note::
KubeVirt relies on |VFIO| userspace driver to pass |PCI| devices into the
|VMI| guest. As a result, when configuring |SRIOV|, define a pool of VF
resources that uses driver: vfio.
Example:
.. note::
Make sure an |SRIOV| interface is configured on a ``DATANETWORK``
(``sriovnet0`` for example below) on the |prod| host. For more details see
:ref:`provisioning-sr-iov-interfaces-using-the-cli`.
#. Create the Network attachment.
.. code-block:: none
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: sriov-net1
annotations:
k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_sriovnet0
spec:
config: '{
"type": "sriov",
"vlan": 5,
"cniVersion": "0.3.1",
"name": "sriov-net1"
}'
#. Launch the VM.
.. code-block:: none
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
labels:
special: vmi-sriov-network
name: vmi-sriov-network
spec:
running: true
template:
metadata:
labels:
kubevirt.io/size: small
kubevirt.io/domain: fedora
spec:
domain:
cpu:
cores: 1
devices:
disks:
- name: containerdisk
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
interfaces:
- masquerade: {}
name: default
- name: sriov-net1
sriov: {}
resources:
requests:
memory: 1024M
networks:
- name: default
pod: {}
- multus:
networkName: sriov-net1
name: sriov-net1
volumes:
- name: containerdisk
containerDisk:
image: docker.io/kubevirt/fedora-cloud-container-disk-demo:devel
- name: cloudinitdisk
cloudInitNoCloud:
userData: |-
#!/bin/bash
echo "fedora" |passwd fedora --stdin

7
doc/source/kube-virt/introduction-bb3a04279bf5.rst
View File

@ -1,8 +1,8 @@
.. _introduction-bb3a04279bf5:
============
Introduction
============
=====================
KubeVirt Introduction
=====================
The KubeVirt system application in StarlingX includes: KubeVirt, Containerized
Data Importer (|CDI|), and the Virtctl client tool.
@ -26,4 +26,3 @@ migrating |VMs|, canceling live migrations and uploading |VM| disk images.
See https://kubevirt.io/user-guide for more details.

124
doc/source/kube-virt/kubevirt-architecture-c92c8908e775.rst Normal file
View File

@ -0,0 +1,124 @@
.. _kubevirt-architecture-c92c8908e775:
=====================
KubeVirt Architecture
=====================
The following diagram describes the KubeVirt architecture:
.. figure:: figures/kubevirt-architecture.png
:width: 800
.. rubric:: ``virt-api-server``
HTTP API server serves as the entry point for all virtualization related flows.
The API server updates the virtualization related custom resource definition
(see below).
As the main entry point to KubeVirt it is responsible for defaulting and
validation of the provided VMI |CRDs|.
.. rubric:: VMI (CRD)
|VMI| definitions are kept as custom resources inside the Kubernetes API
server.
The |VMI| definition defines all properties of the |VM| itself, for example:
- Machine type
- CPU type
- Amount of RAM and |vCPUs|
- Number and type of |NICs|
.. rubric:: ``virt-controller``
The ``virt-controller`` has all the cluster wide virtualization functionality
and it is responsible for monitoring the |VMI| (|CRs|) and managing the
associated pods. Currently, the controller ensure the creation and management
of the life-cycle of the pods associated with the |VMI| objects.
A |VMI| object will always be associated with a pod during its life-time.
However, migration of a |VMI| the pod instance might change over time.
.. rubric:: ``virt-launcher``
For every |VMI| object one pod is created. This pod's primary container runs
the ``virt-launcher`` KubeVirt component.
Kubernetes or kubelet is not running the |VMIs|. Instead a daemon on every host
in the cluster launches a |VMI| process for every pod which is associated with
a |VMI| object whenever it is scheduled on a host.
The main purpose of the ``virt-launcher`` pod is to provide ``cgroups`` and
``namespaces``, used to host the |VMI| process.
``virt-handler`` signals ``virt-launcher`` to start a |VMI| by passing the
|VMI|'s |CRD| object to ``virt-launcher``. ``virt-launcher`` uses a local
``libvirtd`` instance within its container to start the |VMI|. The
``virt-launcher`` monitors the |VMI| process and terminates once the |VMI| has
exited.
If the Kubernetes runtime attempts to shutdown the ``virt-launcher`` pod before
|VMI| has exited, ``virt-launcher`` forwards signals from Kubernetes to the
|VMI| process and attempts to hold off the termination of the pod until the
|VMI| has shutdown successfully.
.. rubric:: ``virt-handler``
Every host needs a single instance of ``virt-handler``. It can be delivered as
a DaemonSet.
Like the virt-controller, the ``virt-handler`` is also reactive and watches
for changes of the |VMI| object, once detected it will perform all necessary
operations to change a |VMI| to meet the required state.
This behavior is similar to the choreography between the Kubernetes |VMI|
server and the kubelet.
The functions of the ``virt-handler`` are:
- Keeps a cluster-level |VMI| specification in sync with a corresponding
``libvirt`` domain.
- Reports domain state and spec changes to the cluster.
- Invokes node-centric plugins which can fulfill networking and storage
requirements defined in |VMI| specs.
.. rubric:: ``Libvirtd``
An instance of ``libvirtd`` is present in every |VMI| pod. ``virt-launcher``
uses ``libvirtd`` to manage the life-cycle of the |VMI| process.
Additional components from |prod| Kubernetes are:
- Storage
- Networking
- |RBAC|
.. rubric:: Simplified Diagram
.. figure:: figures/kubrvirt-simplified-diagram.png
:width: 800
---------------------------
Containerized Data Importer
---------------------------
The |CDI| project provides facilities for enabling |PVCs| to be used as disks
for KubeVirt |VMs| by way of DataVolumes. The three main |CDI| use cases are:
- Import a disk image from a web server or container registry to a DataVolume.
- Clone an existing |PVC| to a DataVolume.
- Upload a local disk image to a DataVolume.

8
doc/source/kube-virt/removal-97cc897941bc.rst → doc/source/kube-virt/kubevirt-removal-97cc897941bc.rst
View File

@ -1,8 +1,8 @@
.. _removal-97cc897941bc:
.. _kubevirt-removal-97cc897941bc:
=======
Removal
=======
================
KubeVirt Removal
================
.. rubric:: |proc|

156
doc/source/kube-virt/live-migration-support-for-vms-bea635bacc50.rst Normal file
View File

@ -0,0 +1,156 @@
.. WARNING: Add no lines of text between the label immediately following
.. and the title.
.. _live-migration-support-for-vms-bea635bacc50:
==============================
Live Migration Support for VMs
==============================
Live migration is a process during which a running |VMI| moves to another
compute node while the guest workload continues to run and remain accessible.
Enable the Live Migration Support
---------------------------------
Live migration is enabled by default in recent versions of KubeVirt. Versions
prior to v0.56 must be enabled in the feature gates. The feature gates field in
the KubeVirt Custom Resource must be expanded by adding the live migration to
it.
Limitation for live migrations:
- |VMs| using a |PVC| must have a shared |RWX| access mode to be live
migrated.
- Live migration is not allowed with a pod network binding of bridge
interface type ().
- Live migration requires ports 49152, 49153 to be available in the
``virt-launcher`` pod. If these ports are explicitly specified in
a masquarade interface, live migration will not function.
Initiate Live Migration
-----------------------
Live migration is initiated by posting a |VMIM| object to the cluster. The
example below starts a migration process for a virtual machine instance
``vmi-fedora``.
.. code-block:: yaml
cat <<EOF> migration.yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceMigration
metadata:
name: migration-job
spec:
vmiName: vmi-fedora
EOF
kubectl apply -f migration.yaml
Use virtctl to initiate Live Migration
--------------------------------------
Live migration can also be initiated using ``virtctl``.
.. code-block:: none
virtctl migrate vmi-fedora
Live Migration for SRIOV based VMs
----------------------------------
It is possible to live migrate |SRIOV| based |VMs|, but there are some
limitations:
- Specify the MAC address statically and on host the name of the |SRIOV|
interface should be same on source and target host.
- Specify the Static IP address.
Below is an example manifest for |SRIOV| |VM| using a static MAC address:
.. code-block:: none
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
labels:
special: vmi-sriov-network
name: vmi-sriov-test-vm
spec:
running: true
template:
metadata:
labels:
kubevirt.io/size: small
kubevirt.io/domain: fedora
spec:
domain:
cpu:
cores: 4
devices:
disks:
- name: containerdisk
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
interfaces:
- macAddress: "02:00:00:00:00:01"
name: sriov-net1
sriov: {}
rng: {}
resources:
requests:
memory: 1024M
networks:
- multus:
networkName: sriov-net1
name: sriov-net1
volumes:
- containerDisk:
image: docker.io/kubevirt/fedora-cloud-container-disk-demo:devel
name: containerdisk
- cloudInitNoCloud:
networkData: |
ethernets:
sriov-net1:
addresses:
- 10.10.10.12/24
gateway: 10.10.10.1
match:
macAddress: "02:00:00:00:00:01"
nameservers:
addresses:
- 10.96.0.10
search:
- default.svc.cluster.local
- svc.cluster.local
- cluster.local
set-name: sriov-link-enabled
version: 2
userData: |-
#!/bin/bash
echo "fedora" |passwd fedora --stdin
name: cloudinitdisk
Below is an example manifest to initiate the live migration:
.. code-block:: none
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceMigration
metadata:
name: migration-job-1
spec:
vmiName: vmi-sriov-test-vm

183
doc/source/kube-virt/node-assignment-using-nodeselector-affinity-and-antiminusaffi-06ad222ceb13.rst Normal file
View File

@ -0,0 +1,183 @@
.. WARNING: Add no lines of text between the label immediately following
.. and the title.
.. _node-assignment-using-nodeselector-affinity-and-antiminusaffi-06ad222ceb13:
=======================================================================================
Node Assignment Using NodeSelector, Affinity and Anti-Affinity, Taints, and Tolerations
=======================================================================================
Setting ``spec.nodeSelector`` requirements, constrains the scheduler to only
schedule VMs on nodes, which contain the specified labels. In the following
example the |VMI| contains the label ``app: kload`` (in this case on
controller-1).
Example of a VM manifest using ``nodeSelector``:
.. code-block:: none
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
name: testvmi-nocloud
spec:
nodeSelector:
app: kload
terminationGracePeriodSeconds: 30
domain:
resources:
requests:
memory: 1024M
devices:
disks:
- name: containerdisk
disk:
bus: virtio
- name: emptydisk
disk:
bus: virtio
- disk:
bus: virtio
name: cloudinitdisk
volumes:
- name: containerdisk
containerDisk:
image: kubevirt/fedora-cloud-container-disk-demo:latest
- name: emptydisk
emptyDisk:
capacity: "2Gi"
- name: cloudinitdisk
cloudInitNoCloud:
userData: |-
#cloud-config
password: fedora
chpasswd: { expire: False }
The ``spec.affinity`` field allows specifying hard- and soft-affinity for
|VMs|. It is possible to write matching rules against workloads (|VMs| and
pods) and Nodes. Since |VMs| are a workload type based on pods, Pod-affinity
affects |VMs| as well.
Pod affinity allows you to specify which POD/VM should be scheduled together
(on the same node or in the same topology, like a zone).
Where ``requiredDuringSchedulingIgnoredDuringExecution`` means that the
scheduler must place the pod/VM on a node that matches the affinity rules
during scheduling, but once the pod is running, the rule is ignored if other
changes happen (e.g., pods with the same label are removed).
The rule here says that the pod must be scheduled on a node that has other
POD/VM with the following characteristics: Label Selector: It looks for POD/VM
with the label security and the value ``S1``. This is defined in the
``matchExpressions`` part of the configuration.
``TopologyKey: failure-domain.beta.kubernetes.io/zone`` means that the pods
with the security: ``S1`` label should be in the same zone as the POD/VM being
scheduled. Essentially, the pod should be scheduled in the same failure domain
(zone) as other pods with the security: ``S1`` label.
Pod anti-affinity specifies that certain pods should not be scheduled together
(on the same node or topology).
Where ``preferredDuringSchedulingIgnoredDuringExecution`` is a "soft" rule
where the scheduler prefers to place the pod according to the anti-affinity
rule, but it is not required. If the rule cannot be met, the POD/VM will still
be scheduled, but it will try to follow the rule when possible.
The rule here says that the scheduler prefers to place the POD/VM on a node
that avoids other pods with the following characteristics: Label Selector:
POD/VM with the label security and value ``S2``.
``TopologyKey: kubernetes.io/hostname`` means that the anti-affinity applies to
the hostname (i.e., the pod should prefer not to be placed on the same node as
POD/VM with the security: ``S2`` label).
``Weight``: The weight of 100 indicates the strength of the preference. A
higher weight means the scheduler will try harder to respect the rule, but it
is still not guaranteed.
Example of a |VM| manifest using affinity and anti-affinity:
.. code-block:: none
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
spec:
nodeSelector:
cpu: slow
storage: fast
domain:
resources:
requests:
memory: 64M
devices:
disks:
- name: mypvcdisk
lun: {}
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: security
operator: In
values:
- S1
topologyKey: failure-domain.beta.kubernetes.io/zone
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: security
operator: In
values:
- S2
topologyKey: kubernetes.io/hostname
volumes:
- name: mypvcdisk
persistentVolumeClaim:
claimName: mypvc
Affinity as described above, is a property of |VMs| that attracts them to a set
of nodes (either as a preference or a hard requirement). Taints are the
opposite - they allow a node to repel a set of |VMs|.
Taints and tolerations work together to ensure that |VMs| are not scheduled
onto inappropriate nodes. One or more taints are applied to a node; this
ensures that the node should not accept any |VMs| that do not tolerate the
taints. Tolerations are applied to |VMs|, and allow (but do not require) the
|VMs| to schedule onto nodes with matching taints.
Example of |VM| manifest using taint and tolerance.
You add a taint to a node using ``kubectl taint``. For example, ``kubectl taint
nodes node1 key=value:NoSchedule``.
Below is an example of adding a toleration of this taint to a VM:
.. code-block:: none
metadata:
name: testvmi-ephemeral
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
spec:
nodeSelector:
cpu: slow
storage: fast
domain:
resources:
requests:
memory: 64M
devices:
disks:
- name: mypvcdisk
lun: {}
tolerations:
- key: "key"
operator: "Equal"
value: "value"
effect: "NoSchedule"

89
doc/source/kube-virt/persistent-storage-for-vms-8ddc8fa611aa.rst Normal file
View File

@ -0,0 +1,89 @@
.. WARNING: Add no lines of text between the label immediately following
.. and the title.
.. _persistent-storage-for-vms-8ddc8fa611aa:
==========================
Persistent Storage for VMs
==========================
Allows connecting a ``PersistentVolumeClaim`` to a |VM| disk.
Use a ``PersistentVolumeClaim`` when the ``VirtualMachineInstance`` disk needs
to persist after the |VM| terminates. This allows for the |VM|'s data to remain
persistent between restarts.
A ``PersistentVolume`` can be in ``filesystem`` or ``block`` mode.
Below is a simple example which attaches a ``PersistentVolumeClaim`` as a disk
to a VM:
.. code-block:: none
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
name: testvmi-pvc
spec:
domain:
resources:
requests:
memory: 64M
devices:
disks:
- name: mypvcdisk
lun: {}
volumes:
- name: mypvcdisk
persistentVolumeClaim:
claimName: mypvc
Upload VM Image to CDI
----------------------
You can upload/import a VM image to |CDI|, creating a |PVC| on system, so you
can use the |PVC| to boot or create the |VM|.
#. Get the upload proxy Server IP.
.. code-block:: none
kubectl -n cdi get svc |grep cdi-uploadproxy
cdi-uploadproxy ClusterIP 10.111.132.43 <none> 443/TCP 85m
#. Use ``virtctl`` client upload the image to |CDI|.
.. code-block:: none
virtctl image-upload --pvc-name=cirros-vm-disk-test-2 --pvc-size=500Mi --image-path=/home/sysadmin/kubevirt-valid/cirros-0.5.1-x86_64-disk.img --uploadproxy-url=https://<uploadproxy-svc-ip> --insecure
.. note::
Now you can use ``pvc cirros-vm-disk-test-2`` in |VM| specs.
.. code-block:: none
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
name: testvmi-pvc-1
spec:
domain:
resources:
requests:
memory: 64M
devices:
disks:
- name: mypvcdisk
lun: {}
volumes:
- name: mypvcdisk
persistentVolumeClaim:
claimName: cirros-vm-disk-test-2
If you want to upload an image located outside the cluster, the cluster
admin should expose the |CDI| upload proxy to the outside, for example with
NodePort.
For more details see :ref:`set-up-cdi-proxy-ad165d884417`.

93
doc/source/kube-virt/startup-scripts-6402154a6f37.rst Normal file
View File

@ -0,0 +1,93 @@
.. WARNING: Add no lines of text between the label immediately following
.. and the title.
.. _startup-scripts-6402154a6f37:
===============
Startup Scripts
===============
KubeVirt supports the ability to assign a startup script to a |VMI| instance
which is executed automatically when the |VM| initializes.
These scripts are commonly used to automate injection of users and |SSH| keys
into |VMs| in order to provide remote access to the machine. For example, a
startup script can be used to inject credentials into a |VM| that allows an
Ansible job running on a remote host to access and provision the |VM|.
Startup scripts are not limited to any specific use case though. They can be
used to run any arbitrary script in a |VM| on boot.
Example of use:
Assign static IP using ``cloudInitNoCloud``.
.. code-block:: none
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
labels:
special: vmi-sriov-network
name: vmi-sriov-network
spec:
running: true
template:
metadata:
labels:
kubevirt.io/size: small
kubevirt.io/domain: fedora
spec:
domain:
cpu:
cores: 4
devices:
disks:
- name: containerdisk
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
interfaces:
- masquerade: {}
name: default
- macAddress: "02:00:00:00:00:01"
name: sriov-net1
sriov: {}
rng: {}
resources:
requests:
memory: 1024M
networks:
- name: default
pod: {}
- multus:
networkName: sriov-net1
name: sriov-net1
volumes:
- containerDisk:
image: docker.io/kubevirt/fedora-cloud-container-disk-demo:devel
name: containerdisk
- cloudInitNoCloud:
networkData: |
ethernets:
sriov-net1:
addresses:
- 10.10.10.14/24
gateway: 10.10.10.1
match:
macAddress: "02:00:00:00:00:01"
nameservers:
addresses:
- 10.96.0.10
search:
- default.svc.cluster.local
- svc.cluster.local
set-name: sriov-link-enabled
version: 2
# userData: '#!/bin/bash echo "fedora" | passwd fedora --stdin'
userData: |-
#!/bin/bash
echo "fedora" |passwd fedora --stdin
name: cloudinitdisk

121
doc/source/kube-virt/static-ip-assignment-via-cloudinit-configuration-d053375e78fa.rst Normal file
View File

@ -0,0 +1,121 @@
.. WARNING: Add no lines of text between the label immediately following
.. and the title.
.. _static-ip-assignment-via-cloudinit-configuration-d053375e78fa:
================================================
Static IP Assignment via Cloudinit Configuration
================================================
KubeVirt supports cloud-init's NoCloud and ConfigDrive datasources, which
involve injecting startup scripts into a |VM| instance using of an ephemeral
disk. |VMs| with the cloud-init package installed detect the ephemeral disk and
execute custom userdata scripts at boot.
In the example below, the |VM| is using NoCloud for assigning the static IP and
MAC address on |SRIOV| interface.
.. code-block:: none
cat <<EOF>cloud-NoCloud-test.yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
labels:
special: vmi-sriov-network
name: vmi-sriov-test-vm-1
spec:
running: true
template:
metadata:
labels:
kubevirt.io/size: small
kubevirt.io/domain: fedora
spec:
domain:
cpu:
cores: 4
devices:
disks:
- name: containerdisk
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
interfaces:
- masquerade: {}
name: default
- macAddress: "02:00:00:00:00:03"
name: sriov-net1
sriov: {}
rng: {}
resources:
requests:
memory: 1024M
networks:
- name: default
pod: {}
- multus:
networkName: sriov-net1
name: sriov-net1
volumes:
- containerDisk:
image: docker.io/kubevirt/fedora-cloud-container-disk-demo:devel
name: containerdisk
- cloudInitNoCloud:
networkData: |
ethernets:
sriov-net1:
addresses:
- 10.10.10.42/24
gateway: 10.10.10.1
match:
macAddress: "02:00:00:00:00:03"
nameservers:
addresses:
- 10.96.0.10
search:
- default.svc.cluster.local
- svc.cluster.local
- cluster.local
set-name: sriov-link-enabled
version: 2
userData: |-
#!/bin/bash
echo "fedora" |passwd fedora --stdin
name: cloudinitdisk
EOF
kubectl apply -f cloud-NoCloud-test.yaml
[sysadmin@controller-0 kubevirt-GA-testing(keystone_admin)]$ kubectl get vmi
NAME AGE PHASE IP NODENAME READY
vmi-sriov-test-vm-1 3h17m Running 172.16.166.152 controller-1 True
[sysadmin@controller-0 kubevirt-GA-testing(keystone_admin)]$ virtctl console vmi-sriov-test-vm-1
Successfully connected to vmi-sriov-test-vm-1 console. The escape sequence is ^]vmi-sriov-test-vm-1 login: fedora
Password:
[fedora@vmi-sriov-test-vm-1 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 52:54:00:90:b4:d7 brd ff:ff:ff:ff:ff:ff
altname enp1s0
inet 10.0.2.2/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
valid_lft 86301690sec preferred_lft 86301690sec
inet6 fe80::5054:ff:fe90:b4d7/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 02:00:00:00:00:03 brd ff:ff:ff:ff:ff:ff
altname enp8s0
inet 10.10.10.42/24 brd 10.10.10.10.255 scope global noprefixroute eth1
valid_lft forever preferred_lft forever
inet6 fe80::ff:fe00:3/64 scope link
valid_lft forever preferred_lft forever
[fedora@vmi-sriov-test-vm-1 ~]$

339
doc/source/kube-virt/use-shared-resource-cpu-ad295227aa0c.rst Normal file
View File

@ -0,0 +1,339 @@
.. _use-shared-resource-cpu-ad295227aa0c:
=========================
Use Shared Resource (CPU)
=========================
In |prod| the shared CPUs are allocated under Application pool. All unused CPUs
are by default allocated under Application CPU pool.
Use the manifest example below for the shared CPU:
.. code-block:: yaml
:emphasize-lines: 15,16
cat <<EOF> sample-vm-diff-huge-pages-size.yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: ubuntu-bionic-1g
spec:
running: true
template:
metadata:
labels:
kubevirt.io/size: small
kubevirt.io/domain: ubuntu-bionic
spec:
domain:
cpu:
cores: 1
devices:
disks:
- name: containervolume
disk:
bus: virtio
- name: cloudinitvolume
disk:
bus: virtio
interfaces:
- name: default
masquerade: {}
resources:
requests:
memory: "10Gi"
memory:
hugepages:
pageSize: "1Gi"
networks:
- name: default
pod: {}
volumes:
- name: containervolume
containerDisk:
image: tedezed/ubuntu-container-disk:20.0
- name: cloudinitvolume
cloudInitNoCloud:
userData: |-
#cloud-config
chpasswd:
list: |
ubuntu:ubuntu
root:root
expire: False
EOF
.. note::
In VM manifest, the CPUs, which are not defined as dedicated, will be
allocated from the Application (Shared) Pool.
--------------------------
Assign Dedicated Resources
--------------------------
Dedicated CPU Resources
-----------------------
Certain workloads requiring predictable latency and enhanced performance during
execution would benefit from obtaining dedicated CPU resources. KubeVirt,
relying on the Kubernetes CPU manager, is able to pin guest's vCPUs to the
host's pCPUs.
.. note::
Enable CPU Manager on |prod| host before using dedicated CPU resources.
Kubernetes does not provide CPU Manager detection, so you need to add
CpuManager feature gate manually to KubeVirt CR:
.. code-block:: none
apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
name: kubevirt
namespace: kubevirt
spec:
configuration:
developerConfiguration:
featureGates:
- LiveMigration
- Macvtap
- Snapshot
- CPUManager
Then, check the label:
.. code-block:: none
~(keystone_admin)]$ kubectl describe node | grep cpumanager cpumanager=true
Request Dedicated CPU Resources from Application CPU Pool
---------------------------------------------------------
Setting ``spec.domain.cpu.dedicatedCpuPlacement`` to true in a VMI spec will
indicate the desire to allocate dedicated CPU resource to the VMI.
Kubevirt will verify that all the necessary conditions are met, for the
Kubernetes CPU manager to pin the ``virt-launcher`` container to dedicated host
CPUs. Once, ``virt-launcher`` is running, the VMI's vCPUs will be pinned to the
pCPUS that has been dedicated for the ``virt-launcher`` container.
Expressing the desired amount of VMI's vCPUs can be done by either setting the
guest topology in ``spec.domain.cpu`` (sockets, cores, threads) or
``spec.domain.resources.[requests/limits].cpu`` to a whole number integer
``([1-9]+)`` indicating the number of vCPUs requested for the VMI. Number of
vCPUs is counted as sockets * cores * threads or if ``spec.domain.cpu`` is
empty then it takes value from ``spec.domain.resources.requests.cpu`` or
``spec.domain.resources.limits.cpu``.
Example of sample manifest file:
.. code-block:: yaml
:emphasize-lines: 14,15,16,17,18
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: ubuntu-bionic-dedicated-cpu
spec:
running: true
template:
metadata:
labels:
kubevirt.io/size: small
kubevirt.io/domain: ubuntu-bionic
spec:
domain:
cpu:
cores: 2
sockets: 1
threads: 1
dedicatedCpuPlacement: true
devices:
disks:
- name: containervolume
disk:
bus: virtio
- name: cloudinitvolume
disk:
bus: virtio
interfaces:
- name: default
masquerade: {}
resources:
requests:
memory: 2048M
networks:
- name: default
pod: {}
volumes:
- name: containervolume
containerDisk:
image: tedezed/ubuntu-container-disk:20.0
- name: cloudinitvolume
cloudInitNoCloud:
userData: |-
#cloud-config
chpasswd:
list: |
ubuntu:ubuntu
root:root
expire: False
Isolated CPU Resources
----------------------
|prod| supports running the most critical low-latency applications on host CPUs
which are completely isolated from the host process scheduler. This allows you
to customize Kubernetes CPU management when the policy is set to static so that
low-latency applications run with optimal efficiency.
Request Isolated CPU Resources
------------------------------
.. note::
Make sure |prod| host is configured with isolated CPU cores.
Refer to documentation for isolated CPU
:ref:`isolating-cpu-cores-to-enhance-application-performance`.
By specifying the ``windriver.com/isolcpus: x`` in VM specs , allocates the
CPUs from application isolated core pools.
Below is the example manifest requesting the isolated cores.
.. code-block:: yaml
:emphasize-lines: 31,32,33,34,35,36
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: ubuntu-bionic-isol-cores
spec:
running: true
template:
metadata:
labels:
kubevirt.io/size: small
kubevirt.io/domain: ubuntu-bionic
spec:
domain:
cpu:
sockets: 1
cores: 4
threads: 1
dedicatedCpuPlacement: true
devices:
disks:
- name: containervolume
disk:
bus: virtio
- name: cloudinitvolume
disk:
bus: virtio
interfaces:
- name: default
masquerade: {}
resources:
requests:
memory: 4Gi
windriver.com/isolcpus: 4
limits:
memory: 4Gi
windriver.com/isolcpus: 4
networks:
- name: default
pod: {}
volumes:
- name: containervolume
containerDisk:
image: tedezed/ubuntu-container-disk:20.0
- name: cloudinitvolume
cloudInitNoCloud:
userData: |-
#cloud-config
chpasswd:
list: |
ubuntu:ubuntu
root:root
expire: False
Huge Pages
----------
Huge pages, in the context of Kubevirt VM, refers to a Linux kernel feature
that allows for the efficient use of large pages, also known as large memory
pages. Huge pages can be leveraged to enhance the performance of
memory-intensive applications, especially those with large datasets.
Request Huge Pages
------------------
.. note::
Make sure |prod| host is pre-configured with huge pages.
For more details on huge pages configuration, refer to
:ref:`allocating-host-memory-using-the-cli`.
You can request the memory in form of huge pages by specifying the
``spec.domain.memory.hugePages.pageSize``.
Example:
.. code-block:: yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: ubuntu-bionic-1g
spec:
running: true
template:
metadata:
labels:
kubevirt.io/size: small
kubevirt.io/domain: ubuntu-bionic
spec:
domain:
cpu:
cores: 1
devices:
disks:
- name: containervolume
disk:
bus: virtio
- name: cloudinitvolume
disk:
bus: virtio
interfaces:
- name: default
masquerade: {}
resources:
requests:
memory: "10Gi"
memory:
hugepages:
pageSize: "1Gi"
networks:
- name: default
pod: {}
volumes:
- name: containervolume
containerDisk:
image: tedezed/ubuntu-container-disk:20.0
- name: cloudinitvolume
cloudInitNoCloud:
userData: |-
#cloud-config
chpasswd:
list: |
ubuntu:ubuntu
root:root
expire: False

342
doc/source/kube-virt/virtual-machine-dd561f6db3fd.rst Normal file
View File

@ -0,0 +1,342 @@
.. _virtual-machine-dd561f6db3fd:
===============
Virtual Machine
===============
Every |VMI| object represents a single running virtual machine instance.
While |VM| represents the abstraction of virtual machine instances.
.. contents:: |minitoc|
:local:
:depth: 2
---------------------------------
Create Virtual Machines Manifests
---------------------------------
The ``virtctl`` sub command ``create vm`` allows easy creation of
VirtualMachine manifests from the command line. It leverages instance types and
preferences and provides several flags to control details of the created |VM|.
For example there are flags to specify the name or run strategy of a |VM| or
flags to add volumes to a |VM|. Instance types and preferences can either be
specified directly or it is possible to let KubeVirt infer those from the
volume used to boot the |VM|.
For a full set of flags and their description use the following command:
.. code-block:: none
virtctl create vm -h
Create Virtual Machines on a Cluster
------------------------------------
The output of ``virtctl create vm`` can be piped into ``kubectl`` to directly
create a |VM| on a cluster.
For example:
.. code-block:: none
# Create a VM with name my-vm on the cluster
virtctl create vm --name my-vm | kubectl create -f -
virtualmachine.kubevirt.io/my-vm created
Specify Cloud-init User Data
----------------------------
To pass ``cloud-init`` user data to ``virtctl`` it needs to be encoded into a
``base64`` string.
For example:
.. code-block:: none
# Put your cloud-init user data into a file.
# This will add an authorized key to the default user.
# To get the default username read the documentation for the cloud image
$ cat cloud-init.txt
#cloud-config
ssh_authorized_keys:
- ssh-rsa AAAA...
# Base64 encode the contents of the file without line wraps and store it in a variable
$ CLOUD_INIT_USERDATA=$(base64 -w 0 cloud-init.txt)
# Show the contents of the variable
$ echo $CLOUD_INIT_USERDATA I2Nsb3VkLWNvbmZpZwpzc2hfYXV0aG9yaXplZF9rZXlzOgogIC0gc3NoLXJzYSBBQUFBLi4uCg==
You can now use this variable as an argument to the ``--cloud-init-user-data``
flag:
.. code-block:: none
virtctl create vm --cloud-init-user-data $CLOUD_INIT_USERDATA
Create VM Directly from Manifest
--------------------------------
You can create the |VM| or virtual machine instances directly from manifest.
.. code-block:: yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: sample-test-vm
spec:
running: true
template:
metadata:
labels:
kubevirt.io/size: small
kubevirt.io/domain: ubuntu-bionic
spec:
domain:
cpu:
cores: 1
devices:
disks:
- name: containervolume
disk:
bus: virtio
- name: cloudinitvolume
disk:
bus: virtio
interfaces:
- name: default
masquerade: {}
resources:
requests:
memory: 1024M
networks:
- name: default
pod: {}
volumes:
- name: containervolume
containerDisk:
image: tedezed/ubuntu-container-disk:20.0
- name: cloudinitvolume
cloudInitNoCloud:
userData: |-
#cloud-config
chpasswd:
list: |
ubuntu:ubuntu
root:root
expire: False
----------
Access VMs
----------
Once a |VM| is started, you are able to connect to the consoles it exposes.
Usually there are two types of consoles:
- Serial Console
The serial console of a virtual machine can be accessed by using
the console command:
.. code-block:: none
virtctl console testvm
- Graphical Console (VNC)
If NoVnc is installed on Cluster and exposed over NodePort, you can access
virtual machines with ``http://<cluster IP>:port``.
.. note::
Refer the section :ref:`Install NoVNC <installation-66477d7646db>` for
more details.
---------
Lifecycle
---------
Every |VMI| represents a single virtual machine instance. In general, the
management of |VMIs| is kept similar to how Pods are managed. Every |VM| that
is defined in the cluster is expected to be running, just like pods. Deleting a
|VMI| is equivalent to shutting it down, this is also equivalent to how pods
behave.
Launch a Virtual Machine
------------------------
To start a |VMI|, you need to create a |VMI| object using ``kubectl``:
.. code-block:: none
kubectl create -f vmi.yaml
Example of ``vmi.yaml``:
.. code-block:: yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: sample-test-vm
spec:
running: true
template:
metadata:
labels:
kubevirt.io/size: small
kubevirt.io/domain: ubuntu-bionic
spec:
domain:
cpu:
cores: 1
devices:
disks:
- name: containervolume
disk:
bus: virtio
- name: cloudinitvolume
disk:
bus: virtio
interfaces:
- name: default
masquerade: {}
resources:
requests:
memory: 1024M
networks:
- name: default
pod: {}
volumes:
- name: containervolume
containerDisk:
image: tedezed/ubuntu-container-disk:20.0
- name: cloudinitvolume
cloudInitNoCloud:
userData: |-
#cloud-config
chpasswd:
list: |
ubuntu:ubuntu
root:root
expire: False
List the VM
-----------
|VMIs| can be listed by querying for |VMI| objects:
.. code-block:: none
kubectl get vmis
Retrieve a VMI Definition
-------------------------
A single |VMI| definition can be retrieved by getting the specific |VMI|
object.
.. code-block:: none
kubectl get vmis testvmi
Remove a VMI
------------
To stop the |VMI|, you need to delete the corresponding |VMI| object using
kubectl.
.. code-block:: none
$ kubectl delete -f vmi.yaml
OR
.. code-block:: none
$ kubectl delete vmis testvmi
.. note::
Stopping a |VMI| implies that it will be deleted from the cluster. You will
not be able to start this |VMI| object again.
Start and Stop a VM
-------------------
|VMs|, in contrast to |VMIs|, have a running state. Thus, on a |VM| you can
define if it should be running, or not. |VMIs| are always running and consuming
resources, if they are defined in the cluster
``virtctl`` is used to start a |VM|:
.. code-block:: none
$ virtctl start my-vm
And to stop a |VM|:
.. code-block:: none
$ virtctl stop my-vm
.. note::
If you run the force stop to the |VM|, it may result in data
inconsistencies, or there may be data loss.
Pause and Unpause a VM
----------------------
.. note::
Pausing in this context refers to ``libvirt``'s :command:`virDomainSuspend`
command. The process is frozen without further access to CPU resources and
I/O but the memory used by the domain at the hypervisor level will stay
allocated.
To pause a |VM|, you need the ``virtctl`` command line tool. The pause command
works on either |VMs| or |VMIs|:
.. code-block:: none
$ virtctl pause vm testvm
OR
.. code-block:: none
$ virtctl pause vmi testvm
Paused |VMIs| have a Paused condition in their status:
.. code-block:: none
$ kubectl get vmi testvm -o=jsonpath='{.status.conditions[?(@.type=="Paused")].message}'
VMI was paused by user
Unpausing works similar to pausing:
.. code-block:: none
$ virtctl unpause vm testvm
OR
.. code-block:: none
$ virtctl unpause vmi testvm

102
doc/source/kube-virt/virtualmachineinstancereplicaset-8518d55de52b.rst Normal file
View File

@ -0,0 +1,102 @@
.. WARNING: Add no lines of text between the label immediately following
.. and the title.
.. _virtualmachineinstancereplicaset-8518d55de52b:
====================================
Virtual Machine Instance Replica Set
====================================
A |VMIRS| tries to ensure that a specified number of |VMI| replicas are
running at any time. Meaning, a |VMIRS| makes sure that a |VMI| or a
homogeneous set of |VMIs| is always up and ready.
Use VMIRS
---------
The |VMIRS| allows you to specify a |VMI| Template in ``spec.template``. It
consists of ``ObjectMetadata`` in ``spec.template.metadata``, and a |VMI| spec
in ``spec.template.spec``. The specification of the |VM| is equal to the
specification of the |VM| in the |VMI| workload.
``spec.replicas`` can be used to specify how many replicas are wanted. If
unspecified, the default value is 1. This value can be updated anytime. The
controller reacts to the changes.
Example manifest for ``replicaset``:
.. code-block:: none
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceReplicaSet
metadata:
name: testreplicaset
spec:
replicas: 3
selector:
matchLabels:
myvmi: myvmi
template:
metadata:
name: test
labels:
myvmi: myvmi
spec:
domain:
devices:
disks:
- disk:
name: containerdisk
resources:
requests:
memory: 64M
volumes:
- name: containerdisk
containerDisk:
image: kubevirt/cirros-container-disk-demo:latest
Configure Horizontal VM Autoscaler
----------------------------------
.. code-block:: none
[sysadmin@controller-0 ravi-test(keystone_admin)]$ cat hpa.yaml
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: test-hpa
spec:
scaleTargetRef:
kind: VirtualMachineInstanceReplicaSet
name: testreplicaset
apiVersion: kubevirt.io/v1
minReplicas: 3
maxReplicas: 5
targetCPUUtilizationPercentage: 50
---
Verify :
[sysadmin@controller-0 (keystone_admin)]$ kubectl describe horizontalpodautoscaler.autoscaling
Name: test-hpa
Namespace: default
Labels: <none>
Annotations: autoscaling.alpha.kubernetes.io/conditions:
[{"type":"AbleToScale","status":"True","lastTransitionTime":"2023-11-30T18:17:34Z","reason":"ScaleDownStabilized","message":"recent recomm...
autoscaling.alpha.kubernetes.io/current-metrics:
[\{"type":"Resource","resource":{"name":"cpu","currentAverageUtilization":1,"currentAverageValue":"2m"}}]
CreationTimestamp: Thu, 30 Nov 2023 19:17:19 +0100
Reference: VirtualMachineInstanceReplicaSet/testreplicaset
Target CPU utilization: 50%
Current CPU utilization: 1%
Min replicas: 3
Max replicas: 5
VirtualMachineInstanceReplicaSet pods: 3 current / 3 desired
Events: <none>
.. note::
Based on CPU usage in above example the autoscaler will scale up and down
the ``replicaset``.

129
doc/source/kube-virt/vm-snapshot-and-restore-21158b60cd56.rst Normal file
View File

@ -0,0 +1,129 @@
.. WARNING: Add no lines of text between the label immediately following
.. and the title.
.. _vm-snapshot-and-restore-21158b60cd56:
=======================
VM Snapshot and Restore
=======================
|VM| snapshot allows you to snapshot the running |VM| with existing
configuration and restore back to configuration point.
Snapshot a VM
-------------
Snapshotting a |VM| is supported for online and offline |VMs|.
When snapshotting a running |VM| the controller will check for the qemu guest
agent in the |VM|. If the agent exists it will freeze the |VM| filesystems
before taking the snapshot and unfreeze after the snapshot. It is recommended
to take online snapshots with the guest agent for a better snapshot, if not
present, a best effort snapshot will be taken.
.. rubric:: |proc|
To enable snapshot functionality system does require snapshot |CRD| and
snapshot controller to be created on system. Follow the steps below:
#. Run to install snapshot |CRD| and snapshot controller on Kubernets:
- ``kubectl apply -f`` https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
- ``kubectl apply -f`` https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
- ``kubectl apply -f`` https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
- ``kubectl apply -f`` https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
- ``kubectl apply -f`` https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml
#. Create ``VolumeSnapshotClass`` for ``cephfs`` and ``rbd``:
.. code-block:: none
cat <<EOF>cephfs-storageclass.yaml
—
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: csi-cephfsplugin-snapclass
driver: cephfs.csi.ceph.com
parameters:
clusterID: 60ee9439-6204-4b11-9b02-3f2c2f0a4344
csi.storage.k8s.io/snapshotter-secret-name: ceph-pool-kube-cephfs-data
csi.storage.k8s.io/snapshotter-secret-namespace: default
deletionPolicy: Delete
EOF
cat <<EOF>rbd-storageclass.yaml
—
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: csi-rbdplugin-snapclass
driver: rbd.csi.ceph.com
parameters:
clusterID: 60ee9439-6204-4b11-9b02-3f2c2f0a4344
csi.storage.k8s.io/snapshotter-secret-name: ceph-pool-kube-rbd
csi.storage.k8s.io/snapshotter-secret-namespace: default
deletionPolicy: Delete
EOF
.. note::
Get the cluster ID from: ``kubectl describe sc cephfs, rbd``.
#. Create snapshot manifest of running |VM| using the example yaml below:
.. code-block:: none
cat<<EOF>cirros-snapshot.yaml
apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineSnapshot
metadata:
name: snap-cirros
spec:
source:
apiGroup: kubevirt.io
kind: VirtualMachine
name: pvc-test-vm
failureDeadline: 3m
EOF
#. Apply the snapshot manifest and verify if the snapshot is successfully
created.
.. code-block:: none
kubectl apply -f cirros-snapshot.yaml
[sysadmin@controller-0 kubevirt-GA-testing(keystone_admin)]$ kubectl get VirtualMachineSnapshot
NAME SOURCEKIND SOURCENAME PHASE READYTOUSE CREATIONTIME ERROR
snap-cirros VirtualMachine pvc-test-vm Succeeded true 28m
Example manifest to restore the snapshot:
.. code-block:: none
<<EOF>cirros-restore.yaml
apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineRestore
metadata:
name: restore-cirros
spec:
target:
apiGroup: kubevirt.io
kind: VirtualMachine
name: pvc-test-vm
virtualMachineSnapshotName: snap-cirros
EOF
kubectl apply -f cirros-restore.yaml
Verify the snapshot restore:
.. code-block:: none
[sysadmin@controller-0 kubevirt-GA-testing(keystone_admin)]$ kubectl get VirtualMachineRestore
NAME TARGETKIND TARGETNAME COMPLETE RESTORETIME ERROR
restore-cirros VirtualMachine pvc-test-vm true 34m

95
doc/source/kube-virt/vm-using-secret-as-startup-configuration-4a8255e26b1f.rst Normal file
View File

@ -0,0 +1,95 @@
.. WARNING: Add no lines of text between the label immediately following
.. and the title.
.. _vm-using-secret-as-startup-configuration-4a8255e26b1f:
========================================
VM Using Secret as Startup Configuration
========================================
A secret can be presented to the |VM| as disk or as a filesystem.
The disk method does not support dynamic change propagation and the filesystem
method does not support live migration. Therefore, depending on the use-case,
one or the other may be more suitable.
Example of the creation of a Secret:
.. code-block:: none
apiVersion: v1
kind: Secret
metadata:
name: app-secret
type: Opaque
data:
username: YWxheA==
password: TGk2OW51eCo=
Example of a |VM| using secret as filesystem:
.. code-block:: none
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
labels:
special: vmi-fedora
name: vmi-fedora-secret
spec:
domain:
devices:
filesystems:
- name: app-secret-fs
virtiofs: {}
disks:
- disk:
bus: virtio
name: containerdisk
machine:
type: ""
resources:
requests:
memory: 1024M
terminationGracePeriodSeconds: 0
volumes:
- name: containerdisk
containerDisk:
image: quay.io/containerdisks/fedora:latest
- cloudInitNoCloud:
userData: |-
#cloud-config
chpasswd:
expire: false
password: fedora
user: fedora
bootcmd:
# mount the Secret
- "sudo mkdir /mnt/app-secret"
- "sudo mount -t virtiofs app-secret-fs /mnt/app-secret"
name: cloudinitdisk
- secret:
secretName: app-secret
name: app-secret-fs
Then, you can login to |VM| to verify:
.. code-block:: none
[fedora@vmi-fedora-secret ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda5 4.0G 453M 3.1G 13% /
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 450M 0 450M 0% /dev/shm
tmpfs 180M 720K 179M 1% /run
tmpfs 450M 0 450M 0% /tmp
/dev/vda2 966M 61M 840M 7% /boot
/dev/vda3 100M 12M 89M 12% /boot/efi
/dev/vda5 4.0G 453M 3.1G 13% /home
app-secret-fs 94G 8.0K 94G 1% /mnt/app-secret
tmpfs 90M 4.0K 90M 1% /run/user/1000
[fedora@vmi-fedora-secret ~]$ ls -lrth /mnt/app-secret
total 0
lrwxrwxrwx. 1 root 107 15 Jan 15 16:43 username -> ..data/username
lrwxrwxrwx. 1 root 107 15 Jan 15 16:43 password -> ..data/password

72
doc/source/kube-virt/vm-using-service-account-as-filesystem-5fd4deb7339a.rst Normal file
View File

@ -0,0 +1,72 @@
.. WARNING: Add no lines of text between the label immediately following
.. and the title.
.. _vm-using-service-account-as-filesystem-5fd4deb7339a:
======================================
VM Using Service Account as Filesystem
======================================
A ``serviceaccount`` volume references a Kubernetes ``serviceaccount``. A
``serviceaccount`` can be presented to the |VM| as disk or as a filesystem.
The disk method does not support dynamic change propagation and the filesystem
method does not support live migration. Therefore, depending on the use-case,
one or the other may be more suitable.
By using filesystem, ``serviceaccounts`` are shared through ``virtiofs``. In
contrast with using disk for sharing ``serviceaccounts``, filesystem allows you
to dynamically propagate changes on ``serviceaccounts`` to |VMIs| (i.e. the
|VM| does not need to be rebooted).
.. rubric:: Limitation
Currently, |VMIs| cannot be live migrated since ``virtiofs`` does not support
live migration.
Example of a |VM| creation using default service account:
.. code-block:: none
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
labels:
special: vmi-fedora-sa
name: vmi-fedora
spec:
domain:
devices:
filesystems:
- name: serviceaccount-fs
virtiofs: {}
disks:
- disk:
bus: virtio
name: containerdisk
machine:
type: ""
resources:
requests:
memory: 1024M
terminationGracePeriodSeconds: 0
volumes:
- name: containerdisk
containerDisk:
image: quay.io/containerdisks/fedora:latest
- cloudInitNoCloud:
userData: |-
#cloud-config
chpasswd:
expire: false
password: fedora
user: fedora
bootcmd:
# mount the ConfigMap
- "sudo mkdir /mnt/serviceaccount"
- "sudo mount -t virtiofs serviceaccount-fs /mnt/serviceaccount"
name: cloudinitdisk
- name: serviceaccount-fs
serviceAccount:
serviceAccountName: default

7
doc/source/shared/abbrevs.txt
View File

@ -174,6 +174,7 @@
.. |RPC| replace:: :abbr:`RPC (Remote Procedure Call)`
.. |RST| replace:: :abbr:`rST (reStructuredText)`
.. |RVMC| replace:: :abbr:`RVMC (Redfish Virtual Media Controller)`
.. |RWX| replace:: :abbr:`RWX (Read Write Many)`
.. |SAN| replace:: :abbr:`SAN (Subject Alternative Name)`
.. |SANs| replace:: :abbr:`SANs (Subject Alternative Names)`
.. |SAS| replace:: :abbr:`SAS (Serial Attached SCSI)`
@ -221,6 +222,8 @@
.. |UFT| replace:: :abbr:`UFT (Unified Flow Tool)`
.. |UUID| replace:: :abbr:`UUID (Universally Unique Identifier)`
.. |UUIDs| replace:: :abbr:`UUIDs (Universally Unique Identifiers)`
.. |vCPU| replace:: :abbr:`vCPU (Virtual Central Processing Unit)`
.. |vCPUs| replace:: :abbr:`vCPUs (Virtual Central Processing Units)`
.. |VF| replace:: :abbr:`VF (Virtual Function)`
.. |VFIO| replace:: :abbr:`VFIO (Virtual Function I/O)`
.. |VFs| replace:: :abbr:`VFs (Virtual Functions)`
@ -231,6 +234,10 @@
.. |VLANs| replace:: :abbr:`VLANs (Virtual Local Area Networks)`
.. |VM| replace:: :abbr:`VM (Virtual Machine)`
.. |VMs| replace:: :abbr:`VMs (Virtual Machines)`
.. |VMI| replace:: :abbr:`VMI (Virtual Machine Instance)`
.. |VMIs| replace:: :abbr:`VMIs (Virtual Machine Instances)`
.. |VMIM| replace:: :abbr:`VMIM (Virtual Machine Instance Migration)`
.. |VMIRS| replace:: :abbr:`VMIRS (Virtual Machine Instance Replica Set)`
.. |VNC| replace:: :abbr:`VNC (Virtual Network Computing)`
.. |VNF| replace:: :abbr:`VNF (Virtual Network Function)`
.. |VNFs| replace:: :abbr:`VNFs (Virtual Network Functions)`

1
doc/source/spelling_wordlist.txt
View File

@ -1078,6 +1078,7 @@ virsh
virt
virtio
Virtio
virtiofs
virtualization
Virtualization
virtualized