191 lines
6.9 KiB
Markdown
191 lines
6.9 KiB
Markdown
# generic vDPA
|
||
|
||
## 介绍
|
||
vDPA是一种基于VirtIO 半虚拟化技术的提出的数据面加速的框架。
|
||
|
||
DPU卡呈现的VirtIO类型的存储/网络/文件系统设备通过对接vDPA框架，提供与硬件持平的IO性能；并通过统一上层管理的接口，实现多种设备类型、多种DPU卡的接口统一。
|
||
|
||
## 软件架构
|
||
|
||
|
||
## 部署教程
|
||
|
||
目录说明：
|
||
|
||
+ **kernel**：generic vdpa基本功能及热迁移依赖的内核代码
|
||
+ **qemu**: generic vdpa设备基本功能及热迁移依赖的qemu代码
|
||
+ **libvirt**: generic vdpa设备管理及支持生命周期依赖的libvirt代码
|
||
+ **doc**: 项目资料等
|
||
|
||
generic vDPA当前开源社区尚未完全支持，因此需要在开源软件的基础上，打上补丁代码来支持相关功能，主要涉及的有三个开源软件：kernel、qemu、libvirt。
|
||
|
||
基于kernel-6.6、qemu-8.2.0、libvirt-9.10.0版本进行软件编译以及安装。
|
||
|
||
#### 内核编译以及安装：
|
||
|
||
```shell
|
||
1. cd kernel
|
||
2. wget -O kernel-6.6.tar.gz https://codeload.github.com/torvalds/linux/tar.gz/refs/tags/v6.6
|
||
3. tar xvf kernel-6.6.tar.gz & cd linux-6.6
|
||
4. for p in $(ls ../*.patch); do patch -p 1 -F 0 < $p; done
|
||
5. 打开内核VHOST_VDPA的编译选项
|
||
6. make -j64 && make modules_install -j64 && make install
|
||
```
|
||
|
||
#### qemu编译及安装：
|
||
|
||
```shell
|
||
1. cd qemu
|
||
2. tar xvf qemu-8.2.0.tar.gz && cd qemu-8.2.0
|
||
3. for p in $(ls ../*.patch); do patch -p 1 -F 0 < $p; done
|
||
4. ./configure --target-list=aarch64-softmmu --enable-kvm --enable-vhost-vdpa
|
||
5. make -j64 && make install
|
||
```
|
||
|
||
#### libvirt编译及安装：
|
||
|
||
```shell
|
||
1. cd libvirt
|
||
2. tar xvf libvirt-9.10.0.tar.gz && cd libvirt-9.10.0
|
||
3. meson setup build -Dsystem=true -Ddriver_qemu=enabled -Ddriver_lxc=disabled -Dlogin_shell=disabled
|
||
4. ninja -C build
|
||
5. ninja -C build install
|
||
```
|
||
|
||
#### 操作方法
|
||
|
||
请使用root用户按照如下操作步骤配置
|
||
|
||
1. 开启网卡SRIOV模式：
|
||
|
||
1. 在HostOS的BIOS中开启SMMU的支持，不同厂家服务器的开启可能不同，请参考各服务器的帮助文档。
|
||
|
||
2. HostOS中对需要配置给虚拟机的PF配置SRIOV，创建VF，以virtio-net设备为例
|
||
|
||
```shell
|
||
# lspci -s 83:00.6
|
||
83:00.6 Ethernet controller: Virtio: Virtio network device
|
||
# echo 16 > /sys/bus/pci/devices/0000\:83\:00.6/sriov_numvfs
|
||
# lspci | grep Virtio
|
||
83:00.6 Ethernet controller: Virtio: Virtio network device
|
||
83:01.1 Ethernet controller: Virtio: Virtio network device
|
||
83:01.2 Ethernet controller: Virtio: Virtio network device
|
||
83:01.3 Ethernet controller: Virtio: Virtio network device
|
||
83:01.4 Ethernet controller: Virtio: Virtio network device
|
||
83:01.5 Ethernet controller: Virtio: Virtio network device
|
||
83:01.6 Ethernet controller: Virtio: Virtio network device
|
||
83:01.7 Ethernet controller: Virtio: Virtio network device
|
||
83:02.0 Ethernet controller: Virtio: Virtio network device
|
||
83:02.1 Ethernet controller: Virtio: Virtio network device
|
||
83:02.2 Ethernet controller: Virtio: Virtio network device
|
||
83:02.3 Ethernet controller: Virtio: Virtio network device
|
||
83:02.4 Ethernet controller: Virtio: Virtio network device
|
||
83:02.5 Ethernet controller: Virtio: Virtio network device
|
||
83:02.6 Ethernet controller: Virtio: Virtio network device
|
||
83:02.7 Ethernet controller: Virtio: Virtio network device
|
||
```
|
||
|
||
2. 解绑VF驱动，并绑定对应硬件的厂商vdpa驱动
|
||
|
||
```shell
|
||
echo 0000:83:01.1 > /sys/bus/pci/devices/0000\:08\:01.1/driver/unbind
|
||
echo 0000:83:01.2 > /sys/bus/pci/devices/0000\:08\:01.2/driver/unbind
|
||
echo 0000:83:01.3 > /sys/bus/pci/devices/0000\:08\:01.3/driver/unbind
|
||
echo 0000:83:01.4 > /sys/bus/pci/devices/0000\:08\:01.4/driver/unbind
|
||
echo 0000:83:01.5 > /sys/bus/pci/devices/0000\:08\:01.5/driver/unbind
|
||
echo -n "1af4 1000" > /sys/bus/pci/drivers/${vendor_vdpa}/new_id
|
||
```
|
||
|
||
3. 绑定vDPA设备后，可以通过vdpa命令查询vdpa管理设备列表
|
||
|
||
```shell
|
||
# vdpa mgmtdev show
|
||
pci/0000:83:01.1:
|
||
supported_classes net
|
||
pci/0000:83:01.2:
|
||
supported_classes net
|
||
pci/0000:83:01.3:
|
||
supported_classes net
|
||
pci/0000:83:01.4:
|
||
supported_classes net
|
||
pci/0000:83:01.5:
|
||
supported_classes net
|
||
```
|
||
|
||
4. 完成vdpa设备的创建后，创建vhost-vDPA设备
|
||
|
||
```shell
|
||
vdpa dev add name vdpa0 mgmtdev pci/0000:83:01.1
|
||
vdpa dev add name vdpa1 mgmtdev pci/0000:83:01.2
|
||
vdpa dev add name vdpa2 mgmtdev pci/0000:83:01.3
|
||
vdpa dev add name vdpa3 mgmtdev pci/0000:83:01.4
|
||
vdpa dev add name vdpa4 mgmtdev pci/0000:83:01.5
|
||
```
|
||
|
||
5. 完成vhost-vDPA的设备创建后，可以通过vdpa命令查询vdpa设备列表；也可以通过libvirt命令查询环境的vhost-vDPA设备信息
|
||
|
||
```shell
|
||
# vdpa dev show
|
||
vdpa0: type network mgmtdev pci/0000:83:01.1 vendor_id 6900 max_vqs 3 max_vq_size 256
|
||
vdpa1: type network mgmtdev pci/0000:83:01.2 vendor_id 6900 max_vqs 3 max_vq_size 256
|
||
vdpa2: type network mgmtdev pci/0000:83:01.3 vendor_id 6900 max_vqs 3 max_vq_size 256
|
||
vdpa3: type network mgmtdev pci/0000:83:01.4 vendor_id 6900 max_vqs 3 max_vq_size 256
|
||
vdpa4: type network mgmtdev pci/0000:83:01.5 vendor_id 6900 max_vqs 3 max_vq_size 256
|
||
|
||
# virsh nodedev-list vdpa
|
||
vdpa_vdpa0
|
||
vdpa_vdpa1
|
||
vdpa_vdpa2
|
||
vdpa_vdpa3
|
||
vdpa_vdpa4
|
||
|
||
# virsh nodedev-dumpxml vdpa_vdpa0
|
||
<device>
|
||
<name>vdpa_vdpa0</name>
|
||
<path>/sys/devices/pci0000:00/0000:00:0c.0/0000:83:01.1/vdpa0</path>
|
||
<parent>pci_0000_83_01_1</parent>
|
||
<driver>
|
||
<name>vhost_vdpa</name>
|
||
</driver>
|
||
<capability type='vdpa'>
|
||
<chardev>/dev/vhost-vdpa-0</chardev>
|
||
</capability>
|
||
</device>
|
||
```
|
||
|
||
6. 创建虚拟机时，在虚拟机配置文件中增加vDPA直通设备的配置项，其中，source dev为host上创建的vhost-vdpa设备的字符设备路径
|
||
|
||
```xml
|
||
<devices>
|
||
<hostdev mode='subsystem' type='vdpa'>
|
||
<source dev='/dev/vhost-vdpa-0'/>
|
||
</hostdev>
|
||
</devices>
|
||
```
|
||
|
||
7. 虚拟机创建后，可以在虚拟机内看到对应的virtio-net设备，配置IP后，网卡功能正常
|
||
|
||
```
|
||
# lspci | grep -i eth
|
||
05:00.0 Ethernet controller: Virtio: Virtio network device (rev 01)
|
||
|
||
# ifconfig eth0 192.168.1.100
|
||
# ping 192.168.1.200
|
||
PING 192.168.1.200 (192.168.1.200) 56(84) bytes of data.
|
||
64 bytes from 192.168.1.200: icmp_seq=1 ttl=64 time=0.324 ms
|
||
64 bytes from 192.168.1.200: icmp_seq=2 ttl=64 time=0.141 ms
|
||
|
||
--- 192.168.1.200 ping statistics ---
|
||
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
|
||
rtt min/avg/max/mdev = 0.141/0.232/0.324/0.091 ms
|
||
```
|
||
|
||
|
||
|
||
## Release 计划
|
||
- generic vDPA基本功能实现 - 2024.08.30
|
||
- generic vDPA热迁移功能实现 - 2024.10.30
|
||
- generic vDPA相关文档输出 - 2024.11.30
|
||
|
||
|