# generic vDPA ## 介绍 vDPA是一种基于VirtIO 半虚拟化技术的提出的数据面加速的框架。 DPU卡呈现的VirtIO类型的存储/网络/文件系统设备通过对接vDPA框架，提供与硬件持平的IO性能；并通过统一上层管理的接口，实现多种设备类型、多种DPU卡的接口统一。 ## 软件架构 ![vdpa framework]() ## 部署教程 目录说明： + **kernel**：generic vdpa基本功能及热迁移依赖的内核代码 + **qemu**: generic vdpa设备基本功能及热迁移依赖的qemu代码 + **libvirt**: generic vdpa设备管理及支持生命周期依赖的libvirt代码 + **doc**: 项目资料等 generic vDPA当前开源社区尚未完全支持，因此需要在开源软件的基础上，打上补丁代码来支持相关功能，主要涉及的有三个开源软件：kernel、qemu、libvirt。 基于kernel-6.6、qemu-8.2.0、libvirt-9.10.0版本进行软件编译以及安装。 #### 内核编译以及安装： ```shell 1. cd kernel 2. wget -O kernel-6.6.tar.gz https://codeload.github.com/torvalds/linux/tar.gz/refs/tags/v6.6 3. tar xvf kernel-6.6.tar.gz & cd linux-6.6 4. for p in $(ls ../*.patch); do patch -p 1 -F 0 < $p; done 5. 打开内核VHOST_VDPA的编译选项 6. make -j64 && make modules_install -j64 && make install ``` #### qemu编译及安装： ```shell 1. cd qemu 2. tar xvf qemu-8.2.0.tar.gz && cd qemu-8.2.0 3. for p in $(ls ../*.patch); do patch -p 1 -F 0 < $p; done 4. ./configure --target-list=aarch64-softmmu --enable-kvm --enable-vhost-vdpa 5. make -j64 && make install ``` #### libvirt编译及安装： ```shell 1. cd libvirt 2. tar xvf libvirt-9.10.0.tar.gz && cd libvirt-9.10.0 3. meson setup build -Dsystem=true -Ddriver_qemu=enabled -Ddriver_lxc=disabled -Dlogin_shell=disabled 4. ninja -C build 5. ninja -C build install ``` #### 操作方法 请使用root用户按照如下操作步骤配置 1. 开启网卡SRIOV模式： 1. 在HostOS的BIOS中开启SMMU的支持，不同厂家服务器的开启可能不同，请参考各服务器的帮助文档。 2. HostOS中对需要配置给虚拟机的PF配置SRIOV，创建VF，以virtio-net设备为例 ```shell # lspci -s 83:00.6 83:00.6 Ethernet controller: Virtio: Virtio network device # echo 16 > /sys/bus/pci/devices/0000\:83\:00.6/sriov_numvfs # lspci | grep Virtio 83:00.6 Ethernet controller: Virtio: Virtio network device 83:01.1 Ethernet controller: Virtio: Virtio network device 83:01.2 Ethernet controller: Virtio: Virtio network device 83:01.3 Ethernet controller: Virtio: Virtio network device 83:01.4 Ethernet controller: Virtio: Virtio network device 83:01.5 Ethernet controller: Virtio: Virtio network device 83:01.6 Ethernet controller: Virtio: Virtio network device 83:01.7 Ethernet controller: Virtio: Virtio network device 83:02.0 Ethernet controller: Virtio: Virtio network device 83:02.1 Ethernet controller: Virtio: Virtio network device 83:02.2 Ethernet controller: Virtio: Virtio network device 83:02.3 Ethernet controller: Virtio: Virtio network device 83:02.4 Ethernet controller: Virtio: Virtio network device 83:02.5 Ethernet controller: Virtio: Virtio network device 83:02.6 Ethernet controller: Virtio: Virtio network device 83:02.7 Ethernet controller: Virtio: Virtio network device ``` 2. 解绑VF驱动，并绑定对应硬件的厂商vdpa驱动 ```shell echo 0000:83:01.1 > /sys/bus/pci/devices/0000\:08\:01.1/driver/unbind echo 0000:83:01.2 > /sys/bus/pci/devices/0000\:08\:01.2/driver/unbind echo 0000:83:01.3 > /sys/bus/pci/devices/0000\:08\:01.3/driver/unbind echo 0000:83:01.4 > /sys/bus/pci/devices/0000\:08\:01.4/driver/unbind echo 0000:83:01.5 > /sys/bus/pci/devices/0000\:08\:01.5/driver/unbind echo -n "1af4 1000" > /sys/bus/pci/drivers/${vendor_vdpa}/new_id ``` 3. 绑定vDPA设备后，可以通过vdpa命令查询vdpa管理设备列表 ```shell # vdpa mgmtdev show pci/0000:83:01.1: supported_classes net pci/0000:83:01.2: supported_classes net pci/0000:83:01.3: supported_classes net pci/0000:83:01.4: supported_classes net pci/0000:83:01.5: supported_classes net ``` 4. 完成vdpa设备的创建后，创建vhost-vDPA设备 ```shell vdpa dev add name vdpa0 mgmtdev pci/0000:83:01.1 vdpa dev add name vdpa1 mgmtdev pci/0000:83:01.2 vdpa dev add name vdpa2 mgmtdev pci/0000:83:01.3 vdpa dev add name vdpa3 mgmtdev pci/0000:83:01.4 vdpa dev add name vdpa4 mgmtdev pci/0000:83:01.5 ``` 5. 完成vhost-vDPA的设备创建后，可以通过vdpa命令查询vdpa设备列表；也可以通过libvirt命令查询环境的vhost-vDPA设备信息 ```shell # vdpa dev show vdpa0: type network mgmtdev pci/0000:83:01.1 vendor_id 6900 max_vqs 3 max_vq_size 256 vdpa1: type network mgmtdev pci/0000:83:01.2 vendor_id 6900 max_vqs 3 max_vq_size 256 vdpa2: type network mgmtdev pci/0000:83:01.3 vendor_id 6900 max_vqs 3 max_vq_size 256 vdpa3: type network mgmtdev pci/0000:83:01.4 vendor_id 6900 max_vqs 3 max_vq_size 256 vdpa4: type network mgmtdev pci/0000:83:01.5 vendor_id 6900 max_vqs 3 max_vq_size 256 # virsh nodedev-list vdpa vdpa_vdpa0 vdpa_vdpa1 vdpa_vdpa2 vdpa_vdpa3 vdpa_vdpa4 # virsh nodedev-dumpxml vdpa_vdpa0 vdpa_vdpa0 /sys/devices/pci0000:00/0000:00:0c.0/0000:83:01.1/vdpa0 pci_0000_83_01_1 vhost_vdpa /dev/vhost-vdpa-0 ``` 6. 创建虚拟机时，在虚拟机配置文件中增加vDPA直通设备的配置项，其中，source dev为host上创建的vhost-vdpa设备的字符设备路径 ```xml ``` 7. 虚拟机创建后，可以在虚拟机内看到对应的virtio-net设备，配置IP后，网卡功能正常 ``` # lspci | grep -i eth 05:00.0 Ethernet controller: Virtio: Virtio network device (rev 01) # ifconfig eth0 192.168.1.100 # ping 192.168.1.200 PING 192.168.1.200 (192.168.1.200) 56(84) bytes of data. 64 bytes from 192.168.1.200: icmp_seq=1 ttl=64 time=0.324 ms 64 bytes from 192.168.1.200: icmp_seq=2 ttl=64 time=0.141 ms --- 192.168.1.200 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.141/0.232/0.324/0.091 ms ``` ## Release 计划 - generic vDPA基本功能实现 - 2024.08.30 - generic vDPA热迁移功能实现 - 2024.10.30 - generic vDPA相关文档输出 - 2024.11.30