diff --git a/doc/source/configuration/index.rst b/doc/source/configuration/index.rst index 813d05182..5fb6e1cc7 100644 --- a/doc/source/configuration/index.rst +++ b/doc/source/configuration/index.rst @@ -13,6 +13,7 @@ General Configuration cert_config host_config host_interface_network_config + intel_n3000_fpga docker_proxy_config system_config time_sync_config diff --git a/doc/source/configuration/intel_n3000_fpga.rst b/doc/source/configuration/intel_n3000_fpga.rst new file mode 100644 index 000000000..787979e5b --- /dev/null +++ b/doc/source/configuration/intel_n3000_fpga.rst @@ -0,0 +1,330 @@ +===================================== +Host FPGA Configuration - Intel N3000 +===================================== + +This guide describes how to configure and integrate an Intel N3000 +:abbr:`FPGA (Field Programmable Gate Array)` :abbr:`PAC (Programmable Acceleration Card)` +on StarlingX. + +.. contents:: + :local: + :depth: 1 + +-------- +Overview +-------- + +The `Intel FPGA PAC N3000 `_ contains two Intel +XL710 NICs, memory, and an Intel FPGA. The system discovers and inventories the +device as a NIC, with the XL710 ports available in the host port list and host +interface list. + +-------------------- +Update device images +-------------------- + +The Intel FPGA PAC N3000 as shipped from the factory is expected to have +production BMC and factory images. The following procedure describes how to +update the user image on a host. + +Device image types: + +* root-key: The root-key image sets the main authentication key on the + hardware. + +* functional-key: The functional device image performs the desired work on + behalf of the application. If a rootkey device image has been written to the + hardware, then the functional image will only be accepted if it has been + signed by a :abbr:`CSK (code-signing key)` generated from the root key which has + not been revoked. + +* key-revocation: The key-revocation device image will revoke a CSK. If a + root-key device image has been written to the hardware, then the + key-revocation device image will only be accepted if it has been signed by + the root key. + +The following items are specific to the Intel FPGA PAC N3000: + +* The root-key image is called the *root entry hash bitstream* and can only be + set once. +* The functional device image is known as the *user image*. +* The key-revocation device image is known as the *CSK ID cancellation + bitstream*. +* CSKs are revoked by specifying an integer CSK ID. +* 128 CSK ID cancellation slots exist. + + +For the Intel FPGA PAC N3000, a CSK is revoked by specifying an integer ID, and +all CSKs with that ID will be revoked. Writing the root-key device image or a +key-revocation device image is essentially permanent. Reverting to factory +status requires physical access to the card and specialized equipment. + +#. Upload the device image. + + To upload a root-key device image: + + :: + + ~(keystone_admin)$ system device-image-upload imagefile root-key pci_vendor + pci_device --key-signature key_signature --name imagename --description + description --image-version version + + To upload a functional device image: + + :: + + ~(keystone_admin)$ system device-image-upload imagefile functional pci_vendor + pci_device --functional bitstream_id --name imagename --description description + -image-version version + + To upload a revocation key device image: + + :: + + ~(keystone_admin)$ system device-image-upload imagefile key-revocation + pci_vendor pci_device --revoke-key-id revoke_key_id --name imagename -- + description description --image-version version + + where: + + :: + + imagefile # The filepath of the binary device image file. + pci_vendor # The hexadecimal string identifying the PCI vendor ID of the device this image applies to. + pci_device # The hexadecimal string identifying the PCI device ID of the device this image applies to. + key_signature # A hexadecimal string identifying the root key device image. + revoke_key_id # A decimal key ID for the key revocation device image. + bitstream_id # A hexadecimal string of the functional device image. + name # The name of the device image (optional). + description # The description of the device image (optional). + image-version # The version of the device image (optional). + +#. Assign a device label to the device. + + Labels are key-value pairs that are assigned to host PCI devices and are + used to specify attributes of the devices. Labels can be used to identify + certain properties of the PCI devices where the same device image can be + used. + + The command syntax is: + + :: + + system host-device-label-assign [--overwrite] hostname_or_id pci_name_or_address + name=value [name=value ...] + + Overwrite the label using the ``--overwrite`` option. This option + is not allowed while the image update is in progress after running + ``host-device-image-update``. Once assigned, a device label can be + referenced by multiple ``device-image-apply`` commands. + +#. Apply the device image on one or all supported devices. + + .. note:: + + A device firmware update in progress alarm is raised once the first + device image is applied. + + The ``system device-image-state-list`` will show the status of the device as + pending. + + * Apply a device image to all supported devices: + + :: + + ~(keystone_admin)$ system device-image-apply image_uuid + + * Alternatively, apply a device image to devices with a specified label: + + :: + + ~(keystone_admin)$ system device-image-apply image_uuid key1=value1 + +#. Write pending device images on the host to hardware. + + :: + + ~(keystone)admin)$ system host-device-image-update hostname + + .. note:: + + This operation currently supports one pending device image at a time. + + Any previously-attempted device image writes for this host that are in a + failed state will be reset to pending and retried. + + Root and revocation key updates can be expected to take 1-2 minutes. + + Functional image updates can take approximately 40 minutes for the Intel + FPGA PAC N3000. + + * Once a device update is complete, ``system device-image-state-list`` + will show the status as completed for that device/image. + + * Once all pending device updates for the host are complete, + ``system host-show hostname`` will again display an empty string for + ``device_image_update``. + +#. Lock and unlock the host. + +#. (Optional) Upload, apply, and update any additional key-revocation device + images or functional device images as needed. + + New device images can be uploaded as needed, and already-uploaded images can + be applied with new labels. Devices can also have new labels applied to + them, and any device images with matching labels will be automatically + applied. + +-------------------------- +Device management commands +-------------------------- + +This section lists the commands used to control the Intel FPGA PAC N3000. + +:: + + Listing uploaded device images + system device-image-list + + Listing device labels + system host-device-label-list hostname_or_id devicename_or_address + + Removing device labels + system host-device-label-remove hostname_or_id key + + Remove a device image + To remove a device image from all devices + system device-image-remove image_uuid + To remove the device image from all devices with a matching label + system device-image-remove image_uuid key1=value1 + + Initiating a Device Image Update for a Host + system host-device-image-update hostname_or_host_ID + + Displaying the status of device images + system device-image-state-list + +------------------------------- +Enable Forward Error Correction +------------------------------- + +The Intel FPGA PAC N3000 supports :abbr:`FEC (forward error correction)` +capabilities, which are exposed as a PCI device. The PCI device can be used by a +`DPDK `_ enabled container application to perform accelerated 5G LDPC encoding and +decoding operations. + +After the FPGA device is programmed, the list of host devices shows the FEC +device with device ID 0xd8f, as shown below. + +:: + + system host-device-list + + +------------------+--------------+----------+-----------+-----------+---------------------------+-------------------------+-------------------------------------+-----------+---------+ + | name | address | class id | vendor id | device id | class name | vendor name | device name | numa_node | enabled | + +------------------+--------------+----------+-----------+-----------+---------------------------+-------------------------+-------------------------------------+-----------+---------+ + | pci_0000_b7_00_0 | 0000:b7:00.0 | 120000 | 8086 | 0d8f | Processing accelerators | Intel Corporation | Device 0d8f | 1 | True | + +------------------+--------------+----------+-----------+-----------+---------------------------+-------------------------+-------------------------------------+-----------+---------+ + ... + +To enable the FEC device for SR-IOV, set the number of virtual functions (VFs) +and set the appropriate userspace drivers for the physical function (PF) and VF. + +For example: + +:: + + system host-lock + system host-device-modify --driver --vf-driver -N + system host-unlock + + The supported PF driver(s) are: + - igb_uio + + The supported VF driver(s) are currently: + - igb_uio + - vfio + +To pass the FEC device to a container, enter the following requests/limits +into the pod specification: + +:: + + intel.com/intel_fpga_fec: '' + +For example: + +:: + + apiVersion: v1 + kind: Pod + metadata: + name: 5gnr + annotations: + k8s.v1.cni.cncf.io/networks: '[ + { "name": "sriov1" } + ]' + spec: + restartPolicy: Never + containers: + - name: 5gnr + image: "5gnr-image" + volumeMounts: + - mountPath: /mnt/huge-1048576kB + name: hugepage + stdin: true + tty: true + resources: + requests: + memory: 4Gi + intel.com/intel_fpga_fec: '1' + intel.com/pci_sriov_net_datanetwork_a: '1' + limits: + hugepages-1Gi: 4Gi + memory: 4Gi + intel.com/intel_fpga_fec: '1' + intel.com/pci_sriov_net_datanetwork_a: '1' + volumes: + - name: hugepage + emptyDir: + medium: HugePages + +------------------------- +Configure NICs for SR-IOV +------------------------- + +You can configure the Intel XL710 NICs for SR-IOV by first identifying the NICs +on the Intel FPGA PAC N3000 using the following command: + +:: + + system host-port-list + + +--------------------------------------+------------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ + | uuid | name | type | pci address | device | processor | accelerated | device type | + +--------------------------------------+------------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ + | 6c79c0d0-0463-4551-a19a-24d52a9403c6 | enp177s0f0 | ethernet | 0000:b1:00.0 | 0 | 1 | False | Device [0d58] | + +--------------------------------------+------------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ + ... + +Next, set the number of virtual functions (VFs) and set the appropriate +userspace drivers for the VF. + +For example: + +:: + + system host-lock + system host-if-list -a + system host-if-modify -c pci-sriov --vf-driver -N + system interface-datanetwork-assign + system host-unlock + + The supported VF driver(s) are currently: + - vfio + - netdevice + +.. note:: + + If ``--vf-driver`` is not specified in the ``system host-if-modify`` + command, then ``netdevice`` (kernel driver) will be assigned.