starlingx/docs
Code Issues Proposed changes

Merge "Editorial updates on TSN Kata Containers guide"

Browse Source
This commit is contained in:
Zuul 2020-06-11 15:15:30 +00:00 committed by Gerrit Code Review
commit ba525b9e68
1 changed files with 182 additions and 181 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

363
doc/source/developer_resources/stx_tsn_in_kata.rst
View File

@ -10,62 +10,62 @@ Enable TSN in Kata Containers
Background Background
---------- ----------
`Time sensitive networking (TSN) <https://1.ieee802.org/tsn/>`_ is a set of `Time Sensitive Networking (TSN) <https://1.ieee802.org/tsn/>`_ is a set of
standards developed by the IEEE 802.1 Working Group (WG) with the aim of standards developed by the IEEE 802.1 Working Group (WG) with the aim of
guaranteeing determinism in delivering time-sensitive traffic with low and guaranteeing determinism in delivering time-sensitive traffic with low and
bounded latency, while allowing non-time-sensitive traffic to be carried through bounded latency, while allowing non-time-sensitive traffic to be carried through
the same network. the same network.
As a cloud infrastructure software stack for the edge, TSN is a very important As a cloud infrastructure software stack for the edge,
feature for StarlingX as the deterministic low latency is required by edge :abbr:`TSN (Time Sensitive Networking)` is a very important feature for
applications in industrial IOT, video delivery and other ultra-low latency use StarlingX as the deterministic low latency is required by edge applications in
cases. Furthermore, TSN support in containers naturally becomes a demand for industrial IOT, video delivery and other ultra-low latency use cases.
StarlingX as StarlingX is a cloud-native platform. Furthermore, because StarlingX is a cloud-native platform, TSN support in
containers naturally becomes a requirement for StarlingX.
The challenge is some TSN features are only available on Linux from 4.19 or higher A challenge is that some TSN features are only available on Linux kernel
version. StarlingX is built on top of CentOS. As of StarlingX 3.0 release, it is version 4.19 or higher. StarlingX is built on top of CentOS. The StarlingX 3.0
based on CentOS 7 which provides a Linux 3.10 kernel with some backported patches release (currently available) is based on CentOS 7, which provides a Linux 3.10
(StarlingX CentOS 8 upgrade would be finished by 5.0 release as planned). As kernel with some backported patches. (The StarlingX team plans to
we all know, generic containers share the same kernel with the host. upgrade to CentOS 8 in a future release.)
Fortunately, StarlingX already supported Kata containers on the master branch, Generic containers share the same kernel as the host. However, a Kata Container
and it will become an officially supported feature from StarlingX 4.0 release. has its own kernel, which does not depend on the kernel of the host.
A Kata container has its own kernel which doesn't depend on the kernel of the Fortunately, StarlingX already supports Kata Containers on the master branch,
host. Therefore, TSN support in containers on StarlingX starts from Kata and it will become an officially supported feature in the StarlingX 4.0 release.
container. Therefore, TSN support in containers on StarlingX begins with Kata Containers.
`Kata Containers <https://katacontainers.io/>`_ is an open source project to `Kata Containers <https://katacontainers.io/>`_ is an open source project to
build a secure container runtime with lightweight virtual machines that feel build a secure container runtime with lightweight virtual machines that feel
and perform like containers but provide stronger workload isolation using and perform like containers, but provide stronger workload isolation using
hardware virtualization technology as a second layer of defense. hardware virtualization technology as a second layer of defense.
----------------------------------------------------- ---------------------------------------------------------
How to build a Linux kernel with TSN support for Kata Build a Linux kernel with TSN support for Kata Containers
----------------------------------------------------- ---------------------------------------------------------
As of writing this article, the latest Kata release is Kata 1.11.0-rc0. This As of writing this article, the latest Kata release is Kata 1.11.0-rc0. This
release includes a Linux 5.4.32 kernel image with Kata patches. Though the release includes a Linux 5.4.32 kernel image with Kata patches. Though the
kernel version is high enough, TSN features are not fully enabled in the kernel kernel version is high enough, TSN features are not fully enabled in the kernel
build. It is needed to build a customized kernel image based on the same build, so you must build a customized kernel image. The following steps describe
code base. Below is a guide about how to build a customized kernel image for how to build a customized kernel image for Kata.
Kata.
#. Get the ``packaging`` repository of Kata #. Get the ``packaging`` repository of Kata Containers:
:: ::
git clone https://github.com/kata-containers/packaging.git git clone https://github.com/kata-containers/packaging.git
#. Prepare the building environment by executing the command in the directory #. Prepare the build environment by executing this command in the directory
``packaging/kernel``. ``packaging/kernel``:
:: ::
./build-kernel.sh -v 5.4.32 -f -d setup ./build-kernel.sh -v 5.4.32 -f -d setup
#. Prepare a kernel config file ``stx.conf`` with TSN-related options enabled #. Prepare a kernel configuration file, ``stx.conf``, with TSN-related options
as shown below, and put it in the directory. enabled as shown below. Put it in the directory
``~/go/src/github.com/kata-containers/packaging/kernel/configs/fragments/x86_64``. ``~/go/src/github.com/kata-containers/packaging/kernel/configs/fragments/x86_64``:
:: ::
@ -81,43 +81,42 @@ Kata.
# vlan # vlan
CONFIG_VLAN_8021Q=y CONFIG_VLAN_8021Q=y
#. Re-run setup command to update stx.conf to the config #. Re-run the setup command to update ``stx.conf`` to the desired configuration:
:: ::
./build-kernel.sh -v 5.4.32 -f -d setup ./build-kernel.sh -v 5.4.32 -f -d setup
#. Build the kernel image #. Build the kernel image:
:: ::
./build-kernel.sh -v 5.4.32 -f -d build ./build-kernel.sh -v 5.4.32 -f -d build
#. Install the built kernel images to the destination directory #. Install the built kernel images to the destination directory:
:: ::
sudo ./build-kernel.sh -v 5.4.32 -f -d install sudo ./build-kernel.sh -v 5.4.32 -f -d install
Once these commands are done, there are two built kernel images When these commands are done, two built kernel images, ``vmlinux.container``
``vmlinux.container`` and ``vmlinuz.container`` available in the directory and ``vmlinuz.container``, will be available in the directory
``/usr/share/kata-containers``. Save the two files for later use. ``/usr/share/kata-containers``. Save the two files for later use.
--------------------------------------------- --------------------------------------
How to build a container image with TSN stack Build a container image with TSN stack
--------------------------------------------- --------------------------------------
Besides TSN support in the kernel, there are still some packages required to Certain packages are required to build the TSN stack. For example,
build TSN stack. For example, `LinuxPTP <http://linuxptp.sourceforge.net/>`_ is an implementation of the
`LinuxPTP <http://linuxptp.sourceforge.net/>`_ is an implementation of `Precision Time Protocol (PTP) <https://en.wikipedia.org/wiki/Precision_Time_Protocol>`_
`the Precision Time Protocol (PTP)
<https://en.wikipedia.org/wiki/Precision_Time_Protocol>`_
according to IEEE standard 1588 for Linux. according to IEEE standard 1588 for Linux.
Here is the dockerfile used to build the container image. ``Ubuntu 20.04`` was The example below shows the dockerfile used to build the container image.
chosen as the base image for packages with newer versions. ``python3-dateutil``, ``Ubuntu 20.04`` was chosen as the base image for packages with newer versions.
``python3-numpy``, and ``python3-matplotlib`` were installed for performance ``python3-dateutil``, ``python3-numpy``, and ``python3-matplotlib`` were
testing. Here the built container image was named as ``kata_tsn_image``. installed for performance testing. The built container image was named
``kata_tsn_image``.
:: ::
@ -127,12 +126,12 @@ testing. Here the built container image was named as ``kata_tsn_image``.
linuxptp vlan libjansson4 python3-dateutil python3-numpy \ linuxptp vlan libjansson4 python3-dateutil python3-numpy \
python3-matplotlib python3-matplotlib
----------------------------------------- ----------------------------------
How to set up an experimental TSN network Set up an experimental TSN network
----------------------------------------- ----------------------------------
An experimental TSN network as shown in `Figure 1` was set up to verify the TSN The experimental TSN network shown in `Figure 1` was used to verify the TSN
functionality in Kata containers. The network is composed of a switch with functionality in Kata Containers. The network was composed of a switch with
TSN capability and four hosts. TSN capability and four hosts.
.. figure:: ./figures/stx_tsn_network_diagram.png .. figure:: ./figures/stx_tsn_network_diagram.png
@ -140,63 +139,60 @@ TSN capability and four hosts.
:height: 300px :height: 300px
:align: center :align: center
Figure 1: An Experimental TSN Network *Figure 1: Experimental TSN network*
#. The TSN switch is made by a generic PC with a TSN switch card #. The TSN switch used a generic PC with a TSN switch card
`PCIe-0400-TSN <https://www.kontron.com/products/systems/tsn-switches/ `PCIe-0400-TSN <https://www.kontron.com/products/systems/tsn-switches/
network-interfaces-tsn/pcie-0400-tsn-network-interface-card.html>`_ inserted. network-interfaces-tsn/pcie-0400-tsn-network-interface-card.html>`_ inserted.
Please refer to Please refer to the
`the User Guide of PCIe-0400-TSN `PCIe-0400-TSN User Guide
<https://www.kontron.com/downloads/manuals/ <https://www.kontron.com/downloads/manuals/
userguide_pcie-0400-tsn_v0.13.pdf?product=151637>`_ userguide_pcie-0400-tsn_v0.13.pdf?product=151637>`_
for detailed configurations. for detailed configuration options.
#. The hosts are four #. The hosts were four
`Intel Hades Canyon NUC <https://simplynuc.com/hades-canyon/>`_ `Intel Hades Canyon NUC <https://simplynuc.com/hades-canyon/>`_
which are equipped with two NICs each. One of the two NICs is which were equipped with two NICs each. One of the two NICs was the
`Intel I210 NIC <https://ark.intel.com/content/www/us/en/ark/products/series/ `Intel Ethernet Controller I210 series <https://ark.intel.com/content/www/us/en/ark/products/series/64399/intel-ethernet-controller-i210-series.html>`_
64399/intel-ethernet-controller-i210-series.html>`_ which had TSN support.
which has TSN support.
* ``Node 1`` is the latest StarlingX built from the master branch which * ``Node 1`` used the latest StarlingX built from the master branch which
supports Kata containers. ``Node 1`` will be used as the data sender in the supports Kata containers. ``Node 1`` was used as the data sender in the
later test. tests in this guide.
* ``Node 2``, ``Node 3``, and ``Node 4`` were all installed with * ``Node 2``, ``Node 3``, and ``Node 4`` were all installed with
`Ubuntu 18.04`. ``Node 2`` additionally installed ``LinuxPTP`` which will be Ubuntu 18.04. ``Node 2`` additionally installed ``LinuxPTP`` which was
used as the data receiver. ``Node 3`` and ``Node 4`` will be used to used as the data receiver. ``Node 3`` and ``Node 4`` were used to
send/receive best-effort traffic to stress the TSN network. send/receive best-effort traffic to stress the TSN network.
------------------------------------------ -----------------------------------
How to enable and verify TSN functionality Enable and verify TSN functionality
------------------------------------------ -----------------------------------
Till now, the preparation is done. It is time to enable and verify the TSN Preparation is complete and you can enable and verify the TSN functionality in
functionality in Kata containers. The whole process can be summarized as three Kata Containers. The whole process can be summarized in three steps:
steps:
#. Perform the time synchronization across the whole TSN network. #. Perform time synchronization across the whole TSN network.
#. Create a Kata container with Intel I210 passed into. #. Launch a Kata Container with a physical NIC passed in.
#. Make necessary configurations on the Kata container and the TSN switch to #. Make necessary configuration changes to the Kata Container and the TSN switch
enable TSN functionality. After that, run some tests to verify the TSN to enable TSN functionality. After that, run tests to verify the TSN
functionality. functionality.
*********************************************************** ***********************************************************
Step 1. Perform time synchronization across the TSN network Step 1. Perform time synchronization across the TSN network
*********************************************************** ***********************************************************
Two programs, ``ptp4l`` and ``phc2sys`` coming from the project ``LinuxPTP`` Two programs from the ``LinuxPTP`` project, ``ptp4l`` and ``phc2sys``,
were used to do the job. Here is how the time synchronization was performed on were used to do time synchronization on the TSN network.
the TSN network.
.. figure:: ./figures/time_sync_topology.png .. figure:: ./figures/time_sync_topology.png
:width: 500px :width: 500px
:height: 300px :height: 300px
:align: center :align: center
Figure 2: Time Synchronization Topology *Figure 2: Time synchronization topology*
#. Configure NTP servers on the TSN switch and ``Node 1 (StarlingX)`` to #. Configure NTP servers on the TSN switch and ``Node 1 (StarlingX)`` to
synchronize their system clocks with the external clock. synchronize their system clocks with the external clock.
@ -211,66 +207,68 @@ the TSN network.
#. Launch ``phc2sys`` on ``Node 2 (Ubuntu)`` to synchronize its system clock #. Launch ``phc2sys`` on ``Node 2 (Ubuntu)`` to synchronize its system clock
with its PTP clock. with its PTP clock.
The time synchronization on the Kata container will be deferred to ``Step 3``. Time synchronization on the Kata Container is done later in this process.
There is no need to do the time synchronization on ``Node 3`` and ``Node 4`` You do not need to set up time synchronization on ``Node 3`` and ``Node 4``
since they are only used to send/receive best-effort traffic in the experiment. since they were used to send/receive best-effort traffic in the experiment.
***************************************************** *************************************************************
Step 2. Launch a Kata container with I210 passed into Step 2. Launch a Kata Container with a physical NIC passed in
***************************************************** *************************************************************
Before creating a Kata container, the two kernel images ``vmlinux.container`` Before creating a Kata Container, copy the two kernel images ``vmlinux.container``
and ``vmlinuz.container`` should be copied to the directory and ``vmlinuz.container`` to the directory
``/usr/share/kata-containers/`` of ``Node 1 (StarlingX)``. ``/usr/share/kata-containers/`` of ``Node 1 (StarlingX)``.
The I210 NIC on the host needs to be passed into a Kata container. Here is The Intel Ethernet Controller I210 on the host must be passed into a Kata
how to achieve it. More details can be found at Container by completing the following steps. More details can be found at
"`How To Pass a Physical NIC Into a Kata Container `How To Pass a Physical NIC Into a Kata Container
<https://github.com/kata-containers/documentation/pull/619/files>`_" <https://github.com/kata-containers/documentation/pull/619/files>`_.
::
1. Find the pci address of the I210 NIC. Here the pci address is #. Configure the Kata Container:
"0000:05:00.0" and the ID is "8086:157b" which are used in the
following steps.
lspci -nn -D | grep Ethernet
0000:00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-LM [8086:15b7] (rev 31)
0000:05:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:157b] (rev 03)
2. export BDF="0000:05:00.0" ::
3. readlink -e /sys/bus/pci/devices/$BDF/iommu_group # Find the PCI address of the I210 NIC. Here the PCI address is
/sys/kernel/iommu_groups/16 # "0000:05:00.0" and the ID is "8086:157b" which are used in the
# following steps.
lspci -nn -D | grep Ethernet
0000:00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-LM [8086:15b7] (rev 31)
0000:05:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:157b] (rev 03)
4. echo $BDF | sudo tee /sys/bus/pci/devices/$BDF/driver/unbind export BDF="0000:05:00.0"
5. sudo modprobe vfio-pci readlink -e /sys/bus/pci/devices/$BDF/iommu_group
/sys/kernel/iommu_groups/16
6. echo 8086 157b | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id echo $BDF | sudo tee /sys/bus/pci/devices/$BDF/driver/unbind
7. echo $BDF | sudo tee --append /sys/bus/pci/drivers/vfio-pci/bind sudo modprobe vfio-pci
8. ls -l /dev/vfio echo 8086 157b | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
total 0
crw------- 1 root root 241, 0 May 18 15:38 16
crw-rw-rw- 1 root root 10, 196 May 18 15:37 vfio
9. Edit "the file /usr/share/defaults/kata-containers/configuration.toml" to echo $BDF | sudo tee --append /sys/bus/pci/drivers/vfio-pci/bind
set "hotplug_vfio_on_root_bus" to true.
Once these configurations are done, a Kata container can be created with the ls -l /dev/vfio
I210 NIC passed into. Assume the name of the container image is total 0
``kata_tsn_image``. crw------- 1 root root 241, 0 May 18 15:38 16
crw-rw-rw- 1 root root 10, 196 May 18 15:37 vfio
:: # Edit the /usr/share/defaults/kata-containers/configuration.toml file to
# set `hotplug_vfio_on_root_bus` to true.
sudo docker run -it -d --runtime=kata-runtime --rm --device \ #. Create a Kata Container with the Intel Ethernet Controller I210 passed in.
/dev/vfio/16 -v /dev:/dev --privileged --name tsn \ In this example, the name of the container image was ``kata_tsn_image``.
kata_tsn_image /bin/bash
Once it is done, the I210 NIC can be seen in the created container with the name ::
``eth1``.
sudo docker run -it -d --runtime=kata-runtime --rm --device \
/dev/vfio/16 -v /dev:/dev --privileged --name tsn \
kata_tsn_image /bin/bash
When completed, the I210 NIC was seen in the created container with the name
``eth1``.
*************************************** ***************************************
Step 3. Config and test TSN performance Step 3. Config and test TSN performance
@ -281,15 +279,15 @@ The sample application
<https://github.com/intel/iotg_tsn_ref_sw/tree/apollolake-i/sample-app-taprio>`_ <https://github.com/intel/iotg_tsn_ref_sw/tree/apollolake-i/sample-app-taprio>`_
was used in the test. Minor changes were made on the code to format the was used in the test. Minor changes were made on the code to format the
output to adapt to the two tools (``nl-calc`` and ``nl-report``) provided by output to adapt to the two tools (``nl-calc`` and ``nl-report``) provided by
the project the
`netlatency <https://github.com/kontron/netlatency>`_ to plot the result. `netlatency <https://github.com/kontron/netlatency>`_ project and plot the result.
Three test cases were defined in the experiment. Among the three test cases, Three test cases were defined in the experiment. For all three test cases,
``sample-app-taprio`` was running in the Kata container as the data sender and ``sample-app-taprio`` was running in the Kata Container as the data sender and
running on ``Node 2`` as the data receiver. Common configurations for running on ``Node 2`` as the data receiver. Common configurations for
``sample-app-taprio`` are listed here. ``sample-app-taprio`` are listed here.
.. csv-table:: Table 1: Common Configurations for sample-app-taprio .. csv-table:: Table 1: Configuration of sample-app-taprio
:header: "Option", "Value" :header: "Option", "Value"
"Cycle Time", "2ms" "Cycle Time", "2ms"
@ -298,24 +296,25 @@ running on ``Node 2`` as the data receiver. Common configurations for
"VLAN Priority code point", "6" "VLAN Priority code point", "6"
"SO_PRIORITY", "6" "SO_PRIORITY", "6"
In the test, three performance indicators were measured. During the test, three performance indicators were measured.
.. csv-table:: Table 2: Performance Indicators .. csv-table:: Table 2: Performance indicators
:header: "Indicator", "Meaning" :header: "Indicator", "Meaning"
"Scheduled times", "the time from the beginning of a cycle to the NIC of the receiver receives the packet" "Scheduled times", "Time from the beginning of a cycle to when the NIC receives the packet"
"RT application latency", "the time from the beginning of a cycle to when calling the send function" "RT application latency", "Time from the beginning of a cycle to when the send function is called"
"TSN Network jitter", "the jitter of scheduled times" "TSN Network jitter", "Jitter of scheduled times"
* Case 1, no TSN feature enabled. ``sample-app-taprio`` sends a packet at the * Case 1: TSN not enabled.
beginning of each cycle.
Need to perform time synchronization on the Kata container before executing ``sample-app-taprio`` sent a packet at the beginning of each cycle.
``sample-app-taprio``.
Before ``sample-app-taprio`` was executed, time synchronization was performed
on the Kata Container.
:: ::
# launch ptp programs, ptp4l and phc2sys, to synchronize the PTP clock and # Launch PTP programs, ptp4l and phc2sys, to synchronize the PTP clock and
# the system clock. # the system clock.
ptp4l -f /etc/ptp4l.cfg -m & ptp4l -f /etc/ptp4l.cfg -m &
phc2sys -s eth1 -c CLOCK_REALTIME -w -O 0 -m & phc2sys -s eth1 -c CLOCK_REALTIME -w -O 0 -m &
@ -347,50 +346,50 @@ In the test, three performance indicators were measured.
:height: 400px :height: 400px
:align: center :align: center
Figure 3: Performance Report of Case 1 *Figure 3: Case 1 performance report*
As shown in `Figure 3`, the indicator of ``RT application latency`` ranged from As shown in `Figure 3`, the ``RT application latency`` indicator ranged from
`28.184us` to `1259.387us`. There are two reasons for that: 28.184us to 1259.387us, due to the following reasons:
#. Standard kernels instead of real-time kernels were used for both StarlingX #. Standard kernels instead of real-time kernels were used for both StarlingX
platform and the Kata container (for now, Kata containers only supports platform and the Kata Container. (Kata Containers supports the standard
standard kernel). kernel.)
#. ``sample-app-taprio`` was running on the Kata container instead of the #. ``sample-app-taprio`` was running on the Kata Container instead of the
host. host.
Since TSN features were not enabled in this case, there are no any controls on Since TSN features were not enabled, there were no controls on
``Scheduled times``. It depended on the indicator of ``Scheduled times``, and its behavior depended on the ``RT application latency``
``RT application latency`` and the behavior of the whole network. As shown in indicator and the behavior of the whole network. As shown in
the figure, it ranged from `69.824us` to `2487.357us`, and its jitter can reach the figure, it ranged from 69.824us to 2487.357us, and the measured jitter
`1ms`. reached 1ms.
* Case 2, * Case 2: Enable two qdiscs on the Kata Container.
enable two qdiscs,
`TAPRIO <http://man7.org/linux/man-pages/man8/tc-taprio.8.html>`_ and
`ETF <http://man7.org/linux/man-pages/man8/tc-etf.8.html>`_, on the Kata
container. `sample-app-taprio` had additional configurations as shown
in Table 3. Considering the big variance of ``RT application latency`` got in
`Case 1`, the transmitting time was set at `1250us`.
.. csv-table:: Table 3: Additional Configurations for Case 2 `TAPRIO <http://man7.org/linux/man-pages/man8/tc-taprio.8.html>`_ and
`ETF <http://man7.org/linux/man-pages/man8/tc-etf.8.html>`_ were used.
``sample-app-taprio`` had additional configuration settings as shown
in *Table 3*. Considering the large variance of ``RT application latency`` in
Case 1, the transmitting time was set at 1250us.
.. csv-table:: Table 3: Case 2 configuration
:header: "Option", "Value" :header: "Option", "Value"
"Transmit Window", "[1200us, 1300us]" "Transmit Window", "[1200us, 1300us]"
"Offset in Window", "50us" "Offset in Window", "50us"
Make necessary configurations on the Kata container before executing Make necessary configuration changes on the Kata Container before executing
``sample-app-taprio``. ``sample-app-taprio``.
:: ::
# change the number of multi-purpose channels # Change the number of multi-purpose channels
ethtool -L eth1 combined 4 ethtool -L eth1 combined 4
# delete existing qdiscs # Delete existing qdiscs
tc qdisc del dev eth1 root tc qdisc del dev eth1 root
# enable taprio qdisc, SO_PRIORITY 6 was mapped to traffic class 1. # Enable taprio qdisc, SO_PRIORITY 6 was mapped to traffic class 1.
tc -d qdisc replace dev eth1 parent root handle 100 taprio num_tc 4 \ tc -d qdisc replace dev eth1 parent root handle 100 taprio num_tc 4 \
map 3 3 3 3 3 3 1 3 3 3 3 3 3 3 3 3 \ map 3 3 3 3 3 3 1 3 3 3 3 3 3 3 3 3 \
queues 1@0 1@1 1@2 1@3 \ queues 1@0 1@1 1@2 1@3 \
@ -405,17 +404,17 @@ Make necessary configurations on the Kata container before executing
sched-entry S 08 100000 \ sched-entry S 08 100000 \
clockid CLOCK_TAI clockid CLOCK_TAI
# enable etf qdisc on queue 1 which corresponds to traffic class 1 # Enable etf qdisc on queue 1 which corresponds to traffic class 1
tc qdisc replace dev eth1 parent 100:2 etf clockid CLOCK_TAI \ tc qdisc replace dev eth1 parent 100:2 etf clockid CLOCK_TAI \
delta 5000000 offload delta 5000000 offload
# create vlan interface and set egress map. # Create vlan interface and set egress map.
ip link add link eth1 name eth1.3 type vlan id 3 ip link add link eth1 name eth1.3 type vlan id 3
vconfig set_egress_map eth1.3 6 6 vconfig set_egress_map eth1.3 6 6
ifconfig eth1 up ifconfig eth1 up
ip link set eth1.3 up ip link set eth1.3 up
# launch ptp programs, ptp4l and phc2sys, to synchronize the PTP clock and # Launch PTP programs, ptp4l and phc2sys, to synchronize the PTP clock and
# the system clock. # the system clock.
ptp4l -f /etc/ptp4l.cfg -m & ptp4l -f /etc/ptp4l.cfg -m &
phc2sys -s eth1 -c CLOCK_REALTIME -w -O 0 -m & phc2sys -s eth1 -c CLOCK_REALTIME -w -O 0 -m &
@ -425,18 +424,20 @@ Make necessary configurations on the Kata container before executing
:height: 400px :height: 400px
:align: center :align: center
Figure 4: Performance Report of Case 2 *Figure 4: Case 2 performance report*
In this test, the indicator of ``RT Application latency`` got similar result In this test, ``RT Application latency`` showed similar results
with that of `Case 1`. It is expected since there are no any optimizations done to Case 1. This was expected, since there were no optimizations made.
on it. ``Scheduled times`` was well controlled (ranged from `1253.188us` to ``Scheduled times`` was well controlled (ranged from 1253.188us to
`1253.343us`) which indicates the TSN feature is functional. The indicator of 1253.343us), which indicates the TSN feature was functional. The measured
``TSN Network jitter`` also proved it. ``TSN Network jitter`` also proves TSN was functional.
* Case 3, based on the setting of `Case 2`, enable * Case 3: Stress test.
This scenario used the Case 2 settings and enabled
`802.1qbv <http://www.ieee802.org/1/pages/802.1bv.html>`_ support on the TSN `802.1qbv <http://www.ieee802.org/1/pages/802.1bv.html>`_ support on the TSN
switch, and ``iperf3`` were running on ``Node 3`` and ``Node 4`` to transfer switch. Also, ``iperf3`` was used on ``Node 3`` and ``Node 4`` for massive
massive best-effort traffic to stress the overall network communication. best-effort traffic to stress the overall network communication.
:: ::
@ -475,20 +476,20 @@ on it. ``Scheduled times`` was well controlled (ranged from `1253.188us` to
:height: 400px :height: 400px
:align: center :align: center
Figure 5: Performance Report of Case 3 *Figure 5: Case 3 performance report*
The result was very similar with that of `Case 2`. It demonstrated that even a The results were very similar to Case 2. The test demonstrated that even when
great amount of best-effort traffic was sent to the TSN network, the a large amount of best-effort traffic was sent to the TSN network, the
time-sensitive packets sent from ``sample-app-taprio`` was not impacted. The time-sensitive packets sent from ``sample-app-taprio`` were not impacted. The
determinism was still guaranteed. determinism was still guaranteed.
------- -------
Summary Summary
------- -------
In this guide, we introduced how to enable TSN support in Kata containers on In this guide, we introduced how to enable TSN support in Kata Containers on the
StarlingX platform. The experimental results demonstrated the capability of StarlingX platform. The experimental results demonstrated the capability of TSN
TSN in Kata containers. The cycle time (2ms) is not small enough for in Kata Containers. Currently, the cycle time (2ms) is not low enough for some
some critical use cases. In the future, some optimizations could be critical use cases. In the future, optimizations could be made to achieve
done to achieve better performance, such as, replacing standard kernel better performance, such as replacing the standard kernel with a real-time
with real-time kernel. kernel.