The version of rdma-core in StarlingX is based on version 37 provided by
Mellanox's OpenFabrics Enterprise Distribution, and the rdma-core
package's libibverbs has a library ABI version 34. Furthermore,
libbnxt_re's configure.ac script indicates that libbnxt_re is
(currently) compatible with rdma-core versions up to 35.
Despite the aforementioned, while preparing commit 9baff8aa884a
("Introduce libbnxt_re version 220.0.5.0"), we were under the impression
that since the ABI version is 34, libbnxt_re would work fine, given that
it compiled without issues.
However, during run-time testing with the Linux-RDMA community's
perftest package, we observed that test programs such as "ib_send_bw"
would crash due to segmentation faults. Further debugging indicated that
libbnxt_re was calling an incorrect function by dereferencing a function
pointer in an "ops" structure, whose definition had been modified
between rdma-core v35 and v37 by inserting a new function pointer, which
caused some function pointer offsets to change. Furthermore, it was
noticed that libbnxt_re's build procedure was assuming that libbnxt_re
was being built against rdma-core v35, instead of what we had thought to
be v34. In conclusion, we had encountered an incompatibility.
Note that Broadcom has since released a newer driver and library bundle
(v221.1.28.0), but the newer version of libbnxt_re in that bundle does
not support rdma-core v37 either.
Due to what is discussed above, this commit patches libbnxt_re so that
it is compatible with rdma-core v37. We found that the API changes
between rdma-core v35 and v37 are Mellanox-specific and have no impact
on libbnxt_re. As a result, this commit only imports rdma-core v37.3's
headers and ensures that libbnxt_re can be built against the newer
headers.
Verification:
- An ISO image was built with this commit using an incremental and
monolithic build procedure.
- The ISO image was installed and bootstrapped in low-latency All-in-One
simplex mode on a server that has a quad-port Broadcom NetXtreme-E
57504 network adapter.
- ib_send_bw and ib_send_lat utilities in the Linux-RDMA community's
perftest package were observed to no longer crash due to segmentation
faults with these changes.
- Basic tests were carried by connecting the third and fourth ports of
the network adapter with a fiber and running traffic across the two
ports. The test commands were as follows:
# Server
sudo ib_send_bw -F -D 10 -d bnxt_re2
# Client
sudo ib_send_bw -F -D 10 -d bnxt_re3 10.240.240.12
According to the following commands, RDMA traffic was successfully
processed by the aforementioned ports:
ethtool -S enp101s0f2
ethtool -S enp101s0f3
sudo cat /sys/kernel/debug/bnxt_re/bnxt_re{2,3}/info
Story: 2009915
Task: 44916
Change-Id: I79528fb2d04e972dd69750f6bb3d132f9de5ada8
Fixes: 9baff8aa884a ("Introduce libbnxt_re version 220.0.5.0")
Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>
4112936362