Debian: Add libbnxt_re Infiniband verbs library

This commit adds the libbnxt_re Infiniband verbs library to Debian-based
StarlingX.

In comparison to commit 9baff8aa884a ("Introduce libbnxt_re version
220.0.5.0") and commit 4112936362c6 ("libbnxt_re: Add support for
rdma-core v37") that introduce the same library to CentOS-based
StarlingX, the only change in this commit was to remove the use of the
"stdatomic.h" header's compatibility-oriented version shipped with
libbxnt_re. This header file is already provided by Debian's newer
toolchain.

The two patches make the library compatible with rdma-core v37 provided
by Mellanox OFED's version of rdma-core. For further information, please
see commit 4112936362c6 ("libbnxt_re: Add support for rdma-core v37").

Verification notes:
- An ISO image with the standard kernel was successfully built with this
  commit and the resulting ISO image was installed onto a physical
  server with a quad-port Broadcom NetXtreme-E network adapter, of which
  the third and fourth ports were connected to each other with a
  loopback cable.

- When we installed the Linux RDMA community's perftest package on
  Debian-based StarlingX with apt-get and attempted to run the benchmark
  programs, we encountered a dynamically-linked library error caused by
  the "libefa.so" Infiniband verbs library (that perftest's utilities
  appear to link against) not existing on the system. This occurs
  because the Mellanox OFED version of the rdma-core package in
  StarlingX does not build the libefa.so library.

  To work around this, we built the perftest package in a Debian
  Bullseye container, after having modified perftest's configure.ac file
  to avoid linking against libefa.so. The resulting binaries were
  successfully used on the aforementioned server to verify the
  libbnxt_re Infiniband verbs library and the bnxt_re driver.

  The test procedure consisted of using a server ib_write_bw instance on
  the 3rd port of the network adapter and another instance as a client
  on the 4th port of the network adapter. (Recall that ports 3 and 4
  were connected to each other.)

- The preempt-rt kernel was not tested due to not being able to generate
  a working ISO image that installed onto a virtual machine. (The
  installation process was aborted with a kickstart file parse error.)

Story: 2009915
Task: 45139

Change-Id: I917c18ef262afdf3ddfaa7e31af2583b8f74300b
Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>

debian_pkg_dirs

@@ -7,6 +7,7 @@ kernel-modules/intel-igb_uio
 kernel-modules/intel-opae-fpga
 kernel-modules/mlnx-ofa_kernel
 kernel-modules/qat17
+userspace/broadcom/libbnxt_re
 userspace/mellanox/rdma-core
 userspace/mellanox/mstflint
 userspace/mellanox/mlnx-tools

userspace/broadcom/libbnxt_re/debian/deb_folder/changelog

@@ -0,0 +1,7 @@
+libbnxt-re (220.0.5.0-1) unstable; urgency=medium
+
+  * Initial release of the libbnxt_re Infiniband verbs library for StarlingX.
+    This library corresponds to "libbnxt_re-220.0.5.0.tar.gz" in Broadcom's
+    "bcm_220.0.83.0" release.
+
+ -- M. Vefa Bicakci <vefa.bicakci@windriver.com>  Mon, 18 Apr 2022 20:07:24 +0000

userspace/broadcom/libbnxt_re/debian/deb_folder/control

@@ -0,0 +1,15 @@
+Source: libbnxt-re
+Section: net
+Priority: optional
+Maintainer: StarlingX Developers <starlingx-discuss@lists.starlingx.io>
+Build-Depends: debhelper-compat (= 13), libibverbs-dev
+Standards-Version: 4.5.1
+Rules-Requires-Root: no
+
+Package: libbnxt-re
+Architecture: any
+Depends: ${shlibs:Depends}, ${misc:Depends}
+Description: libbnxt_re Infiniband verbs library
+ libbnxt_re provides a device-specific userspace driver for
+ Broadcom NetXtreme RoCE Adapters for use with the libibverbs
+ library.

userspace/broadcom/libbnxt_re/debian/deb_folder/copyright

@@ -0,0 +1,48 @@
+Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+Upstream-Name: libbnxt_re
+Upstream-Contact: Broadcom Inc.
+Source: libbnxt_re-220.0.5.0.tar.gz
+Comment:
+ Source archive is included as
+  bcm_220.0.83.0/Linux/Linux_RoCE/RoCE_Lib/libbnxt_re-220.0.5.0.tar.gz
+ in the compressed tar archive at the following address:
+  https://docs.broadcom.com/docs-and-downloads/ethernet-network-adapters/NXE/BRCM_220.0.83.0/bcm_220.0.83.0.tar.gz
+
+Files: *
+Copyright: Copyright 2016-2022 Broadcom Inc.
+Comment:
+ Documentation in the source archive indicates copyright years 2016-2021.
+ However, the source code was released in 2022. Hence, the copyright year above
+ is indicated by this file as 2016-2022.
+License: GPL-2 or BSD-2-clause
+
+License: GPL-2
+ This program is free software; you can redistribute it and/or modify it under
+ the terms and conditions of the GNU General Public License, version 2, as
+ published by the Free Software Foundation.
+
+ This program is distributed in the hope it will be useful, but WITHOUT ANY
+ WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
+ PARTICULAR PURPOSE. See the GNU General Public License for more details.
+
+ You should have received a copy of the GNU General Public License along with
+ this program; if not, write to the Free Software Foundation, Inc., 51 Franklin
+ St - Fifth Floor, Boston, MA 02110-1301 USA.
+
+ The full GNU General Public License is included in this distribution in the
+ file called "COPYING".
+
+ On Debian-based systems the full text of the GNU General Public License
+ version 2 license can be found in `/usr/share/common-licenses/GPL-2'.
+
+License: BSD-2-clause
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+
+ 1. Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+ 2. Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in
+    the documentation and/or other materials provided with the
+    distribution.

userspace/broadcom/libbnxt_re/debian/deb_folder/patches/0002-configure.ac-Detect-rdma-core-v36-and-v37-too.patch

@@ -0,0 +1,118 @@
+From ffd1f85f70551c897e6e62b89acf8b170c1c82ea Mon Sep 17 00:00:00 2001
+From: "M. Vefa Bicakci" <vefa.bicakci@windriver.com>
+Date: Thu, 31 Mar 2022 16:14:22 -0400
+Subject: [PATCH] configure.ac: Detect rdma-core v36 and v37 too
+
+This commit makes the configure.ac script detect rdma-core v36 and v37
+as well. Of these, we are only interested in v37, because Mellanox's
+OpenFabrics Enterprise Distribution is based on rdma-core v37.
+
+Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>
+---
+ configure.ac | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++-
+ 1 file changed, 75 insertions(+), 1 deletion(-)
+
+diff --git a/configure.ac b/configure.ac
+index 9e4e980886d4..3ece437e664a 100644
+--- a/configure.ac
++++ b/configure.ac
+@@ -68,6 +68,10 @@ m4_define([rcore_version],[m4_esyscmd(rc=$(grep -o 		\
+ 	_ibv_device_ops verbsincpath/include/infiniband/verbs.h |	\
+ 	tail -1);						\
+ 	if test ! -z "$rc"; then				\
++		rcverv37=$(grep "IBV_WC_DRIVER3" \
++			   verbsincpath/include/infiniband/verbs.h|tail -1);	\
++		rcverv36=$(grep "ibv_query_qp_data_in_order" \
++			   verbsincpath/include/infiniband/verbs.h|tail -1);	\
+ 		rcverv35=$(grep "ibv_is_fork_initialized" \
+ 			   verbsincpath/include/infiniband/verbs.h|tail -1);	\
+ 		rcverv34=$(grep "ibv_reg_dmabuf_mr" \
+@@ -90,7 +94,11 @@ m4_define([rcore_version],[m4_esyscmd(rc=$(grep -o 		\
+ 			verbsincpath/include/infiniband/verbs.h|tail -1);	\
+ 		rcverv17=$(grep -o ibv_alloc_td			\
+ 			verbsincpath/include/infiniband/verbs.h|tail -1);	\
+-		if test ! -z "$rcverv35"; then			\
++		if test ! -z "$rcverv37"; then			\
++			echo -n "v37";				\
++		elif test ! -z "$rcverv36"; then			\
++			echo -n "v36";				\
++		elif test ! -z "$rcverv35"; then			\
+ 			echo -n "v35";				\
+ 		elif test ! -z "$rcverv34"; then			\
+ 			echo -n "v34";				\
+@@ -175,6 +183,72 @@ if test "$rc" != ""; then
+ 	AC_DEFINE([HAVE_IBV_CMD_MODIFY_QP_EX],[1],[ibv_cmd_modify_qp_ex supported])
+ 	RDMA_CORE_HEADERS=`echo -n __include_path__`
+ 	case  "$(echo -n rcore_version)" in
++	v37)
++		AC_MSG_RESULT(yes using v37)
++		AC_DEFINE([IBV_WC_DRIVER3_IN_IBV_WC_OPCODE],[1],
++			  [Rdma-Core Package has IBV_WC_DRIVER3 in ibv_wc_opcode enum])
++		AC_DEFINE([IBV_WC_DRIVER2_IN_IBV_WC_OPCODE],[1],
++			  [Rdma-Core Package has IBV_WC_DRIVER2 in ibv_wc_opcode enum])
++		AC_DEFINE([IBV_QUERY_QP_DATA_IN_ORDER_IN_CONTEXT_OPS],[1],
++			  [Rdma-Core Package has query_qp_data_in_order in verbs_context_ops])
++		AC_DEFINE([VERBS_ONLY_QUERY_DEVICE_EX_DEFINED],[1],
++			  [Rdma-Core Package has only query_device_ex])
++		AC_DEFINE([IBV_CMD_MODIFY_QP_EX_HAS_7_ARG],[1],
++			  [Rdma-Core Package ibv_cmd_modify_qp_ex has 7 arguments])
++		AC_DEFINE([IBV_FREE_CONTEXT_IN_CONTEXT_OPS],[1],
++			  [Rdma-Core Package has free context in ibv_context_ops])
++		AC_DEFINE([REG_MR_VERB_HAS_5_ARG],[1],
++			  [Rdma-Core Package reg_mr has 5 argument])
++		AC_DEFINE([IBV_CMD_ALLOC_MW_HAS_1_ARG],[1],
++			  [Rdma-Core Package ibv_cmd_alloc_mw has single argument])
++		AC_DEFINE([PROVIDER_DRIVER_HAS_2_ARGS],[1],
++			  [Rdma-Core Package PROVIDER_DRIVER macro has 2 arguments])
++		AC_DEFINE([ALLOC_CONTEXT_HAS_PRIVATE_DATA],[1],
++			  [Rdma-Core Package alloc_context has private data])
++		AC_DEFINE([VERBS_MR_DEFINED],[1],
++			  [Rdma-Core Package has verbs_mr defined])
++		AC_DEFINE([VERBS_INIT_AND_ALLOC_CONTEXT_HAS_5_ARG],[1],
++			  [Rdma-Core Package has 5 Arg for verbs_init_and_alloc_context macro])
++		AC_DEFINE([RCP_HAS_PROVIDER_DRIVER],[1],
++			  [Rdma-Core Package has PROVIDER_DRIVER macro])
++		AC_DEFINE([RCP_USE_IB_UVERBS],[1],
++			[Rdma-Core Package uses IB user verbs API])
++		AC_DEFINE([RCP_USE_ALLOC_CONTEXT],[1],
++			[Rdma-Core Package uses alloc_context instead of init_context])
++		rc=`grep -o ibv_read_sysfs_file src/rc-compat/v37/driver.h|tail -1`
++	;;
++	v36)
++		AC_MSG_RESULT(yes using v36)
++		AC_DEFINE([IBV_WC_DRIVER2_IN_IBV_WC_OPCODE],[1],
++			  [Rdma-Core Package has IBV_WC_DRIVER2 in ibv_wc_opcode enum])
++		AC_DEFINE([IBV_QUERY_QP_DATA_IN_ORDER_IN_CONTEXT_OPS],[1],
++			  [Rdma-Core Package has query_qp_data_in_order in verbs_context_ops])
++		AC_DEFINE([VERBS_ONLY_QUERY_DEVICE_EX_DEFINED],[1],
++			  [Rdma-Core Package has only query_device_ex])
++		AC_DEFINE([IBV_CMD_MODIFY_QP_EX_HAS_7_ARG],[1],
++			  [Rdma-Core Package ibv_cmd_modify_qp_ex has 7 arguments])
++		AC_DEFINE([IBV_FREE_CONTEXT_IN_CONTEXT_OPS],[1],
++			  [Rdma-Core Package has free context in ibv_context_ops])
++		AC_DEFINE([REG_MR_VERB_HAS_5_ARG],[1],
++			  [Rdma-Core Package reg_mr has 5 argument])
++		AC_DEFINE([IBV_CMD_ALLOC_MW_HAS_1_ARG],[1],
++			  [Rdma-Core Package ibv_cmd_alloc_mw has single argument])
++		AC_DEFINE([PROVIDER_DRIVER_HAS_2_ARGS],[1],
++			  [Rdma-Core Package PROVIDER_DRIVER macro has 2 arguments])
++		AC_DEFINE([ALLOC_CONTEXT_HAS_PRIVATE_DATA],[1],
++			  [Rdma-Core Package alloc_context has private data])
++		AC_DEFINE([VERBS_MR_DEFINED],[1],
++			  [Rdma-Core Package has verbs_mr defined])
++		AC_DEFINE([VERBS_INIT_AND_ALLOC_CONTEXT_HAS_5_ARG],[1],
++			  [Rdma-Core Package has 5 Arg for verbs_init_and_alloc_context macro])
++		AC_DEFINE([RCP_HAS_PROVIDER_DRIVER],[1],
++			  [Rdma-Core Package has PROVIDER_DRIVER macro])
++		AC_DEFINE([RCP_USE_IB_UVERBS],[1],
++			[Rdma-Core Package uses IB user verbs API])
++		AC_DEFINE([RCP_USE_ALLOC_CONTEXT],[1],
++			[Rdma-Core Package uses alloc_context instead of init_context])
++		rc=`grep -o ibv_read_sysfs_file src/rc-compat/v36/driver.h|tail -1`
++	;;
+ 	v35)
+ 		AC_MSG_RESULT(yes using v35)
+ 		AC_DEFINE([VERBS_ONLY_QUERY_DEVICE_EX_DEFINED],[1],
+-- 
+2.29.2
+

userspace/broadcom/libbnxt_re/debian/deb_folder/patches/series

@@ -0,0 +1,2 @@
+0001-rc-compat-Add-headers-from-v37.3.patch
+0002-configure.ac-Detect-rdma-core-v36-and-v37-too.patch

userspace/broadcom/libbnxt_re/debian/deb_folder/rules

@@ -0,0 +1,53 @@
+#!/usr/bin/make -f
+#
+# Copyright (c) 2022 Wind River Systems, Inc.
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. The ASF licenses this
+# file to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# output every command that modifies files on the build system.
+#export DH_VERBOSE = 1
+
+include /usr/share/dpkg/architecture.mk
+
+libdir=/usr/lib/$(DEB_HOST_MULTIARCH)
+debdir=debian/libbnxt-re
+
+%:
+	dh $@
+
+override_dh_auto_install:
+	dh_auto_install
+	# Now, the following symbolic link exists:
+	#   /usr/lib/x86_64-linux-gnu/libbnxt_re.so -> libbnxt_re-rdmav34.so
+	#
+	# We need to create the following symbolic link as well:
+	#   /usr/lib/x86_64-linux-gnu/libibverbs/libbnxt_re-rdmav34.so -> ../libbnxt_re-rdmav34.so
+	#
+	# To avoid hard-coding the version (v34), read the link.
+	libname="$$(readlink "$(debdir)/$(libdir)/libbnxt_re.so")" && \
+	    mkdir -p "$(debdir)/$(libdir)/libibverbs/" && \
+	    ln -s "../$${libname}" "$(debdir)/$(libdir)/libibverbs/$${libname}"
+	# Remove static library and libtool archive files.
+	# - Static library is not necessary given that no other packages statically
+	#   link against the libbnxt_re library.
+	# - Libtool archive files (*.la) are not recommended to be included in
+	#   Debian packages:
+	#   https://www.debian.org/doc/manuals/maint-guide/advanced.en.html#library
+	#   https://wiki.debian.org/ReleaseGoals/LAFileRemoval
+	rm -vf \
+	    $(debdir)/$(libdir)/libbnxt_re.a \
+	    $(debdir)/$(libdir)/libbnxt_re.la

userspace/broadcom/libbnxt_re/debian/deb_folder/source/format

@@ -0,0 +1 @@
+3.0 (quilt)

userspace/broadcom/libbnxt_re/debian/dl_hook

@@ -0,0 +1,51 @@
+#!/bin/bash
+#
+# Copyright (c) 2022 Wind River Systems, Inc.
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. The ASF licenses this
+# file to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# The only parameter is the name of the new directory that will contain
+# the extracted source code. Be aware that this new directory will be
+# created in the same directory as where this script is located when
+# building.
+# Tools needed: mkdir, rm, tar
+
+DL_TARBALL=bcm_220.0.83.0.tar.gz
+SRC_TARBALL=Linux/Linux_RoCE/RoCE_Lib/libbnxt_re-220.0.5.0.tar.gz
+TMPDIR="bcm_tmp"
+
+DESTDIR="${1}"
+if test -z "${DESTDIR}"; then
+    echo "Missing destination directory on command line."
+    exit 1
+fi
+
+rm -rf "${TMPDIR}"
+mkdir -p "${TMPDIR}"
+
+if ! tar -C "${TMPDIR}" --strip-components=1 -xvf "${DL_TARBALL}"; then
+    echo "Could not extract into ${TMPDIR}: ${DL_TARBALL}"
+    exit 1
+fi
+
+mkdir -p "${DESTDIR}"
+if ! tar -C "${DESTDIR}" --strip-components=1 -xvf "${TMPDIR}/${SRC_TARBALL}"; then
+    echo "Could not extract into ${DESTDIR}: ${TMPDIR}/${SRC_TARBALL}"
+    exit 1
+fi
+
+rm -rf "${TMPDIR}"

userspace/broadcom/libbnxt_re/debian/meta_data.yaml

@@ -0,0 +1,14 @@
+---
+debver: 220.0.5.0
+debname: libbnxt-re
+dl_hook: dl_hook
+dl_files:
+  bcm_220.0.83.0.tar.gz:
+    topdir: null
+    url: "https://docs.broadcom.com/docs-and-downloads/\
+          ethernet-network-adapters/NXE/BRCM_220.0.83.0/\
+          bcm_220.0.83.0.tar.gz"
+    sha256sum: 85bdfc30b4bd3e184e3b57a48055c11085e3b97593f7b4a8347fa50a9d571336
+revision:
+  dist: $STX_DIST
+  PKG_GITREVCOUNT: true