On behalf of JB, submitting updated files against patchset 4.

Completed Review Comments for Patch Set 2

Added SNMPv3 Updates

Deleted 2 topics (Configure SNMP Trap Destinations, Add an SNMP Community String Using the CLI)
Updated topics with SNMPv3 updates (Enable SNMP Support, Traps, Active Alarm Tables, Event Tables)
Added new topic (Uninstall SNMP)

Story: 2008132
Task: 41395

Change-Id: I8bcff5d2acbbdcc98519e552bb16406110407388
Signed-off-by: Juanita-Balaraj <juanita.balaraj@windriver.com>
Signed-off-by: Stone <ronald.stone@windriver.com>
This commit is contained in:
Juanita-Balaraj 2021-02-01 15:18:49 -05:00 committed by Rafael Jardim
parent 5a7ec8c408
commit 959c15c7b4
11 changed files with 662 additions and 317 deletions

3
doc/source/.vscode/settings.json vendored Normal file
View File

@ -0,0 +1,3 @@
{
"restructuredtext.confPath": ""
}

View File

@ -1,111 +0,0 @@
.. xti1552680491532
.. _adding-an-snmp-community-string-using-the-cli:
==========================================
Add an SNMP Community String Using the CLI
==========================================
To enable :abbr:`SNMP (Simple Network Management Protocol)` services you need
to define one or more SNMP community strings using the command line interface.
.. rubric:: |context|
No default community strings are defined on |prod| after the initial
commissioning of the cluster. This means that no SNMP operations are enabled
by default.
The following exercise illustrates the system commands available to manage and
query SNMP community strings. It uses the string **commstr1** as an example.
.. caution::
For security, do not use the string **public**, or other community strings
that could easily be guessed.
.. rubric:: |prereq|
All commands must be executed on the active controller's console, which can be
accessed using the OAM floating IP address. You must acquire Keystone **admin**
credentials in order to execute the commands.
.. rubric:: |proc|
#. Add the SNMP community string commstr1 to the system.
.. code-block:: none
~(keystone_admin)$ system snmp-comm-add -c commstr1
+-----------+--------------------------------------+
| Property | Value |
+-----------+--------------------------------------+
| access | ro |
| uuid | eccf5729-e400-4305-82e2-bdf344eb868d |
| community | commstr1 |
| view | .1 |
+-----------+--------------------------------------+
The following are attributes associated with the new community string:
**access**
The SNMP access type. In |prod| all community strings provide read-only
access.
**uuid**
The UUID associated with the community string.
**community**
The community string value.
**view**
The is always the full MIB tree.
#. List available community strings.
.. code-block:: none
~(keystone_admin)$ system snmp-comm-list
+----------------+--------------------+--------+
| SNMP community | View | Access |
+----------------+--------------------+--------+
| commstr1 | .1 | ro |
+----------------+--------------------+--------+
#. Query details of a specific community string.
.. code-block:: none
~(keystone_admin)$ system snmp-comm-show commstr1
+------------+--------------------------------------+
| Property | Value |
+------------+--------------------------------------+
| access | ro |
| created_at | 2014-08-14T21:12:10.037637+00:00 |
| uuid | eccf5729-e400-4305-82e2-bdf344eb868d |
| community | commstr1 |
| view | .1 |
+------------+--------------------------------------+
#. Delete a community string.
.. code-block:: none
~(keystone_admin)$ system snmp-comm-delete commstr1
Deleted community commstr1
.. rubric:: |result|
Community strings in |prod| provide query access to any SNMP monitor
workstation that can reach the controller's OAM address on UDP port 161.
You can verify SNMP access using any monitor tool. For example, the freely
available command :command:`snmpwalk` can be issued from any host to list
the state of all SNMP Object Identifiers \(OID\):
.. code-block:: none
$ snmpwalk -v 2c -c commstr1 10.10.10.100 > oids.txt
In this example, 10.10.10.100 is the |prod| OAM floating IP address. The output,
which is a large file, is redirected to the file oids.txt.

View File

@ -1,89 +0,0 @@
.. sjb1552680530874
.. _configuring-snmp-trap-destinations:
================================
Configure SNMP Trap Destinations
================================
:abbr:`SNMP (Simple Network Management Protocol)` trap destinations are hosts
configured in |prod| to receive unsolicited SNMP notifications.
.. rubric:: |context|
Destination hosts are specified by IP address, or by host name if it can be
properly resolved by |prod|. Notifications are sent to the hosts using a
designated community string so that they can be validated.
.. rubric:: |proc|
#. Configure IP address 10.10.10.1 to receive SNMP notifications using the
community string commstr1.
.. code-block:: none
~(keystone_admin)$ system snmp-trapdest-add -c commstr1 --ip_address 10.10.10.1
+------------+--------------------------------------+
| Property | Value |
+------------+--------------------------------------+
| uuid | c7b6774e-7f45-40f5-bcca-3668de2a186f |
| ip_address | 10.10.10.1 |
| community | commstr1 |
| type | snmpv2c_trap |
| port | 162 |
| transport | udp |
+------------+--------------------------------------+
The following are attributes associated with the new community string:
**uuid**
The UUID associated with the trap destination object.
**ip\_address**
The trap destination IP address.
**community**
The community string value to be associated with the notifications.
**type**
snmpv2c\_trap, the only supported message type for SNMP traps.
**port**
The destination UDP port that SNMP notifications are sent to.
**transport**
The transport protocol used to send notifications.
#. List defined trap destinations.
.. code-block:: none
~(keystone_admin)$ system snmp-trapdest-list
+------------+----------------+------+--------------+-----------+
| IP Address | SNMP Community | Port | Type | Transport |
+-------------+----------------+------+--------------+-----------+
| 10.10.10.1 | commstr1 | 162 | snmpv2c_trap | udp |
+-------------+----------------+------+--------------+-----------+
#. Query access details of a specific trap destination.
.. code-block:: none
~(keystone_admin)$ system snmp-trapdest-show 10.10.10.1
+------------+--------------------------------------+
| Property | Value |
+------------+--------------------------------------+
| uuid | c7b6774e-7f45-40f5-bcca-3668de2a186f |
| ip_address | 10.10.10.1 |
| community | commstr1 |
| type | snmpv2c_trap |
| port | 162 |
| transport | udp |
+------------+--------------------------------------+
#. Disable the sending of SNMP notifications to a specific IP address.
.. code-block:: none
~(keystone_admin)$ system snmp-trapdest-delete 10.10.10.1
Deleted ip 10.10.10.1

View File

@ -6,24 +6,502 @@
Enable SNMP Support Enable SNMP Support
=================== ===================
:abbr:`SNMP (Simple Network Management Protocol)` support must be enabled .. contents::
before you can begin using it to monitor a system. :local:
:depth: 2
|SNMP| support must be enabled and configured before you can begin using it to
monitor |prod|.
.. rubric:: |context| .. rubric:: |context|
In order to have a workable SNMP configuration you must use the command line In order to enable and configure |SNMP|, complete the following steps.
interface on the active controller to complete the following steps.
.. rubric:: |proc| .. rubric:: |proc|
#. Define at least one SNMP community string. #. On the active controller, acquire Keystone administrative privileges.
See |fault-doc|: :ref:`Adding an SNMP Community String Using the CLI .. code-block:: none
<adding-an-snmp-community-string-using-the-cli>` for details.
#. Configure at least one SNMP trap destination. $ source /etc/platform/openrc
~(keystone_admin)]$
This will allow alarms and logs to be reported as they happen. #. Use the following command to list the system applications and check whether
SNMP is uploaded or applied. If SNMP is already "uploaded", go to
Step 5 to configure and enable SNMP. If SNMP is already "applied", SNMP is
already configured and enabled, see, :ref:`Change Configuration of the SNMP application <change-configuration-of-SNMP>`
to make configuration changes.
For more information, see :ref:`Configuring SNMP Trap Destinations .. code-block:: none
<configuring-snmp-trap-destinations>`.
~(keystone)admin)$ system application-list
+--------------------------+-------------------------+-----------------------------------+----------------------------------------+---------------+-----------+
| application | version | manifest name | manifest file | status | progress |
+--------------------------+-------------------------+-----------------------------------+----------------------------------------+---------------+-----------+
| cert-manager | 1.0-6 | cert-manager-manifest | certmanager-manifest.yaml | applied | completed |
| nginx-ingress-controller | 1.0-0 | nginx-ingress-controller-manifest | nginx_ingress_controller_manifest.yaml | applied | completed |
| oidc-auth-apps | 1.0-29 | oidc-auth-manifest | manifest.yaml | uploaded | completed |
| platform-integ-apps | 1.0-9 | platform-integration-manifest | manifest.yaml | applied | completed |
| rook-ceph-apps | app-version-placeholder | manifest-placeholder | tarfile-placeholder | upload-failed | None |
+--------------------------+-------------------------+-----------------------------------+----------------------------------------+---------------+-----------+
#. In order to load the SNMP application definitions (Armada manifest and Helm
charts), where **[snmp-<Major>.<Minor>-<Build>.tgz]**, for example,
**snmp-1.0-2.tgz** is the app filename, with version **1.0-2**,
available at the following path "/usr/local/share/applications/helm",
use the following command.
.. code-block:: none
~(keystone)admin)$ system application-upload <path>/snmp-1.0-2.tgz
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | False |
| app_version | 1.0-2 |
| created_at | 2020-11-30T14:45:20.442133+00:00 |
| manifest_file | snmp_manifest.yaml |
| manifest_name | snmp-manifest |
| name | snmp |
| progress | None |
| status | uploading |
| updated_at | None |
+---------------+----------------------------------+
#. List the SNMP application using the following command to see the status
of the upload and wait for the upload to complete.
.. code-block:: none
~(keystone)admin)$ system application-list
+--------------------------+-------------------------+-----------------------------------+----------------------------------------+---------------+-----------+
| application | version | manifest name | manifest file | status | progress |
+--------------------------+-------------------------+-----------------------------------+----------------------------------------+---------------+-----------+
| cert-manager | 1.0-6 | cert-manager-manifest | certmanager-manifest.yaml | applied | completed |
| nginx-ingress-controller | 1.0-0 | nginx-ingress-controller-manifest | nginx_ingress_controller_manifest.yaml | applied | completed |
| oidc-auth-apps | 1.0-29 | oidc-auth-manifest | manifest.yaml | uploaded | completed |
| platform-integ-apps | 1.0-9 | platform-integration-manifest | manifest.yaml | applied | completed |
| rook-ceph-apps | app-version-placeholder | manifest-placeholder | tarfile-placeholder | upload-failed | None |
| snmp | 1.0-2 | snmp-manifest | snmp_manifest.yaml | uploaded | completed |
+--------------------------+-------------------------+-----------------------------------+----------------------------------------+---------------+-----------+
#. Create a Helm chart values file (for example, user_conf.yaml) with the
definition of the **configmap:user_conf** attribute, defining your SNMP
configuration of V2 communities, V2 trap sinks, V3 users and/or V3 trap
sessions, as shown in the example below. The **configmap:user_conf**
variable in the SNMP Helm chart is a multi-line variable that follows the
syntax of Net-SNMP's snmpd.conf file for configuring the SNMP agent, see
`http://www.net-snmp.org/docs/man/snmpd.conf.html <http://www.net-snmp.org/docs/man/snmpd.conf.html>`__,
for a detailed description of the commands.
.. warning::
Since this file may contain sensitive security information, this file
should be removed from the system after executing the command and
stored off-box, or regenerated, if required.
.. code-block:: none
cat <<EOF > user_conf.yaml
configmap:
user_conf: |-
# Configure V2 Community
# rocommunity COMMUNITY [SOURCE [OID | -V VIEW [CONTEXT]]]
rocommunity testcommunity default -V all
# Configure V2 Trap Sink / Destination
# trap2sink HOST [COMMUNITY [PORT]]
trap2sink 10.10.10.1:162 testcommunity
# Configure V3 User
# createUser [-e ENGINEID] username (MD5|SHA) authpassphrase [DES|AES] [privpassphrase]
createUser testuser MD5 testpassword DES
# Configure RW access for V3 User
# rouser [-s SECMODEL] USER [noauth|auth|priv [OID | -V VIEW [CONTEXT]]]
rouser testuser priv
# Configure V3 Trap Session / Destination
# trapsess -v 3 -u USER -a [MD5|SHA] -A authpassphrase -l [noauth|auth|priv] -x [DES|AES] -X privpassphrase [<transport-specifier>:]<transport-address>
trapsess -v 3 -u testuser -a MD5 -A testpassword -l authPriv -x DES -X testpassword udp:10.10.10.1:162
EOF
**\(Optional\)** You can add your own EngineID value, instead of having it
auto-created. This keeps the EngineID value the same, even when the SNMP
application restarts. The EngineID is required if you are using an SNMP
trap viewer or SNMP monitoring tool. Add the 'engineID [STRING]' value in
the **configmap:user_conf** variable, as shown below.
.. code-block:: none
cat <<EOF > user_conf.yaml
configmap:
user_conf: |-
...
engineID [STRING]
...
EOF
#. Update the values of the **configmap:user_conf** attribute on the Helm
chart using the following command.
.. code-block:: none
~(keystone_admin)$ system helm-override-update --values user_conf.yaml snmp snmp kube-system
+----------------+------------------------------------------------------------------------------------------------------------+
| Property | Value |
+----------------+------------------------------------------------------------------------------------------------------------+
| name | snmp |
| namespace | kube-system |
| user_overrides | configmap: |
| | user_conf: |- |
| | createUser testuser MD5 testpassword DES |
| | rouser testuser priv |
| | rocommunity testcommunity default -V all |
| | trapsess -v 3 -u testuser -a MD5 -A testpassword -l authPriv -x DES -X testpassword udp:10.10.10.1:162 |
| | trap2sink 10.10.10.1:162 testcommunity |
+----------------+------------------------------------------------------------------------------------------------------------+
#. Run the following command to apply the changes and start the SNMP
application.
.. code-block:: none
~(keystone)admin)$ system application-apply snmp
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | False |
| app_version | 1.0-2 |
| created_at | 2020-11-30T14:45:20.442133+00:00 |
| manifest_file | snmp_manifest.yaml |
| manifest_name | snmp-manifest |
| name | snmp |
| progress | None |
| status | applying |
| updated_at | 2020-11-30T14:45:23.088575+00:00 |
+---------------+----------------------------------+
#. List the SNMP application and check the status. Wait for the SNMP
application to have fully started and is in the "applied" state.
.. code-block:: none
~(keystone)admin)$ system application-list
+--------------------------+-------------------------+-----------------------------------+----------------------------------------+---------------+-----------+
| application | version | manifest name | manifest file | status | progress |
+--------------------------+-------------------------+-----------------------------------+----------------------------------------+---------------+-----------+
| cert-manager | 1.0-6 | cert-manager-manifest | certmanager-manifest.yaml | applied | completed |
| nginx-ingress-controller | 1.0-0 | nginx-ingress-controller-manifest | nginx_ingress_controller_manifest.yaml | applied | completed |
| oidc-auth-apps | 1.0-29 | oidc-auth-manifest | manifest.yaml | uploaded | completed |
| platform-integ-apps | 1.0-9 | platform-integration-manifest | manifest.yaml | applied | completed |
| rook-ceph-apps | app-version-placeholder | manifest-placeholder | tarfile-placeholder | upload-failed | None |
| snmp | 1.0-2 | snmp-manifest | snmp_manifest.yaml | applied | completed |
+--------------------------+-------------------------+-----------------------------------+----------------------------------------+---------------+-----------+
#. Create a Helm chart values file (for example, snmp_port.yaml) with UDP and
TCP port mapping rules, for the nginx-ingress-controller application, to
expose the SNMP services on the required ports. Use external ports 161/UDP
and 162/TCP.
**kube-system/snmpd-service:161** is the standard SNMP Agent's UDP port for
receiving SNMP requests. It should be configured as mapped to external UDP
port **161**, the default for SNMP Agents. This port can be modified, see,
:ref:`Modifying 161/UDP port <modifying-161udp-port>` procedure for details
on modifying this port.
**kube-system/snmpd-service:162** is used internally by the SNMP
application to receive trap info from |prod|. It should be configured
as mapped to external TCP port **162**. This port can be modified, see
:ref:`Modifying 162/TCP port <modifying-162tcp-port>` procedure below for
details on modifying this port.
.. code-block:: none
cat <<EOF > snmp_port.yaml
udp:
161: "kube-system/snmpd-service:161"
tcp:
162: "kube-system/snmpd-service:162"
EOF
#. Update the values of the SNMP port mappings in the Helm Chart for the
nginx-ingress-controller application.
.. code-block:: none
~(keystone_admin)$ system helm-override-update --values snmp_port.yaml nginx-ingress-controller nginx-ingress kube-system
+----------------+------------------------------------------+
| Property | Value |
+----------------+------------------------------------------+
| name | nginx-ingress |
| namespace | kube-system |
| user_overrides | tcp: |
| | "162": kube-system/snmpd-service:162 |
| | udp: |
| | "161": kube-system/snmpd-service:161 |
| | |
+----------------+------------------------------------------+
#. Apply the changes to the nginx-ingress-controller application.
.. code-block:: none
~(keystone_admin)$ system application-apply nginx-ingress-controller
+---------------+----------------------------------------+
| Property | Value |
+---------------+----------------------------------------+
| active | True |
| app_version | 1.0-0 |
| created_at | 2020-10-19T04:59:40.505583+00:00 |
| manifest_file | nginx_ingress_controller_manifest.yaml |
| manifest_name | nginx-ingress-controller-manifest |
| name | nginx-ingress-controller |
| progress | None |
| status | applying |
| updated_at | 2020-11-10T17:27:21.509548+00:00 |
+---------------+----------------------------------------+
.. _change-configuration-of-SNMP:
--------------------------------------------
Change configuration of the SNMP application
--------------------------------------------
If the SNMP application is already applied, use the following procedures to
update its configuration.
.. rubric:: |proc|
#. Create a Helm chart values file (for example, user_conf.yaml) with the
definition of the **configmap:user_conf** attribute defining your SNMP
configuration of V2 communities, V2 trap sinks, V3 users and/or V3 trap
sessions, as shown in the example below. The **configmap:user_conf**
variable in the SNMP Helm chart is a multi-line variable that follows the
syntax of Net-SNMP's snmpd.conf file for configuring the SNMP agent, see
`http://www.net-snmp.org/docs/man/snmpd.conf.html <http://www.net-snmp.org/docs/man/snmpd.conf.html>`__,
for a detailed description of the commands.
.. warning::
Since this file may contain sensitive security information, this file
should be removed from the system after executing the command and
stored off-box, or regenerated, if required.
.. code-block:: none
cat <<EOF > user_conf.yaml
configmap:
user_conf: |-
# Configure V2 Community
# rocommunity COMMUNITY [SOURCE [OID | -V VIEW [CONTEXT]]]
rocommunity testcommunity default -V all
# Configure V2 Trap Sink / Destination
# trap2sink HOST [COMMUNITY [PORT]]
trap2sink 10.10.10.1:162 testcommunity
# Configure V3 User
# createUser [-e ENGINEID] username (MD5|SHA) authpassphrase [DES|AES] [privpassphrase]
createUser testuser MD5 testpassword DES
# Configure RW access for V3 User
# rouser [-s SECMODEL] USER [noauth|auth|priv [OID | -V VIEW [CONTEXT]]]
rouser testuser priv
# Configure V3 Trap Session / Destination
# trapsess -v 3 -u USER -a [MD5|SHA] -A authpassphrase -l [noauth|auth|priv] -x [DES|AES] -X privpassphrase [<transport-specifier>:]<transport-address>
trapsess -v 3 -u testuser -a MD5 -A testpassword -l authPriv -x DES -X testpassword udp:10.10.10.1:162
EOF
**\(Optional\)** You can add your own EngineID value, instead of having it
auto-created. This keeps the EngineID value the same, even when the SNMP
application restarts. The EngineID is required if you are using an SNMP
trap viewer or SNMP monitoring tool. Add the 'engineID [STRING]' value in
the **configmap:user_conf** variable, as shown below.
.. code-block:: none
cat <<EOF > user_conf.yaml
configmap:
user_conf: |-
...
engineID [STRING]
...
EOF
#. Update the values of the **configmap:user_conf** attribute on the Helm
chart using the following command.
.. code-block:: none
~(keystone_admin)$ system helm-override-update --values user_conf.yaml snmp snmp kube-system
+----------------+------------------------------------------------------------------------------------------------------------+
| Property | Value |
+----------------+------------------------------------------------------------------------------------------------------------+
| name | snmp |
| namespace | kube-system |
| user_overrides | configmap: |
| | user_conf: |- |
| | createUser testuser MD5 testpassword DES |
| | rouser testuser priv |
| | rocommunity testcommunity default -V all |
| | trapsess -v 3 -u testuser -a MD5 -A testpassword -l authPriv -x DES -X testpassword udp:10.10.10.1:162 |
| | trap2sink 10.10.10.1:162 testcommunity |
+----------------+------------------------------------------------------------------------------------------------------------+
#. Apply the changes.
.. code-block:: none
~(keystone_admin)$ system application-apply snmp
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | True |
| app_version | 1.0-2 |
| created_at | 2020-11-10T23:25:47.226453+00:00 |
| manifest_file | snmp_manifest.yaml |
| manifest_name | snmp-manifest |
| name | snmp |
| progress | None |
| status | applying |
| updated_at | 2020-11-10T23:30:59.276031+00:00 |
+---------------+----------------------------------+
.. _modifying-161udp-port:
**********************
Modifying 161/UDP port
**********************
Modify the external UDP port used for receiving SNMP requests.
.. note::
After changing this external UDP port, any external SNMP managers being
used must be updated to send their SNMP requests to |prod| using this
UDP port, instead of the default UDP port 161.
.. rubric:: |proc|
#. Create a Helm chart values file (for example, snmp_port.yaml) with
external ports 161/UDP and 162/TCP port mapping definitions, for the SNMP
services for the nginx-ingress-controller's Helm chart, as shown in the
example below.
Update the external port in the UDP port mapping for internal port
**kube-system/snmpd-service:161**. The example below shows the external
port updated to **1061**.
.. code-block:: none
cat <<EOF > snmp_port.yaml
udp:
1061: "kube-system/snmpd-service:161"
tcp:
162: "kube-system/snmpd-service:162"
EOF
#. Update the values of the SNMP ports on the Helm chart for the
nginx-ingress-controller application.
.. code-block:: none
~(keystone_admin)$ system helm-override-update --values snmp_port.yaml nginx-ingress-controller nginx-ingress kube-system
+----------------+------------------------------------------+
| Property | Value |
+----------------+------------------------------------------+
| name | nginx-ingress |
| namespace | kube-system |
| user_overrides | tcp: |
| | "162": kube-system/snmpd-service:162 |
| | udp: |
| | "1061": kube-system/snmpd-service:161 |
| | |
+----------------+------------------------------------------+
#. Apply the changes in the nginx-ingress-controller application.
.. code-block:: none
~(keystone_admin)$ system application-apply nginx-ingress-controller
+---------------+----------------------------------------+
| Property | Value |
+---------------+----------------------------------------+
| active | True |
| app_version | 1.0-0 |
| created_at | 2020-10-19T04:59:40.505583+00:00 |
| manifest_file | nginx_ingress_controller_manifest.yaml |
| manifest_name | nginx-ingress-controller-manifest |
| name | nginx-ingress-controller |
| progress | None |
| status | applying |
| updated_at | 2020-11-10T17:27:21.509548+00:00 |
+---------------+----------------------------------------+
.. _modifying-162tcp-port:
**********************
Modifying 162/TCP port
**********************
Modify the external port used by the SNMP application to receive trap
information from |prod|.
To change the port 162/TCP, you need to modify both the SNMP application Helm
chart, and the nginx-ingress-controller Helm chart. The new port must be set
to the same port number in the two Helm charts, otherwise SNMP
traps will not be generated.
.. rubric:: |proc|
#. Modify your SNMP Helm chart values file (for example, user_conf.yaml) by
adding the line "trap-server-port: [new port]" as shown in the example
below \("30162" is the new port in this example\).
.. code-block:: none
cat <<EOF > user_conf.yaml
configmap:
user_conf: |-
createUser testuser MD5 testpassword DES
rouser testuser priv
rocommunity testcommunity default -V all
trapsess -v 3 -u testuser -a MD5 -A testpassword -l authPriv -x DES -X testpassword udp:10.10.10.1:162
trap2sink 10.10.10.1:162 testcommunity
trap-server-port: 30162
EOF
#. Run the following commands to apply the configuration.
.. code-block:: none
~(keystone_admin)$ system helm-override-update --values user_conf.yaml snmp snmp kube-system
~(keystone_admin)$ system application-apply snmp
#. Modify your nginx ingress controller Helm chart values file
(for example, snmp_port.yaml). Update the external port in the TCP port
mapping for internal port **kube-system/snmpd-service:162**. The example
below shows the external port updated to **30162**.
- The new port number must match the port number specified in your SNMP
Helm chart values file (for example, user_conf.yaml).
- Do not modify port number "162" in **kube-system/snmpd-service:162**.
.. code-block:: none
cat <<EOF > snmp_port.yaml
udp:
161: "kube-system/snmpd-service:161"
tcp:
30162: "kube-system/snmpd-service:162"
EOF
#. Run the following commands to apply the configuration.
.. code-block:: none
~(keystone_admin)$ system helm-override-update --values snmp_port.yaml nginx-ingress-controller nginx-ingress kube-system
~(keystone_admin)$ system application-apply nginx-ingress-controller

View File

@ -42,11 +42,7 @@
- :ref:`SNMP Overview <snmp-overview>` - :ref:`SNMP Overview <snmp-overview>`
- :ref:`Enabling SNMP Support <enabling-snmp-support>` - :ref:`Enabling SNMP Support <enabling-snmp-support>`
- :ref:`Traps <traps>` - :ref:`Traps <traps>`
- :ref:`Configuring SNMP Trap Destinations <configuring-snmp-trap-destinations>`
- :ref:`SNMP Event Table <snmp-event-table>` - :ref:`SNMP Event Table <snmp-event-table>`
- :ref:`Adding an SNMP Community String Using the CLI <adding-an-snmp-community-string-using-the-cli>`
- :ref:`Setting SNMP Identifying Information <setting-snmp-identifying-information>` - :ref:`Setting SNMP Identifying Information <setting-snmp-identifying-information>`
- :ref:`Troubleshooting Log Collection <troubleshooting-log-collection>` - :ref:`Troubleshooting Log Collection <troubleshooting-log-collection>`

View File

@ -87,11 +87,10 @@ SNMP
snmp-overview snmp-overview
enabling-snmp-support enabling-snmp-support
traps traps
configuring-snmp-trap-destinations
snmp-active-alarm-table snmp-active-alarm-table
snmp-event-table snmp-event-table
adding-an-snmp-community-string-using-the-cli
setting-snmp-identifying-information setting-snmp-identifying-information
uninstalling-snmp
********************************** **********************************
Distributed Cloud alarm management Distributed Cloud alarm management

View File

@ -37,46 +37,46 @@ system. Each entry in the table includes the following variables:
An external SNMP Manager can examine the Active Alarm table contents by doing An external SNMP Manager can examine the Active Alarm table contents by doing
an SNMP Walk of the table. an SNMP Walk of the table.
For example, below is the output for a simple :command:`snmpwalk` cli tool For example, below is the output for a simple **snmpwalk** CLI tool showing a
showing a table with three rows (i.e. three active alarms). table with three rows (i.e. three active alarms).
.. code-block:: none .. code-block:: none
$ snmpwalk -v2c -c public udp:10.10.10.2:161 WRS-ALARM-MIB::wrsAlarmActiveTable $ snmpwalk -v2c -c public udp:10.10.10.2:161 WRS-ALARM-MIB::wrsAlarmActiveTable
WRS-ALARM-MIB::wrsAlarmActiveIndex.1 = Wrong Type (should be Gauge32 or Unsigned32): INTEGER: 1 WRS-ALARM-MIB::wrsAlarmActiveIndex.1 = Wrong Type (should be Gauge32 or Unsigned32): INTEGER: 1
WRS-ALARM-MIB::wrsAlarmActiveIndex.2 = Wrong Type (should be Gauge32 or Unsigned32): INTEGER: 2 WRS-ALARM-MIB::wrsAlarmActiveIndex.2 = Wrong Type (should be Gauge32 or Unsigned32): INTEGER: 2
WRS-ALARM-MIB::wrsAlarmActiveIndex.3 = Wrong Type (should be Gauge32 or Unsigned32): INTEGER: 3 WRS-ALARM-MIB::wrsAlarmActiveIndex.3 = Wrong Type (should be Gauge32 or Unsigned32): INTEGER: 3
WRS-ALARM-MIB::wrsAlarmActiveUuid.1 = STRING: 742c2d64-df2e-4feb-8607-1ae6de11f15 WRS-ALARM-MIB::wrsAlarmActiveUuid.1 = STRING: 742c2d64-df2e-4feb-8607-1ae6de11f15
WRS-ALARM-MIB::wrsAlarmActiveUuid.2 = STRING: 742c2d64-df2e-4feb-8607-1ae6de11f15 WRS-ALARM-MIB::wrsAlarmActiveUuid.2 = STRING: 742c2d64-df2e-4feb-8607-1ae6de11f15
WRS-ALARM-MIB::wrsAlarmActiveUuid.3 = STRING: 742c2d64-df2e-4feb-8607-1ae6de11f15 WRS-ALARM-MIB::wrsAlarmActiveUuid.3 = STRING: 742c2d64-df2e-4feb-8607-1ae6de11f15
WRS-ALARM-MIB::wrsAlarmActiveAlarmId.1 = STRING: "100.114" WRS-ALARM-MIB::wrsAlarmActiveAlarmId.1 = STRING: "100.114"
WRS-ALARM-MIB::wrsAlarmActiveAlarmId.2 = STRING: "100.114" WRS-ALARM-MIB::wrsAlarmActiveAlarmId.2 = STRING: "100.114"
WRS-ALARM-MIB::wrsAlarmActiveAlarmId.3 = STRING: "100.114" WRS-ALARM-MIB::wrsAlarmActiveAlarmId.3 = STRING: "100.114"
WRS-ALARM-MIB::wrsAlarmActiveEntityInstanceId.1 = STRING: system=7dd633ba-96f9-47ef-8531-983e4ca89fa3.host=controller-0.ntp WRS-ALARM-MIB::wrsAlarmActiveEntityInstanceId.1 = STRING: system=7dd633ba-96f9-47ef-8531-983e4ca89fa3.host=controller-0.ntp
WRS-ALARM-MIB::wrsAlarmActiveEntityInstanceId.2 = STRING: system=7dd633ba-96f9-47ef-8531-983e4ca89fa3.host=controller-0.ntp=162.159.200.123 WRS-ALARM-MIB::wrsAlarmActiveEntityInstanceId.2 = STRING: system=7dd633ba-96f9-47ef-8531-983e4ca89fa3.host=controller-0.ntp=162.159.200.123
WRS-ALARM-MIB::wrsAlarmActiveEntityInstanceId.3 = STRING: system=7dd633ba-96f9-47ef-8531-983e4ca89fa3.host=controller-0.ntp=213.199.225.40 WRS-ALARM-MIB::wrsAlarmActiveEntityInstanceId.3 = STRING: system=7dd633ba-96f9-47ef-8531-983e4ca89fa3.host=controller-0.ntp=213.199.225.40
WRS-ALARM-MIB::wrsAlarmActiveDateAndTime.1 = STRING: 2020-11-11,13:8:4.0,+0:0 WRS-ALARM-MIB::wrsAlarmActiveDateAndTime.1 = STRING: 2020-11-11,13:8:4.0,+0:0
WRS-ALARM-MIB::wrsAlarmActiveDateAndTime.2 = STRING: 2020-11-13,13:13:53.0,+0:0 WRS-ALARM-MIB::wrsAlarmActiveDateAndTime.2 = STRING: 2020-11-13,13:13:53.0,+0:0
WRS-ALARM-MIB::wrsAlarmActiveDateAndTime.3 = STRING: 2020-11-13,13:13:53.0,+0:0 WRS-ALARM-MIB::wrsAlarmActiveDateAndTime.3 = STRING: 2020-11-13,13:13:53.0,+0:0
WRS-ALARM-MIB::wrsAlarmActiveAlarmSeverity.1 = INTEGER: major(3) WRS-ALARM-MIB::wrsAlarmActiveAlarmSeverity.1 = INTEGER: major(3)
WRS-ALARM-MIB::wrsAlarmActiveAlarmSeverity.2 = INTEGER: minor(2) WRS-ALARM-MIB::wrsAlarmActiveAlarmSeverity.2 = INTEGER: minor(2)
WRS-ALARM-MIB::wrsAlarmActiveAlarmSeverity.3 = INTEGER: minor(2) WRS-ALARM-MIB::wrsAlarmActiveAlarmSeverity.3 = INTEGER: minor(2)
WRS-ALARM-MIB::wrsAlarmActiveReasonText.1 = STRING: NTP configuration does not contain any valid or reachable NTP servers. WRS-ALARM-MIB::wrsAlarmActiveReasonText.1 = STRING: NTP configuration does not contain any valid or reachable NTP servers.
WRS-ALARM-MIB::wrsAlarmActiveReasonText.2 = STRING: NTP address 162.159.200.123 is not a valid or a reachable NTP server. WRS-ALARM-MIB::wrsAlarmActiveReasonText.2 = STRING: NTP address 162.159.200.123 is not a valid or a reachable NTP server.
WRS-ALARM-MIB::wrsAlarmActiveReasonText.3 = STRING: NTP address 213.199.225.40 is not a valid or a reachable NTP server. WRS-ALARM-MIB::wrsAlarmActiveReasonText.3 = STRING: NTP address 213.199.225.40 is not a valid or a reachable NTP server.
WRS-ALARM-MIB::wrsAlarmActiveEventType.1 = INTEGER: operationalViolation(7) WRS-ALARM-MIB::wrsAlarmActiveEventType.1 = INTEGER: operationalViolation(7)
WRS-ALARM-MIB::wrsAlarmActiveEventType.2 = INTEGER: operationalViolation(7) WRS-ALARM-MIB::wrsAlarmActiveEventType.2 = INTEGER: operationalViolation(7)
WRS-ALARM-MIB::wrsAlarmActiveEventType.3 = INTEGER: operationalViolation(7) WRS-ALARM-MIB::wrsAlarmActiveEventType.3 = INTEGER: operationalViolation(7)
WRS-ALARM-MIB::wrsAlarmActiveProbableCause.1 = INTEGER: threshold-crossed(50) WRS-ALARM-MIB::wrsAlarmActiveProbableCause.1 = INTEGER: threshold-crossed(50)
WRS-ALARM-MIB::wrsAlarmActiveProbableCause.2 = INTEGER: threshold-crossed(50) WRS-ALARM-MIB::wrsAlarmActiveProbableCause.2 = INTEGER: threshold-crossed(50)
WRS-ALARM-MIB::wrsAlarmActiveProbableCause.3 = INTEGER: threshold-crossed(50) WRS-ALARM-MIB::wrsAlarmActiveProbableCause.3 = INTEGER: threshold-crossed(50)
WRS-ALARM-MIB::wrsAlarmActiveProposedRepairAction.1 = STRING: Monitor and if condition persists, contact next level of support. WRS-ALARM-MIB::wrsAlarmActiveProposedRepairAction.1 = STRING: Monitor and if condition persists, contact next level of support.
WRS-ALARM-MIB::wrsAlarmActiveProposedRepairAction.2 = STRING: Monitor and if condition persists, contact next level of support. WRS-ALARM-MIB::wrsAlarmActiveProposedRepairAction.2 = STRING: Monitor and if condition persists, contact next level of support.
WRS-ALARM-MIB::wrsAlarmActiveProposedRepairAction.3 = STRING: Monitor and if condition persists, contact next level of support. WRS-ALARM-MIB::wrsAlarmActiveProposedRepairAction.3 = STRING: Monitor and if condition persists, contact next level of support.
WRS-ALARM-MIB::wrsAlarmActiveServiceAffecting.1 = INTEGER: false(0) WRS-ALARM-MIB::wrsAlarmActiveServiceAffecting.1 = INTEGER: false(0)
WRS-ALARM-MIB::wrsAlarmActiveServiceAffecting.2 = INTEGER: false(0) WRS-ALARM-MIB::wrsAlarmActiveServiceAffecting.2 = INTEGER: false(0)
WRS-ALARM-MIB::wrsAlarmActiveServiceAffecting.3 = INTEGER: false(0) WRS-ALARM-MIB::wrsAlarmActiveServiceAffecting.3 = INTEGER: false(0)
WRS-ALARM-MIB::wrsAlarmActiveSuppressionAllowed.1 = INTEGER: true(1) WRS-ALARM-MIB::wrsAlarmActiveSuppressionAllowed.1 = INTEGER: true(1)
WRS-ALARM-MIB::wrsAlarmActiveSuppressionAllowed.2 = INTEGER: true(1) WRS-ALARM-MIB::wrsAlarmActiveSuppressionAllowed.2 = INTEGER: true(1)
WRS-ALARM-MIB::wrsAlarmActiveSuppressionAllowed.3 = INTEGER: true(1) WRS-ALARM-MIB::wrsAlarmActiveSuppressionAllowed.3 = INTEGER: true(1)

View File

@ -43,11 +43,11 @@ Each entry in the table includes the following variables:
An external SNMP Manager can examine the Event table contents by doing an SNMP An external SNMP Manager can examine the Event table contents by doing an SNMP
Walk of the table. Walk of the table.
For example, below is the output for a simple :command:`snmpwalk` cli tool. For example, below is the output for a simple :command:`snmpwalk` CLI tool.
.. code-block:: none .. code-block:: none
$ snmpwalk -v2c -c public udp:10.10.10.2:161 WRS-ALARM-MIB::wrsEventTable $ snmpwalk -v2c -c public udp:10.10.10.2:161 WRS-ALARM-MIB::wrsEventTable
WRS-ALARM-MIB::wrsEventIndex.1 = Wrong Type (should be Gauge32 or Unsigned32): INTEGER: 1 WRS-ALARM-MIB::wrsEventIndex.1 = Wrong Type (should be Gauge32 or Unsigned32): INTEGER: 1
WRS-ALARM-MIB::wrsEventIndex.2 = Wrong Type (should be Gauge32 or Unsigned32): INTEGER: 2 WRS-ALARM-MIB::wrsEventIndex.2 = Wrong Type (should be Gauge32 or Unsigned32): INTEGER: 2

View File

@ -6,16 +6,16 @@
SNMP Overview SNMP Overview
============= =============
|prod| can generate :abbr:`SNMP (Simple Network Management Protocol)` traps for |prod| provides a containerized |SNMP| solution using Net-SNMP, supporting both
|prod| Alarm Events and Customer Log Events. SNMPv2c and SNMPv3.
|prod| can generate SNMP traps for Alarm Events and Customer Log Events.
|prod| also supports SNMP GETs and WALKs of an Active Alarm table and a |prod| also supports SNMP GETs and WALKs of an Active Alarm table and a
historical Event (alarm SET/CLEAR and log) table. historical Event (alarm SET/CLEAR and log) table.
This includes alarms based on hardware sensors monitored by board management SNMP functionality is integrated into |prod| as an optionally configurable
controllers. system application.
.. xreflink For more information, see |node-doc|: :ref:`Sensors Tab <sensors-tab>`.
.. contents:: .. contents::
:local: :local:
@ -31,22 +31,24 @@ Support for Simple Network Management Protocol \(SNMP\) is implemented as follow
.. _snmp-overview-ul-bjv-cjd-cp: .. _snmp-overview-ul-bjv-cjd-cp:
- access is disabled by default, must be enabled manually from the command
line interface
- available using the controller's node floating OAM IP address, over the - available using the controller's node floating OAM IP address, over the
standard SNMP UDP port 161 standard SNMP UDP port 161, by default
- supported version is SNMPv2c - SNMPv2c and SNMPv3 are supported versions
- access is read-only for all SNMP communities - read-only access for all SNMP communities or all SNMPv3 users
- all SNMP communities have access to the entire OID tree, there is no - supported SNMP operations are GET, GETNEXT, GETBULK, SNMPv2C-TRAP2,
support for VIEWS SNMPv3-TRAP
- supported SNMP operations are GET, GETNEXT, GETBULK, and SNMPv2C-TRAP2 .. note::
SNMPv3 INFORM, and SNMP SET operations are not supported.
- the SNMP SET operation is not supported - the SNMPv3 security levels that are supported are:
NoAuthNoPriv, authNoPriv, authPriv
- support for MD5 for auth, and DES for priv, see
`http://www.net-snmp.org/ <http://www.net-snmp.org/>`__
For information on enabling SNMP support, see For information on enabling SNMP support, see
:ref:`Enabling SNMP Support <enabling-snmp-support>`. :ref:`Enabling SNMP Support <enabling-snmp-support>`.
@ -62,45 +64,14 @@ and SNMP groups, as follows:
.. _snmp-overview-ul-ulb-ypl-hp: .. _snmp-overview-ul-ulb-ypl-hp:
- System Group, **.iso.org.dod.internet.mgmt.mib-2.system** - System Group: **.iso.org.dod.internet.mgmt.mib-2.system**
- SNMP Group, **.iso.org.dod.internet.mgmt.mib-2.snmp** - SNMP Group: **.iso.org.dod.internet.mgmt.mib-2.snmp**
- coldStart and warmStart Traps - coldStart and warmStart Traps
The following system attributes are used in support of the SNMP implementation. - support for Enterprise Registration and Alarm MIBs, see
They can be displayed using the :command:`system show` command. `https://opendev.org/starlingx/snmp-armada-app/src/branch/master/stx-snmp-helm/centos/docker/stx-snmp/mibs <https://opendev.org/starlingx/snmp-armada-app/src/branch/master/stx-snmp-helm/centos/docker/stx-snmp/mibs>`__
**contact**
A read-write system attribute used to populate the **sysContact** attribute
of the SNMP System group.
**location**
A read-write system attribute used to populate the **sysLocation** attribute
of the SNMP System group.
**name**
A read-write system attribute used to populate the **sysName** attribute of
the SNMP System group.
**software\_version**
A read-only system attribute set automatically by the system. Its value is
used to populate the **sysDescr** attribute of the SNMP System group.
For information on setting the **sysContact**, **sysLocation**, and **sysName**
attributes, see
:ref:`Setting SNMP Identifying Information <setting-snmp-identifying-information>`.
The following SNMP attributes are used as follows:
**sysObjectId**
Set to **iso.org.dod.internet.private.enterprise.wrs.titanium** \(1.3.6.1.4.1.1.2\).
**sysUpTime**
Set to the up time of the active controller.
**sysServices**
Set to the nominal value of 72 to indicate that the host provides services at layers 1 to 7.
.. _snmp-overview-section-N100C9-N1001F-N10001: .. _snmp-overview-section-N100C9-N1001F-N10001:

View File

@ -28,17 +28,19 @@ The following traps are defined.
- **wrsAlarmHierarchicalClear** - **wrsAlarmHierarchicalClear**
|prod| alarms are mapped into one of the above traps based on severity, and
customer logs are mapped into the 'Message' trap.
.. note:: .. note::
Customer Logs always result in **wrsAlarmMessage** traps. |prod| uses Wind |prod| uses Wind River Systems (**wrs**) Enterprise Registration and Alarm
River Systems (**wrs**) Enterprise Registration and Alarm MIBs. See MIBs. See :ref:`SNMP Overview <snmp-overview>` for details.
:ref:`SNMP Overview <snmp-overview>` for details.
For Critical, Major, Minor, Warning, and Message traps, all variables in the For Critical, Major, Minor, Warning, and Message traps, all variables in the
active alarm table are included as varbinds \(variable bindings\), where each active alarm table are included as varbinds \(variable bindings\), where each
varbind is a pair of fields consisting of an object identifier and a value varbind is a pair of fields consisting of an object identifier and a value
for the object. for the object.
For the Clear trap, varbinds include only the following variables: For the Clear trap, **varbinds** include only the following variables:
.. _traps-ul-uks-byn-nkb: .. _traps-ul-uks-byn-nkb:
@ -50,7 +52,8 @@ For the Clear trap, varbinds include only the following variables:
- <ReasonText> - <ReasonText>
For the HierarchicalClear trap, varbinds include only the following variables: For the HierarchicalClear trap, **varbinds** include only the following
variables:
.. _traps-ul-isn-fyn-nkb: .. _traps-ul-isn-fyn-nkb:
@ -62,5 +65,5 @@ For the HierarchicalClear trap, varbinds include only the following variables:
For all alarms, the Notification Type is based on the severity of the trap or For all alarms, the Notification Type is based on the severity of the trap or
alarm. This is done to facilitate the interaction with most SNMP trap viewers alarm. This is done to facilitate the interaction with most SNMP trap viewers
which typically use the Notification Type to drive the coloring of traps, that which use the Notification Type to drive the coloring of traps, that is, red
is, red for critical, yellow for minor, and so on. for critical, yellow for minor, and so on.

View File

@ -0,0 +1,95 @@
==============
Uninstall SNMP
==============
Use the following procedure to uninstall |SNMP|:
.. rubric:: |proc|
#. Run the following command to check if the SNMP application is installed
(status "applied").
.. code-block:: none
~(keystone_admin)]$ system application-list
+--------------------------+-------------------------+-----------------------------------+------------------------+---------------+-----------+
| application | version | manifest name | manifest file | status | progress |
+--------------------------+-------------------------+-----------------------------------+------------------------+---------------+-----------+
| cert-manager | 1.0-6 | cert-manager-manifest | certmanager-manifest. | applied | completed |
| | | | yaml | | |
| | | | | | |
| nginx-ingress-controller | 1.0-0 | nginx-ingress-controller-manifest | nginx_ingress_controll | applied | completed |
| | | | er_manifest.yaml | | |
| | | | | | |
| oidc-auth-apps | 1.0-29 | oidc-auth-manifest | manifest.yaml | uploaded | completed |
| platform-integ-apps | 1.0-9 | platform-integration-manifest | manifest.yaml | uploaded | completed |
| rook-ceph-apps | app-version-placeholder | manifest-placeholder | tarfile-placeholder | upload-failed | None |
| snmp | 1.0-2 | snmp-manifest | snmp_manifest.yaml | applied | completed |
+--------------------------+-------------------------+-----------------------------------+------------------------+---------------+-----------+
#. Uninstall SNMP.
.. code-block:: none
~(keystone_admin)]$ system application-remove snmp
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | True |
| app_version | 1.0-2 |
| created_at | 2020-11-12T06:26:21.526433+00:00 |
| manifest_file | snmp_manifest.yaml |
| manifest_name | snmp-manifest |
| name | snmp |
| progress | None |
| status | removing |
| updated_at | 2020-11-12T06:51:34.987085+00:00 |
+---------------+----------------------------------+
The SNMP application is removed, but still shows as "Uploaded".
.. code-block:: none
~(keystone_admin)]$ system application-list
+--------------------------+-------------------------+-----------------------------------+-------------------------+---------------+-----------+
| application | version | manifest name | manifest file | status | progress |
+--------------------------+-------------------------+-----------------------------------+-------------------------+---------------+-----------+
| cert-manager | 1.0-6 | cert-manager-manifest | certmanager-manifest. | applied | completed |
| | | | yaml | | |
| | | | | | |
| nginx-ingress-controller | 1.0-0 | nginx-ingress-controller-manifest | nginx_ingress_controlle | applied | completed |
| | | | r_manifest.yaml | | |
| | | | | | |
| oidc-auth-apps | 1.0-29 | oidc-auth-manifest | manifest.yaml | uploaded | completed |
| platform-integ-apps | 1.0-9 | platform-integration-manifest | manifest.yaml | uploaded | completed |
| rook-ceph-apps | app-version-placeholder | manifest-placeholder | tarfile-placeholder | upload-failed | None |
| snmp | 1.0-2 | snmp-manifest | snmp_manifest.yaml | uploaded | completed |
+--------------------------+-------------------------+-----------------------------------+-------------------------+---------------+-----------+
#. Delete the uninstalled SNMP application definitions from the system.
.. code-block:: none
~(keystone_admin)]$ system application-delete snmp
The following message is displayed when the SNMP application is deleted
"Application SNMP deleted".
#. Run the following command to check if the SNMP application is deleted.
.. code-block:: none
~(keystone_admin)]$ system application-list
+--------------------------+-------------------------+-----------------------------------+--------------------------+---------------+-----------+
| application | version | manifest name | manifest file | status | progress |
+--------------------------+-------------------------+-----------------------------------+--------------------------+---------------+-----------+
| cert-manager | 1.0-6 | cert-manager-manifest | certmanager-manifest. | applied | completed |
| | | | yaml | | |
| | | | | | |
| nginx-ingress-controller | 1.0-0 | nginx-ingress-controller-manifest | nginx_ingress_controller | applied | completed |
| | | | _manifest.yaml | | |
| | | | | | |
| oidc-auth-apps | 1.0-29 | oidc-auth-manifest | manifest.yaml | uploaded | completed |
| platform-integ-apps | 1.0-9 | platform-integration-manifest | manifest.yaml | uploaded | completed |
| rook-ceph-apps | app-version-placeholder | manifest-placeholder | tarfile-placeholder | upload-failed | None |
+--------------------------+-------------------------+-----------------------------------+--------------------------+---------------+-----------+