starlingx/metal
Code Issues Proposed changes

Refactor BMC provisioning in Maintenance

Browse Source 
The current mechanism used to preserve the learned bmc protocol in
the filesystem on the active controller is problematic over swact.

This update removes the file storage method in favor of preserving
the learned protocol in the system inventory database as a key/value
pair at the host level in already existing mtce_info database field.

The specified or learned bmc access protocol is then shared with the
hardware monitor through inter-daemon maintenance messaging.

This update refactors bmc provisioning to accommodate bmc protocol
selection at the host rather than system level. Towards that this
update removes system level bmc_access_method selection in favor of
host level selection through bm_type. A bm_type of 'bmc' specifies
that the bmc access protocol for that host be learned. This has the
effect of making it the same as what is delivered today but without
support for changing it as the system level.

A system inventory update will be delivered shortly that enables bmc
access protocol selection at the host level. That update allows the
customer to specify the bmc access protocol at the host level to be
either dynamic (aka learned) or to only use 'redfish' or 'ipmi'.
That system inventory update delivers that information to maintenance
through bm_type via bmc provisioning. Until that update is delivered
bm_type always comes in as 'bmc' which get interpreted as 'dynamic'
to maintain existing configuration.

The following additional issues were also fixed in this update.

1. The nodeTimers module defaults the 'ring' member of timers that are
   not running to false but should be true.

2. Added a pingUtil_restart function to facilitate quicker sensor
   monitoring following provisioning changes and bmc access failures.

3. Enhanced the hardware monitor sensor grouping filter to accommodate
   non-standard Redfish readout labelling so that more sensors fall
   into the existing canned groups ; leads to more monitored sensors.

4. Added a 'http security mode' to hardware monitor messaging. This
   defaults to https as that is all that is supported by the Redfish
   implementation today. This field can be used to specify non-secure
   'http' mode in the future when that gets implemented.

5. Ensure the hardware monitor performs a bmc password re-fetch on every
   provisioning change.

Test Plan:

PASS: Verify bmc access protocol store/fetched from the database (mtce_info)
PASS: Verify inventory push from mtcAgent to hwmond over mtcAgent restart
PASS: Verify inventory push from mtcAgent to hwmond over hwmon restart
PASS: Verify bmc provisioning of ipmi and redfish servers
PASS: Verify learned bmc protocol persists over process restart and swact
PASS: Verify process startup with protocol already learned

Hardware Monitor:

PASS: Verify bmc_type=ipmi handling ; protocol forced to ipmi ; (re)prov
PASS: Verify bmc_type=redfish handling ; protocol forced to redfish ; (re)prov
PASS: Verify bmc_type=dynamic handling ; protocol is learned then persisted
PASS: Verify sensor model delete and relearn over ip address change
PASS: Verify sensor model delete and relearn over bm_type change change
PASS: Verify sensor model not relearned username change
PASS: Verify bm pw is re-fetched over any (re)provisioning change
PASS: Verify bmc re-provisioning soak (test-bmc-reprovisioning.sh 50 loops)
PASS: Verify protocol change handling, file cleanup, model recreation
PASS: Verify End-2-End behavior for bm_type change from redfish to ipmi
PASS: Verify End-2-End behavior for bm_type change from ipmi to redfish
PASS: Verify End-2-End behavior for bm_type change from redfish to dynamic
PASS: Verify End-2-End behavior for bm_type change from ipmi to dynamic
PASS: Verify End-2-End behavior for bm_type change from dynamic to ipmi
PASS: Verify End-2-End behavior for bm_type change from dynamic to redfish
PASS: Verify sensor model creation waits for server power to be on
PASS: Verify sensor relearn by provisioning change during model creation. (soak)

Regression:

PASS: Verify host power off and on.
PASS: Verify BMC access alarm handling (assert and clear)
PASS: Verify mtcAgent and hwmond logs add value
PASS: Verify no core dumps / seg faults.
PASS: Verify no mtcAgent and hwmond memory leak.
PASS: Verify delete of BMC provisioned host
PASS: Verify sensor monitoring, alarming, degrade and then clear cycle
PASS: Verify static analysis report of changed modules.
PASS: Verify host level bm_type=bmc functions as would dynamic selection
PASS: Verify batch provisioning and deprovisioning (7 nodes)
PASS: Verify batch provisioning to different protocol (5 nodes)
PASS: Verify handling of flaky Redfish responses

PEND: Verify System Install

Change-Id: Ic224a9c33e0283a611725b33c90009132cab3382
Closes-Bug: #1853471
Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>
This commit is contained in:
Eric MacDonald 2019-12-04 10:37:01 -05:00
parent eb851a9da0
commit c4b8171ddd
33 changed files with 930 additions and 894 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

171
mtce-common/src/common/bmcUtil.cpp
View File

@ -90,11 +90,8 @@ string bmcUtil_getProtocol_str ( bmc_protocol_enum protocol )
{
case BMC_PROTOCOL__REDFISHTOOL: return(BMC_PROTOCOL__REDFISHTOOL_STR);
case BMC_PROTOCOL__IPMITOOL: return(BMC_PROTOCOL__IPMITOOL_STR);
default:
{
blog ("unknown bmc protocol %d", protocol );
return("unknown");
}
case BMC_PROTOCOL__DYNAMIC: return(BMC_PROTOCOL__DYNAMIC_STR);
default: return(NONE);
}
}
@ -140,8 +137,6 @@ int bmcUtil_init ( void )
{
if ( daemon_is_file_present ( BMC_OUTPUT_DIR ) == false )
daemon_make_dir(BMC_OUTPUT_DIR) ;
if ( daemon_is_file_present ( BMC_HWMON_TMP_DIR ) == false )
daemon_make_dir(BMC_HWMON_TMP_DIR) ;
ipmiUtil_init ();
redfishUtil_init ();
@ -208,159 +203,6 @@ void bmcUtil_info_init ( bmc_info_type & bmc_info )
bmc_info.power_off_action_list.clear();
}
/*************************************************************************
*
* Name : bmcUtil_hwmon_info
*
* Purpose : Creates the hardware monitor info file and content.
*
* Description: The hardware monitor learns the hosts power state and
* current bmc protocol being used.
*
* Future : An extra string is passed in but currently unused.
*
* Returns : nothing
*
*************************************************************************/
void bmcUtil_hwmon_info ( string hostname,
bmc_protocol_enum proto,
bool power_on,
string extra )
{
/* default the bmc info file */
string bmc_info_path_n_filename = BMC_OUTPUT_DIR + hostname ;
/* remove the old BMC info file if present */
daemon_remove_file ( bmc_info_path_n_filename.data() );
/* add the 'protocol' key:val pair */
string info_str = "{\"protocol\":\"" ;
if ( proto == BMC_PROTOCOL__REDFISHTOOL )
info_str.append(BMC_PROTOCOL__REDFISHTOOL_STR);
else
info_str.append(BMC_PROTOCOL__IPMITOOL_STR);
/* add the 'power' state key:val pair */
if ( power_on )
info_str.append("\",\"power_state\":\"on\"");
else
info_str.append("\",\"power_state\":\"off\"");
/* add the extra data if it exists */
if ( ! extra.empty () )
info_str.append(extra);
/* terminate */
info_str.append ("}");
blog ("%s hwmon info: %s", hostname.c_str(), info_str.c_str());
/* write the data to the file */
daemon_log ( bmc_info_path_n_filename.data(), info_str.data() );
}
/*****************************************************************************
*
* Name : bmcUtil_read_bmc_info
* Description : Read power status and protocol from bmc info file
* Parameters : hostname - host name
power_state - read from file
protocol - read from file
* Return : true - file exist
false - file not exist
*
*****************************************************************************/
bool bmcUtil_read_bmc_info( string hostname,
string & power_state,
bmc_protocol_enum & protocol )
{
struct json_object *json_obj = NULL;
string bmc_info_path_n_filename = BMC_OUTPUT_DIR + hostname ;
if ( ! daemon_is_file_present ( bmc_info_path_n_filename.data() ))
return (false);
string filedata = daemon_read_file (bmc_info_path_n_filename.data()) ;
blog ("%s data:%s\n", hostname.c_str(), filedata.data());
json_obj = json_tokener_parse ( (char *)filedata.data() );
if ( json_obj )
{
power_state = jsonUtil_get_key_value_string ( json_obj, "power_state" );
if ( strcmp (power_state.data(), BMC_POWER_ON_STATUS) )
power_state = BMC_POWER_OFF_STATUS ;
string protocol_str = jsonUtil_get_key_value_string ( json_obj, "protocol" );
if ( strcmp (protocol_str.data(), BMC_PROTOCOL__REDFISHTOOL_STR) )
protocol = BMC_PROTOCOL__IPMITOOL ;
else
protocol = BMC_PROTOCOL__REDFISHTOOL ;
json_object_put(json_obj);
ilog ("%s power is %s with bmc communication using %s",
hostname.c_str(),
power_state.c_str(),
bmcUtil_getProtocol_str(protocol).c_str());
return (true);
}
else
{
/* Set to default value for power state and protocol */
power_state = BMC_POWER_OFF_STATUS ;
protocol = BMC_PROTOCOL__IPMITOOL ;
blog ("%s failed to parse bmc info! set to ipmitool by default!\n", hostname.c_str());
return (false);
}
return (true);
}
/*****************************************************************************
*
* Name : bmcUtil_read_hwmond_protocol
* Description : Read hwmon protocol from hwmon_hostname_protocol file
* Parameters : hostname - host name
* Return : bmc protocol
*
*****************************************************************************/
bmc_protocol_enum bmcUtil_read_hwmond_protocol ( string hostname )
{
bmc_protocol_enum protocol = BMC_PROTOCOL__IPMITOOL ;
string hwmond_proto_filename = BMC_HWMON_TMP_DIR + hostname ;
if ( daemon_is_file_present ( hwmond_proto_filename.data() ) == true )
{
string proto_str = daemon_read_file ( hwmond_proto_filename.data() ) ;
if ( !strcmp (proto_str.data(), BMC_PROTOCOL__REDFISHTOOL_STR) )
protocol = BMC_PROTOCOL__REDFISHTOOL ;
}
return protocol;
}
/*****************************************************************************
*
* Name : bmcUtil_write_hwmond_protocol
* Description : Write hwmon protocol to hwmon_hostname_protocol file
* Parameters : hostname - host name
protocol - protocol stored to the file
*
*****************************************************************************/
void bmcUtil_write_hwmond_protocol ( string hostname,
bmc_protocol_enum protocol )
{
string hwmond_proto_filename = BMC_HWMON_TMP_DIR + hostname ;
/* remove old file if present and write current protocol to the file*/
daemon_remove_file ( hwmond_proto_filename.data() );
string proto_str = bmcUtil_getProtocol_str ( protocol ) ;
daemon_log ( hwmond_proto_filename.data(), proto_str.data() );
}
/*************************************************************************
*
* Name : bmcUtil_create_pw_file
@ -544,10 +386,7 @@ void bmcUtil_remove_files ( string hostname, bmc_protocol_enum protocol )
int rc = load_filenames_in_dir ( dir.data(), filelist ) ;
if ( rc )
{
ilog ("%s failed to load files (rc:%d)", hostname.c_str(), rc );
return ;
}
/* files exist as <process>_<hostname>_<suffix> */
if ( !strcmp(MTC_SERVICE_MTCAGENT_NAME, program_invocation_short_name ))
@ -581,11 +420,5 @@ void bmcUtil_remove_files ( string hostname, bmc_protocol_enum protocol )
}
}
}
/* remove the static file that specified the protocol that was used to create this host's sensor model */
string hwmond_proto_filename = BMC_HWMON_TMP_DIR ;
hwmond_proto_filename.append("/") ;
hwmond_proto_filename.append(hostname);
daemon_remove_file ( hwmond_proto_filename.data() );
}
}

20
mtce-common/src/common/bmcUtil.h
View File

@ -21,9 +21,11 @@ using namespace std;
#include "threadUtil.h" /* for ... thread_info_type and utilities */
#define BMC_OUTPUT_DIR ((const char *)("/var/run/bmc/"))
#define BMC_HWMON_TMP_DIR ((const char *)("/etc/mtc/tmp/hwmon/"))
/* supported protocol strings */
#define BMC_PROTOCOL__DYNAMIC_STR ((const char *)("dynamic"))
#define BMC_PROTOCOL__IPMI_STR ((const char *)("ipmi"))
#define BMC_PROTOCOL__REDFISH_STR ((const char *)("redfish"))
#define BMC_PROTOCOL__IPMITOOL_STR ((const char *)("ipmitool"))
#define BMC_PROTOCOL__REDFISHTOOL_STR ((const char *)("redfishtool"))
@ -128,22 +130,6 @@ string bmcUtil_create_data_fn ( string & hostname,
string file_suffix,
bmc_protocol_enum protocol );
/* Read power status and protocol from bmc info file */
bool bmcUtil_read_bmc_info ( string hostname,
string & power_state,
bmc_protocol_enum & protocol);
bmc_protocol_enum bmcUtil_read_hwmond_protocol ( string hostname );
void bmcUtil_write_hwmond_protocol ( string hostname,
bmc_protocol_enum protocol );
/* this utility creates the bmc info file for hardware monitor */
void bmcUtil_hwmon_info ( string hostname,
bmc_protocol_enum proto,
bool power_on,
string extra );
/* Get power state from query response data. */
int bmcUtil_is_power_on ( string hostname,
bmc_protocol_enum protocol,

13
mtce-common/src/common/hostClass.cpp
View File

@ -300,12 +300,7 @@ int hostBaseClass::add_host ( node_inv_type & inv )
if ( host_ptr )
{
host_ptr->ip = inv.ip ;
host_ptr->mac = inv.mac ;
host_ptr->uuid = inv.uuid ;
host_ptr->type = inv.type ;
host_ptr->nodetype = CGTS_NODE_NULL ;
host_ptr->retries = 0 ;
host_ptr->toggle = false ;
@ -427,12 +422,10 @@ void hostBaseClass::memLogDelimit ( void )
void hostBaseClass::mem_log_host ( struct hostBaseClass::host * host_ptr )
{
char str[MAX_MEM_LOG_DATA] ;
snprintf (&str[0], MAX_MEM_LOG_DATA, "%s\t%s - %s - %s - %s\n",
host_ptr->hostname.c_str(),
snprintf (&str[0], MAX_MEM_LOG_DATA, "%s\t%s - %s\n",
host_ptr->hostname.c_str(),
host_ptr->ip.c_str(),
host_ptr->mac.c_str(),
host_ptr->uuid.c_str(),
host_ptr->type.c_str());
host_ptr->uuid.c_str());
mem_log (str);
}

63
mtce-common/src/common/hostClass.h
View File

@ -1,7 +1,7 @@
#ifndef __INCLUDE_HOSTCLASS_H__
#define __INCLUDE_HOSTCLASS_H__
/*
* Copyright (c) 2015 Wind River Systems, Inc.
* Copyright (c) 2015, 2019 Wind River Systems, Inc.
*
* SPDX-License-Identifier: Apache-2.0
*
@ -11,7 +11,7 @@
* @file
* Wind River CGTS Platform Host Maintenance "Host Manager"
* class, support structs and enums.
*/
*/
#include <sys/types.h>
#include <iostream>
@ -34,7 +34,7 @@ class hostBaseClass
{
private:
/**
/**
* A single host entity within the hostBaseClass.
* Used to build a linked list of added/provisioned hosts.
*/
@ -42,22 +42,13 @@ class hostBaseClass
/** The name of the host */
std::string hostname ;
/** The name of the host */
std::string uuid ;
std::string uuid ;
/** The IP address of the host */
/** The IP address of the host */
std::string ip ;
/** The Mac address of the host node */
std::string mac ;
/** A string indicating the host type as 'compute' , 'storage' or 'controller' */
std::string type ;
/** The Type ; host specific service refinement */
int nodetype ;
/** general retry counter */
int retries ;
@ -66,7 +57,7 @@ class hostBaseClass
/** Pointer to the previous host in the list */
struct host * prev;
/** Pointer to the next host in the list */
struct host * next;
} ;
@ -79,35 +70,35 @@ class hostBaseClass
* Preserves the host address in the host_ptr list and increments
* the memory_allocs counter used by the inservice test audit.
*
* @return
* a pointer to the memory of the newly allocated host */
* @return
* a pointer to the memory of the newly allocated host */
struct hostBaseClass::host * newHost ( void );
/** Start heartbeating a new host.
*
*
* host is added to the end of the host linked list.
*
* @param host_info_ptr
* is a pointer containing pertinent info about the physical host
* @return
* @return
* a pointer to the newly added host
*/
struct hostBaseClass::host* addHost ( string hostname );
/** Get pointer to "hostname" host.
*
*
* Host list lookup by pointer from hostname.
*
* @param host_info_ptr
* @param host_info_ptr
* is a pointer containing info required to find the host in the host list
* @return
* @return
* a pointer to the hostname's host
*/
struct hostBaseClass::host* getHost ( string hostname );
/** Free the memory of a previously allocated host.
*
* The memory to be removed is found in the host_ptr list, cleared and
* The memory to be removed is found in the host_ptr list, cleared and
* the memory_allocs counter is decremented.
* If the memory cannot be found then an error is returned.
*
@ -117,24 +108,24 @@ class hostBaseClass
* a signed integer of PASS or -EINVAL
*/
int delHost ( struct hostBaseClass::host * host_ptr );
/** Remove a host from the linked list.
*
* Node is spliced out of the host linked list.
*
* @param node_info_ptr
* @param node_info_ptr
* is a pointer containing info required to find the host in the host list
* @return
* @return
* an integer of PASS or -EINVAL */
int remHost ( string hostname );
/** List of allocated host memory.
*
* An array of host pointers.
*/
*/
hostBaseClass::host * host_ptrs[MAX_HOSTS] ;
/** A memory allocation counter.
*
* Should represent the number of hosts in the linked list.
@ -142,19 +133,19 @@ class hostBaseClass
int memory_allocs ;
/** A memory used counter
*
*
* A variable storing the accumulated host memory
*/
*/
int memory_used ;
void mem_log_host ( struct hostBaseClass::host * host_ptr );
/** Public Interfaces that allow hosts to be
/** Public Interfaces that allow hosts to be
* added or removed from maintenance.
*/
public:
hostBaseClass(); /**< constructor */
hostBaseClass(); /**< constructor */
~hostBaseClass(); /**< destructor */
/**< The service this list is associated with */
@ -171,7 +162,7 @@ public:
/** Add a host to the linked list using public API */
int add_host ( node_inv_type & inv );
/** Mod a host to the linked list using public API */
int mod_host ( node_inv_type & inv );
@ -193,7 +184,7 @@ public:
void memDumpAllState ( void );
void print_node_info ( void ); /**< Print node info banner */
/** This is a list of host names. */
/** This is a list of host names. */
std::list<string> hostlist ;
std::list<string>::iterator hostlist_iter_ptr ;

15
mtce-common/src/common/hostUtil.cpp
View File

@ -122,6 +122,14 @@ bool hostUtil_is_valid_ip_addr ( string ip )
return (false);
}
bool hostUtil_is_valid_username ( string un )
{
if ( !un.empty() )
if ( un.compare(NONE) )
return (true);
return (false);
}
bool hostUtil_is_valid_mac_addr ( string mac )
{
if ( !mac.empty() )
@ -136,10 +144,9 @@ bool hostUtil_is_valid_bm_type ( string bm_type )
if ( !bm_type.empty() )
{
if (( bm_type == "bmc" ) ||
( bm_type == "ilo" ) ||
( bm_type == "ilo3" ) ||
( bm_type == "ilo4" ) ||
( bm_type == "quanta" ))
( bm_type == "dynamic" ) || /* auto-learn */
( bm_type == "redfish" ) ||
( bm_type == "ipmi" ))
{
return (true);
}

1
mtce-common/src/common/hostUtil.h
View File

@ -45,6 +45,7 @@ string hostUtil_getPrefixPath ( void );
bool hostUtil_is_valid_uuid ( string uuid );
bool hostUtil_is_valid_ip_addr ( string ip );
bool hostUtil_is_valid_username ( string un );
bool hostUtil_is_valid_bm_type ( string bm_type );
int hostUtil_mktmpfile ( string hostname, string basename, string & filename, string data );

4
mtce-common/src/common/ipmiUtil.cpp
View File

@ -83,14 +83,14 @@ void ipmiUtil_bmc_info_log ( string hostname, bmc_info_type & bmc_info, int rc )
}
else
{
ilog ("%s Manufacturer: %s [id:%s] [ Device: %s ver %s ]\n",
ilog ("%s manufacturer: %s [id:%s] [ Device: %s ver %s ]\n",
hostname.c_str(),
bmc_info.manufacturer.c_str(),
bmc_info.manufacturer_id.c_str(),
bmc_info.device_id.c_str(),
bmc_info.hw_version.c_str());
ilog ("%s Product Name: %s [id:%s] [ BMC FW: ver %s ]\n",
ilog ("%s product name: %s [id:%s] [ BMC FW: ver %s ]\n",
hostname.c_str(),
bmc_info.product_name.c_str(),
bmc_info.product_id.c_str(),

19
mtce-common/src/common/jsonUtil.cpp
View File

@ -69,11 +69,11 @@ static struct json_object * _json_verify_object ( struct json_object * obj,
status = json_object_object_get_ex (obj, "error", &req_obj );
if (( status == TRUE ) && ( req_obj ))
{
elog ("Found 'error' label instead\n");
jlog ("Found 'error' label instead\n");
}
else
{
elog ("Neither specified nor error label found in object\n");
jlog ("Neither specified nor error label found in object\n");
}
return ((struct json_object *)(NULL)) ;
}
@ -211,16 +211,16 @@ int jsonUtil_get_key_val ( char * json_str_ptr,
/* init to null to avoid trap on early cleanup call with
* bad non-null default pointer value */
struct json_object *raw_obj = (struct json_object *)(NULL);
if ((json_str_ptr == NULL) || ( *json_str_ptr == '\0' ) || ( ! strncmp ( json_str_ptr, "(null)" , 6 )))
{
elog ("Cannot tokenize a null json string\n");
elog ("... json string: %s\n", json_str_ptr );
return (FAIL);
}
size_t len_before = strlen (json_str_ptr);
jlog2 ("String: %s\n", json_str_ptr );
raw_obj = json_tokener_parse( json_str_ptr );
@ -315,7 +315,6 @@ int jsonUtil_inv_load ( char * json_str_ptr,
struct json_object *node_obj = (struct json_object *)(NULL);
struct json_object *next_obj = (struct json_object *)(NULL);
// printf ("String: <%s>\n", json_str_ptr );
if (( json_str_ptr == NULL ) || ( *json_str_ptr == '\0' ) ||
( ! strncmp ( json_str_ptr, "(null)" , 6 )))
{
@ -389,6 +388,7 @@ int jsonUtil_inv_load ( char * json_str_ptr,
info.host[i].oper_subf = _json_get_key_value_string ( node_obj, MTC_JSON_INV_OPER_SUBF );
info.host[i].avail_subf = _json_get_key_value_string ( node_obj, MTC_JSON_INV_AVAIL_SUBF);
info.host[i].clstr_ip = _json_get_key_value_string ( node_obj, MTC_JSON_INV_CLSTRIP );
info.host[i].mtce_info = _json_get_key_value_string ( node_obj, MTC_JSON_INV_MTCE_INFO );
if ( info.host[i].uuid.length() != UUID_LEN )
{
@ -446,7 +446,7 @@ int jsonUtil_patch_load ( char * json_str_ptr,
info.oper_subf = _json_get_key_value_string ( node_obj, MTC_JSON_INV_OPER_SUBF );
info.avail_subf = _json_get_key_value_string ( node_obj, MTC_JSON_INV_AVAIL_SUBF);
info.clstr_ip = _json_get_key_value_string ( node_obj, MTC_JSON_INV_CLSTRIP );
info.mtce_info = _json_get_key_value_string ( node_obj, MTC_JSON_INV_MTCE_INFO );
if (node_obj) json_object_put(node_obj);
return (PASS);
@ -504,7 +504,8 @@ int jsonUtil_load_host ( char * json_str_ptr, node_inv_type & info )
info.id = _json_get_key_value_string ( node_obj, "id" );
info.oper_subf = _json_get_key_value_string ( node_obj, MTC_JSON_INV_OPER_SUBF );
info.avail_subf = _json_get_key_value_string ( node_obj, MTC_JSON_INV_AVAIL_SUBF);
info.clstr_ip = _json_get_key_value_string ( node_obj, MTC_JSON_INV_CLSTRIP );
info.clstr_ip = _json_get_key_value_string ( node_obj, MTC_JSON_INV_CLSTRIP );
info.mtce_info = _json_get_key_value_string ( node_obj, MTC_JSON_INV_MTCE_INFO );
if ( info.uuid.length() != UUID_LEN )
{
@ -1017,7 +1018,7 @@ int jsonUtil_get_list ( char * json_str_ptr, string label, list<string> & key_li
label_obj = _json_verify_object (raw_obj, label.data());
if ( !label_obj )
{
elog ("unable to find label '%s'\n", label.c_str());
jlog ("unable to find label '%s'\n", label.c_str());
rc = FAIL_JSON_OBJECT ;
goto get_list_cleanup ;
}

10
mtce-common/src/common/nodeBase.h
View File

@ -154,6 +154,7 @@ void daemon_exit ( void );
/* supported BMC communication protocols ; access method */
typedef enum
{
BMC_PROTOCOL__DYNAMIC,
BMC_PROTOCOL__IPMITOOL,
BMC_PROTOCOL__REDFISHTOOL,
} bmc_protocol_enum ;
@ -202,16 +203,22 @@ typedef enum
#define MTC_JSON_INV_TYPE "personality"
#define MTC_JSON_INV_FUNC "subfunctions" // personality"
#define MTC_JSON_INV_TASK "task"
#define MTC_JSON_INV_MTCE_INFO "mtce_info"
#define MTC_JSON_INV_ACTION "action"
#define MTC_JSON_INV_UPTIME "uptime"
#define MTC_JSON_INV_BMIP "bm_ip"
#define MTC_JSON_INV_BMTYPE "bm_type"
#define MTC_JSON_INV_BMUN "bm_username"
#define MTC_JSON_INV_BMHTTP "bm_http" // http method 'http'/'https'
#define MTC_JSON_SERVICE "service"
#define MTC_JSON_SEVERITY "severity"
#define MTC_JSON_SENSOR "sensor"
#define MTC_JSON_PROCESS "process"
/* Mtce Info Keys */
#define MTCE_INFO_KEY__BMC_PROTOCOL "bmc_protocol"
/* These Task strings should not be changed without
* the corresponding change in Horizon.
*
@ -462,12 +469,15 @@ typedef struct
std::string bm_ip ;
std::string bm_un ;
std::string bm_type ;
std::string bm_proto ; // access protocol 'ipmi','redfish' or 'dynamic'
std::string bm_http ; // Security Mode 'http' or 'https'
std::string id ;
/* Added to support sub-function state and status */
std::string func ;
std::string oper_subf ;
std::string avail_subf ;
std::string mtce_info ;
} node_inv_type ;
void node_inv_init (node_inv_type & inv);

6
mtce-common/src/common/nodeTimers.cpp
View File

@ -317,7 +317,7 @@ void mtcTimer_reset ( struct mtc_timer & mtcTimer )
if ( mtcTimer.active )
mtcTimer.active = false ;
mtcTimer.ring = false ;
mtcTimer.ring = true ;
}
void mtcTimer_reset ( struct mtc_timer * mtcTimer_ptr )
@ -330,7 +330,7 @@ void mtcTimer_reset ( struct mtc_timer * mtcTimer_ptr )
if ( mtcTimer_ptr->active )
mtcTimer_ptr->active = false ;
mtcTimer_ptr->ring = false ;
mtcTimer_ptr->ring = true ;
}
}
@ -479,7 +479,7 @@ void _timer_init ( struct mtc_timer * mtcTimer_ptr , string hostname, string ser
mtcTimer_ptr->tid = NULL ;
mtcTimer_ptr->secs = 0 ;
mtcTimer_ptr->msec = 0 ;
mtcTimer_ptr->ring = false ;
mtcTimer_ptr->ring = true ;
mtcTimer_ptr->active= false ;
mtcTimer_ptr->error = false ;
mtcTimer_ptr->mutex = false ;

10
mtce-common/src/common/nodeUtil.cpp
View File

@ -120,6 +120,8 @@ void node_inv_init (node_inv_type & inv)
inv.bm_ip.clear();
inv.bm_un.clear();
inv.bm_type.clear();
inv.bm_proto.clear();
inv.bm_http.clear();
inv.action.clear();
inv.uptime.clear();
inv.oper_subf.clear();
@ -138,10 +140,10 @@ void print_inv ( node_inv_type & info )
syslog ( LOG_INFO, "| personality: %s\n", info.type.c_str());
syslog ( LOG_INFO, "| hostname : %s\n", info.name.c_str());
syslog ( LOG_INFO, "| task : %s\n", info.task.c_str());
syslog ( LOG_INFO, "| info : %s\n", info.mtce_info.c_str());
syslog ( LOG_INFO, "| ip : %s\n", info.ip.c_str());
syslog ( LOG_INFO, "| mac : %s\n", info.mac.c_str());
syslog ( LOG_INFO, "| uuid : %s\n", info.uuid.c_str());
syslog ( LOG_INFO, "| operState: %s\n", info.oper_subf.c_str());
syslog ( LOG_INFO, "| adminState: %s\n", info.admin.c_str());
syslog ( LOG_INFO, "| operState: %s\n", info.oper.c_str());
syslog ( LOG_INFO, "| availStatus: %s\n", info.avail.c_str());
@ -184,9 +186,9 @@ bool is_combo_system (unsigned int nodetype_mask )
}
int set_host_functions ( string nodetype_str,
unsigned int * nodetype_bits_ptr,
unsigned int * nodetype_function_ptr,
int set_host_functions ( string nodetype_str,
unsigned int * nodetype_bits_ptr,
unsigned int * nodetype_function_ptr,
unsigned int * nodetype_subfunction_ptr )
{
int rc = PASS ;

14
mtce-common/src/common/pingUtil.cpp
View File

@ -583,6 +583,20 @@ void pingUtil_fini ( ping_info_type & ping_info )
ping_info.stage = PINGUTIL_MONITOR_STAGE__IDLE ;
}
void pingUtil_restart ( ping_info_type & ping_info )
{
ilog ("%s ping monitor restart", ping_info.hostname.c_str());
ping_info.ok = false ;
ping_info.send_retries = 0 ;
ping_info.monitoring = false ;
pingUtil_fini (ping_info);
pingUtil_init (ping_info.hostname, ping_info, ping_info.ip.data());
mtcTimer_reset ( ping_info.timer );
mtcTimer_start ( ping_info.timer, ping_info.timer_handler, 1 );
ping_info.stage = PINGUTIL_MONITOR_STAGE__WAIT;
}
/********************************************************************************
*
* Name : pingUtil_acc_monitor

10
mtce-common/src/common/pingUtil.h
View File

@ -146,4 +146,14 @@ void pingUtil_fini ( ping_info_type & ping_info ); /* the preopened ping socket
int pingUtil_acc_monitor ( ping_info_type & ping_info );
/********************************************************************************
*
* Name : pingUtil_restart
*
* Purpose : Restart the ping monitor
*
*******************************************************************************/
void pingUtil_restart ( ping_info_type & ping_info );
#endif

6
mtce-common/src/common/redfishUtil.cpp
View File

@ -519,7 +519,7 @@ int redfishUtil_get_bmc_info ( string & hostname,
struct json_object *json_obj = json_tokener_parse((char*)json_bmc_info.data());
if ( !json_obj )
{
wlog ("%s bmc info file empty", hostname.c_str());
wlog ("%s bmc info data parse error", hostname.c_str());
return (FAIL_JSON_PARSE) ;
}
@ -594,7 +594,7 @@ int redfishUtil_get_bmc_info ( string & hostname,
bmc_info.processors = jsonUtil_get_key_value_int ( proc_obj, REDFISH_LABEL__COUNT );
redfishUtil_health_info ( hostname, REDFISH_LABEL__PROCESSOR,
proc_obj, status) ;
ilog ("%s has %2d Processors ; %s and %s:%s",
ilog ("%s has %2u Processors ; %s and %s:%s",
hostname.c_str(),
bmc_info.processors,
status.state.c_str(),
@ -617,7 +617,7 @@ int redfishUtil_get_bmc_info ( string & hostname,
bmc_info.memory_in_gigs = jsonUtil_get_key_value_int ( mem_obj, REDFISH_LABEL__MEMORY_TOTAL );
redfishUtil_health_info ( hostname, REDFISH_LABEL__MEMORY,
mem_obj, status) ;
ilog ("%s has %d GiB Memory ; %s and %s:%s",
ilog ("%s has %u GiB Memory ; %s and %s:%s",
hostname.c_str(),
bmc_info.memory_in_gigs,
status.state.c_str(),

10
mtce-common/src/common/secretUtil.cpp
View File

@ -76,7 +76,7 @@ barbicanSecret_type * secretUtil_manage_secret ( libEvent & event,
if ( it->second.stage == MTC_SECRET__START ||
it->second.stage == MTC_SECRET__GET_REF_FAIL )
{
if ( secret_timer.ring == true )
if ( mtcTimer_expired ( secret_timer ) )
{
rc = secretUtil_get_secret ( event, hostname, host_uuid );
if (rc)
@ -99,7 +99,7 @@ barbicanSecret_type * secretUtil_manage_secret ( libEvent & event,
else if ( it->second.stage == MTC_SECRET__GET_REF_RECV ||
it->second.stage == MTC_SECRET__GET_PWD_FAIL )
{
if ( secret_timer.ring == true )
if ( mtcTimer_expired ( secret_timer ) )
{
rc = secretUtil_read_secret ( event, hostname, host_uuid );
if (rc)
@ -211,7 +211,8 @@ int secretUtil_get_secret ( libEvent & event,
event.timeout = HTTP_SECRET_TIMEOUT ;
event.handler = &secretUtil_handler ;
dlog ("Path:%s\n", event.token.url.c_str() );
hlog ("%s secretUtil_get_secret %s\n",
hostname.c_str(), event.token.url.c_str() );
return ( httpUtil_api_request ( event ) ) ;
}
@ -266,7 +267,8 @@ int secretUtil_read_secret ( libEvent & event,
event.timeout = HTTP_SECRET_TIMEOUT ;
event.handler = &secretUtil_handler ;
dlog ("Path:%s\n", event.token.url.c_str() );
hlog ("%s secretUtil_read_secret %s",
hostname.c_str(), event.token.url.c_str() );
return ( httpUtil_api_request ( event ) ) ;
}

6
mtce-common/src/common/threadUtil.cpp
View File

@ -729,7 +729,7 @@ int thread_handler ( thread_ctrl_type & ctrl, thread_info_type & info )
if ( info.runcount != (ctrl.runcount+1))
{
wlog ("%s %s thread runcount jumped from %d to %d (rc:%d)\n",
wlog ("%s %s thread runcount jumped from %d to %d (rc:%u)\n",
ctrl.hostname.c_str(),
ctrl.name.c_str(),
ctrl.runcount,
@ -740,7 +740,7 @@ int thread_handler ( thread_ctrl_type & ctrl, thread_info_type & info )
{
if ( info.status )
{
blog ("%s %s thread completed (rc:%d)\n",
blog ("%s %s thread completed (rc:%u)\n",
ctrl.hostname.c_str(),
ctrl.name.c_str(),
info.status);
@ -757,7 +757,7 @@ int thread_handler ( thread_ctrl_type & ctrl, thread_info_type & info )
info.signal = SIGKILL ;
if ( info.id != 0 )
{
wlog ("%s %s thread kill req (rc:%d)\n",
wlog ("%s %s thread kill req (rc:%u)\n",
ctrl.hostname.c_str(),
ctrl.name.c_str(),
info.status);

616
mtce/src/common/nodeClass.cpp
View File

@ -16,6 +16,7 @@
#include <string>
#include <errno.h> /* for ENODEV, EFAULT and ENXIO */
#include <unistd.h> /* for close and usleep */
#include <algorithm>
using namespace std;
@ -494,6 +495,9 @@ nodeLinkClass::node* nodeLinkClass::addNode( string hostname )
ptr->clstr_ip = "" ;
ptr->clstr_mac = "" ;
/* key value dictionary */
ptr->mtce_info = "" ;
ptr->patching = false ;
ptr->patched = false ;
@ -625,12 +629,13 @@ nodeLinkClass::node* nodeLinkClass::addNode( string hostname )
ptr->mnfa_graceful_recovery = false ;
/* initialize all board management variables for this host */
ptr->bmc_protocol = BMC_PROTOCOL__IPMITOOL ;
ptr->bm_ip = NONE ;
ptr->bm_un = NONE ;
ptr->bm_pw = NONE ;
ptr->bm_cmd= NONE ;
ptr->bm_type = NONE ; /* TODO: OBS */
ptr->bm_http_mode = "https" ;
ptr->bmc_protocol = BMC_PROTOCOL__DYNAMIC ;
ptr->bm_ip = NONE ;
ptr->bm_un = NONE ;
ptr->bm_pw = NONE ;
ptr->bm_cmd = NONE ;
ptr->bm_type = NONE ;
/* restart command tht need to learned for Redfish.
* ipmi commands are hard coded fro legacy support.
@ -642,15 +647,13 @@ nodeLinkClass::node* nodeLinkClass::addNode( string hostname )
ptr->bmc_provisioned = false ; /* assume not provisioned until learned */
ptr->bmc_accessible = false ; /* assume not accessible until proven */
ptr->bmc_access_method_changed = false ;
ptr->bm_ping_info.ok = false ;
if ( hostname == my_hostname )
ptr->power_on = true ;
else
ptr->power_on = false ; /* learned on first BMC connection */
bmc_access_data_init ( ptr ); /* init all the BMC access vars all modes */
/* init the alarm array only to have it updated later
* with current alarm severities */
for ( int id = 0 ; id < MAX_ALARMS ; id++ )
@ -1865,24 +1868,26 @@ int nodeLinkClass::update_key_value ( string hostname, string key , string value
{
int rc = PASS ;
struct nodeLinkClass::node * node_ptr = nodeLinkClass::getNode( hostname );
if ( node_ptr )
if ( node_ptr )
{
/* TODO: Add all database members to this utility */
if ( !key.compare(MTC_JSON_INV_BMIP) )
node_ptr->bm_ip = value ;
else if ( !key.compare(MTC_JSON_INV_TASK) )
node_ptr->task = value ;
else if ( !key.compare(MTC_JSON_INV_MTCE_INFO) )
node_ptr->mtce_info = value ;
else
{
wlog ("%s Unsupported key '%s' update with value '%s'\n",
wlog ("%s Unsupported key '%s' update with value '%s'\n",
hostname.c_str(), key.c_str(), value.c_str());
rc = FAIL_BAD_PARM ;
rc = FAIL_BAD_PARM ;
}
}
else
{
wlog ("Cannot change 'admin' state for unknown hostname (%s)\n",
hostname.c_str());
wlog ("Cannot change 'admin' state for unknown hostname (%s)\n",
hostname.c_str());
}
return (rc);
}
@ -2343,6 +2348,8 @@ int nodeLinkClass::mod_host ( node_inv_type & inv )
node_ptr->hostname.c_str(),
node_ptr->ip.c_str(), inv.ip.c_str());
node_ptr->ip = inv.ip ;
mtcInfo_clr ( node_ptr, MTCE_INFO_KEY__BMC_PROTOCOL );
mtcInvApi_update_mtcInfo ( node_ptr );
/* Tell the guestAgent the new IP */
rc = send_guest_command(node_ptr->hostname,MTC_CMD_MOD_HOST);
@ -2402,13 +2409,6 @@ int nodeLinkClass::mod_host ( node_inv_type & inv )
modify = true ; /* we have some delta */
}
/* PATCHBACK - issue found during BMC refactoring user story
* where there was a race condition found where the bmc dnsmasq file
* was updated with a new bm_ip close to when there was an
* administrative operation (unlock in this case). The newly learned
* bm_ip was overwritten by the now stale bm_ip that came in from
* inventory. The bm_ip should never come from sysinv while in
* internal mode. */
if (( node_ptr->bm_ip.compare ( inv.bm_ip )))
{
if ( inv.bm_ip.empty () )
@ -2428,9 +2428,13 @@ int nodeLinkClass::mod_host ( node_inv_type & inv )
node_ptr->bm_ip = inv.bm_ip ;
mtcInfo_clr ( node_ptr, MTCE_INFO_KEY__BMC_PROTOCOL );
mtcInvApi_update_mtcInfo ( node_ptr );
modify_bm = true ; /* board mgmnt change */
modify = true ; /* we have some delta */
}
if ( node_ptr->bm_type.compare ( inv.bm_type ) )
{
if ( inv.bm_type.empty() )
@ -2442,6 +2446,10 @@ int nodeLinkClass::mod_host ( node_inv_type & inv )
node_ptr->hostname.c_str(),
node_ptr->bm_type.c_str(), inv.bm_type.c_str());
mtcInfo_clr ( node_ptr, MTCE_INFO_KEY__BMC_PROTOCOL );
mtcInvApi_update_mtcInfo ( node_ptr );
node_ptr->bm_type = inv.bm_type ;
modify_bm = true ; /* board mgmnt change */
modify = true ; /* we have some delta */
}
@ -2453,36 +2461,54 @@ int nodeLinkClass::mod_host ( node_inv_type & inv )
}
if ( modify_bm == true )
{
wlog ("%s Board Management provisioning has changed\n", node_ptr->hostname.c_str());
bool bm_type_was_valid = hostUtil_is_valid_bm_type (node_ptr->bm_type) ;
bool bm_type_now_valid = hostUtil_is_valid_bm_type (inv.bm_type) ;
blog ("%s Board Management provisioning has changed\n", node_ptr->hostname.c_str());
/* update bm_type now */
node_ptr->bm_type = inv.bm_type ;
/* BM is provisioned */
if ( bm_type_now_valid == true )
/* Assume this modify has invalid provisioning until ... */
bool valid_provisioning = false ;
if (( hostUtil_is_valid_ip_addr ( node_ptr->bm_ip )) &&
( hostUtil_is_valid_username( node_ptr->bm_un )) &&
( hostUtil_is_valid_bm_type ( node_ptr->bm_type )))
{
/* force (re)provision */
manage_bmc_provisioning ( node_ptr );
/* ... proven otherwise */
valid_provisioning = true ;
}
/* BM is already provisioned but is now deprovisioned */
else if (( bm_type_was_valid == true ) && ( bm_type_now_valid == false ))
/* ------------------------------------ */
/* Start From Already Provisioned Cases */
/* ------------------------------------ */
if ( node_ptr->bmc_provisioned == true )
{
node_ptr->bm_type = NONE ;
node_ptr->bm_ip = NONE ;
node_ptr->bm_un = NONE ;
node_ptr->bm_pw = NONE ;
mtcAlarm_log ( node_ptr->hostname, MTC_LOG_ID__COMMAND_BM_DEPROVISIONED );
set_bm_prov ( node_ptr, false );
/* Deprovisioning Case */
if ( valid_provisioning == false )
{
node_ptr->bm_type = NONE ;
node_ptr->bm_ip = NONE ;
node_ptr->bm_un = NONE ;
node_ptr->bm_pw = NONE ;
set_bm_prov ( node_ptr, false );
mtcAlarm_log ( node_ptr->hostname, MTC_LOG_ID__COMMAND_BM_DEPROVISIONED );
}
/* Reprovisioning Case */
else
{
set_bm_prov ( node_ptr, true );
mtcAlarm_log ( node_ptr->hostname, MTC_LOG_ID__COMMAND_BM_REPROVISIONED );
}
}
/* BM was not provisioned and is still not provisioned */
/* -------------------------------------- */
/* Start From Already Deprovisioned Cases */
/* -------------------------------------- */
else
{
/* Handle all other provisioning changes ; username, ip address */
manage_bmc_provisioning ( node_ptr );
if ( valid_provisioning == true )
{
set_bm_prov ( node_ptr, true );
mtcAlarm_log ( node_ptr->hostname, MTC_LOG_ID__COMMAND_BM_PROVISIONED );
}
else
{
; // do nothing ; deprovisioned with invalid provisioning
}
}
}
}
@ -2628,13 +2654,13 @@ int nodeLinkClass::add_host ( node_inv_type & inv )
int rc = FAIL ;
struct nodeLinkClass::node * node_ptr = static_cast<struct node *>(NULL);
if ((!inv.name.compare("controller-0")) ||
if ((!inv.name.compare("controller-0")) ||
(!inv.name.compare("controller-1")))
{
dlog ("Adding %s\n", inv.name.c_str());
node_ptr = nodeLinkClass::getNode(inv.name);
node_ptr = nodeLinkClass::getNode(inv.name);
}
else if (( inv.name.empty()) ||
else if (( inv.name.empty()) ||
( !inv.name.compare ("none") ) ||
( !inv.name.compare ("None") ))
{
@ -2642,17 +2668,17 @@ int nodeLinkClass::add_host ( node_inv_type & inv )
inv.uuid.c_str());
return (FAIL_INVALID_HOSTNAME) ;
}
else if (( inv.uuid.empty()) ||
else if (( inv.uuid.empty()) ||
( !inv.uuid.compare ("none") ) ||
( !inv.uuid.compare ("None") ))
{
wlog ("Refusing to add host with 'null' or 'invalid' uuid (%s)\n",
wlog ("Refusing to add host with 'null' or 'invalid' uuid (%s)\n",
inv.uuid.c_str());
return (FAIL_INVALID_UUID) ;
}
/* Ensure we don't add a host with critical info that is
* already used by other members of inventory like ;
* already used by other members of inventory like ;
* hostname, uuid, ip, mac, bm_ip */
else if ( ( rc = add_host_precheck ( inv )) > RETRY )
{
@ -2738,6 +2764,7 @@ int nodeLinkClass::add_host ( node_inv_type & inv )
node_ptr->mac = inv.mac ;
node_ptr->uuid = inv.uuid ;
node_ptr->clstr_ip = inv.clstr_ip ;
node_ptr->mtce_info = inv.mtce_info ;
if ( inv.uptime.length() )
{
@ -2752,7 +2779,7 @@ int nodeLinkClass::add_host ( node_inv_type & inv )
node_ptr->thread_extra_info.bm_ip = node_ptr->bm_ip = inv.bm_ip ;
node_ptr->thread_extra_info.bm_un = node_ptr->bm_un = inv.bm_un ;
node_ptr->bm_type = inv.bm_type ;
node_ptr->bm_pw_wait_log_throttle = 0 ;
node_ptr->bm_ping_info.sock = 0 ;
/* initialize the host power and reset control thread */
@ -3048,6 +3075,226 @@ void nodeLinkClass::set_task ( string hostname, string task )
}
}
/******************************************************************************
*
* Name : set_mtcInfo
*
* Purpose : Set the 'mtce info' for the specified host.
* Really only used for the first controller on process startup.
*
*****************************************************************************/
int nodeLinkClass::set_mtcInfo ( string hostname, string & mtce_info )
{
return (set_mtcInfo ( nodeLinkClass::getNode(hostname), mtce_info ));
}
int nodeLinkClass::set_mtcInfo ( struct nodeLinkClass::node * node_ptr,
string & mtce_info)
{
if ( node_ptr == NULL )
return FAIL_NULL_POINTER ;
node_ptr->mtce_info = mtce_info ;
mtcInfo_log(node_ptr);
return (PASS);
}
/******************************************************************************
*
* Name : mtcInfo_set, mtcInfo_get, mtcInfo_clr, mtcInfo_log
*
* Purpose : Manage the node's 'mtce info' database string used as a
* key / value pair dictionary.
*
* Description: Set updates mtce info dictionary with specified key's value
* Get returns the specified key's value from mtce info dictionary
* Clr removes a specified key and its value pair from dictionary
* Log the data in mtce_info ; just loaded or ongoing changes
*
* Assumptions: The database element mtce_info can't hold quoted key
* value pairs. Therefore the support utilities manage
* (add, modify, delete) the key value content without quotes.
*
* These utilities do not push mtce_info changes to the database.
*
* Testing : All these utilities have been tested setting, getting and
* clearing a key value pair when the target key/value pair ...
*
* 1. is the only key/value pair
* 2. is the last key/value pair in a list of 3.
* 3. is the first key/value pair in a list of 3
* 4. is the middle key/value pair list of 3.
*
*****************************************************************************/
int nodeLinkClass::mtcInfo_set ( string hostname, string key, string value )
{
return (mtcInfo_set ( nodeLinkClass::getNode(hostname), key, value ));
}
int nodeLinkClass::mtcInfo_set ( struct nodeLinkClass::node * node_ptr,
string key,
string value )
{
if ( node_ptr == NULL )
return FAIL_NULL_POINTER ;
else if (( node_ptr->mtce_info.empty()) ||
( node_ptr->mtce_info.at(0) != '{' ))
{
/* mtce info empty ; just create new content
*
* {key:value}
*/
node_ptr->mtce_info = '{' + key + ':' + value + '}' ;
}
else
{
/* Remove all whitespace.
*
* Should not be any but would mess up parsing if there
* was so always do it.
*/
node_ptr->mtce_info.erase(remove(node_ptr->mtce_info.begin(),
node_ptr->mtce_info.end(), ' '),
node_ptr->mtce_info.end());
/* find this key */
size_t pos = node_ptr->mtce_info.find(key + ':');
if ( pos != std::string::npos )
{
pos = pos+key.length() +1 ;
string mtce_info_tmp = node_ptr->mtce_info.substr(0, pos);
if (node_ptr->mtce_info.substr(pos, value.length()) != value )
{
/* add the new value following the *key: content */
mtce_info_tmp.append(value);
/* now look for other the key/value data that needs to be
* appended after the revised key/value.
* This is indicated by a following ',' */
size_t pos1 = node_ptr->mtce_info.find(',' , pos-1);
if ( pos1 != std::string::npos )
{
mtce_info_tmp.append(node_ptr->mtce_info.substr(pos1));
}
else
{
/* Otherwise just terminate the dictionary */
mtce_info_tmp += '}';
}
/* save the revised mtce_info string */
node_ptr->mtce_info = mtce_info_tmp ;
}
/* else the value is the same and does not need to be set */
}
else
{
/* add the new key and value */
node_ptr->mtce_info.pop_back();
node_ptr->mtce_info.append(',' + key + ':' + value + '}');
}
}
dlog ("%s %s", node_ptr->hostname.c_str(), node_ptr->mtce_info.c_str());
return(PASS);
}
string nodeLinkClass::mtcInfo_get ( string hostname, string key )
{
struct nodeLinkClass::node * node_ptr = nodeLinkClass::getNode(hostname);
return (mtcInfo_get ( node_ptr, key ));
}
string nodeLinkClass::mtcInfo_get ( struct nodeLinkClass::node * node_ptr,
string key )
{
if ( node_ptr )
{
/* Remove all whitespace. */
node_ptr->mtce_info.erase(remove(node_ptr->mtce_info.begin(),
node_ptr->mtce_info.end(), ' '),
node_ptr->mtce_info.end());
/* find this key */
size_t pos = node_ptr->mtce_info.find(key + ':');
if ( pos != std::string::npos )
{
size_t value_start_pos = pos + key.length() + 1 ;
size_t value_end_pos = node_ptr->mtce_info.find(',', value_start_pos) ;
if ( value_end_pos == std::string::npos )
{
value_end_pos = node_ptr->mtce_info.find('}') ;
}
return (node_ptr->mtce_info.substr(value_start_pos,
value_end_pos-value_start_pos ));
}
}
return "" ;
}
void nodeLinkClass::mtcInfo_clr ( string hostname, string key )
{
struct nodeLinkClass::node * node_ptr = nodeLinkClass::getNode(hostname);
mtcInfo_clr ( node_ptr, key );
}
void nodeLinkClass::mtcInfo_clr ( struct nodeLinkClass::node * node_ptr,
string key )
{
if ( node_ptr && ( ! node_ptr->mtce_info.empty()) )
{
/* Remove all whitespace. */
node_ptr->mtce_info.erase(remove(node_ptr->mtce_info.begin(),
node_ptr->mtce_info.end(), ' '),
node_ptr->mtce_info.end());
/* find this key */
size_t pos = node_ptr->mtce_info.find(key + ':');
if ( pos != std::string::npos )
{
string mtce_info_tmp = node_ptr->mtce_info.substr(0,pos);
/* Now search for next ',' and/or '}' */
size_t pair_end_pos = node_ptr->mtce_info.find(',', pos) ;
if ( pair_end_pos != std::string::npos )
{
mtce_info_tmp.append(node_ptr->mtce_info.substr(pair_end_pos+1));
}
else
{
/* handle not leaving a stray ',' when there are no remaining
* elements in the dictionary ; avoid this k:v, */
if ( ! mtce_info_tmp.empty() )
mtce_info_tmp.pop_back();
/* handle leaving the dictionary empty completely
* empty (no "{}") if there are no more key value pairs */
if ( ! mtce_info_tmp.empty() )
mtce_info_tmp.append("}");
}
dlog ("%s mtcInfo dictionary before: %s after : %s\n",
node_ptr->hostname.c_str(),
node_ptr->mtce_info.c_str(),
mtce_info_tmp.c_str());
node_ptr->mtce_info = mtce_info_tmp ;
}
}
}
void nodeLinkClass::mtcInfo_log ( struct nodeLinkClass::node * node_ptr )
{
if ( node_ptr )
{
if ( node_ptr->mtce_info.length() > 2 )
{
ilog("%s %s",
node_ptr->hostname.c_str(),
node_ptr->mtce_info.substr(1,node_ptr->mtce_info.length()-2).c_str());
}
}
}
/* Lock Rules
*
* 1. Cannot lock this controller
@ -3061,7 +3308,7 @@ bool nodeLinkClass::can_uuid_be_locked ( string uuid , int & reason )
if ( node_ptr )
{
dlog1 ("%s Lock permission query\n", node_ptr->hostname.c_str());
/* Allow lock of already locked 'any' host */
if ( node_ptr->adminState == MTC_ADMIN_STATE__LOCKED )
{
@ -3082,7 +3329,7 @@ bool nodeLinkClass::can_uuid_be_locked ( string uuid , int & reason )
reason = FAIL_UNIT_ACTIVE ;
return (false);
}
/* Rule 2 - Cannot lock inactive controller if the floating storage
/* Rule 2 - Cannot lock inactive controller if the floating storage
* ceph monitor is locked */
if (( get_storage_backend() == CGCS_STORAGE_CEPH ) &&
( is_storage_mon_enabled () == false ))
@ -3918,71 +4165,6 @@ void nodeLinkClass::set_health ( string & hostname, int health )
}
}
/*************************************************************************************
*
* Name : manage_bmc_provisioning
*
* Description: This utility manages a change in bmc provisioning for
* bm region EXTERNAL mode. Creates provisioning logs and
* sends START and STOP monitoring commands to the hardware monitor.
*
* Warning : Should only be called when there is a change to BM provisioning.
* as it will first always first disable provisioning and then
* decides whether it needs to be re-enabled or not.
*
*************************************************************************************/
int nodeLinkClass::manage_bmc_provisioning ( struct node * node_ptr )
{
int rc = PASS ;
bool was_provisioned = node_ptr->bmc_provisioned ;
set_bm_prov ( node_ptr, false);
if ((hostUtil_is_valid_ip_addr ( node_ptr->bm_ip )) &&
(!node_ptr->bm_un.empty()))
{
if ( was_provisioned == true )
{
mtcAlarm_log ( node_ptr->hostname, MTC_LOG_ID__COMMAND_BM_REPROVISIONED );
}
else
{
mtcAlarm_log ( node_ptr->hostname, MTC_LOG_ID__COMMAND_BM_PROVISIONED );
}
set_bm_prov ( node_ptr, true );
}
else if ( was_provisioned == true )
{
send_hwmon_command(node_ptr->hostname,MTC_CMD_STOP_HOST);
mtcAlarm_log ( node_ptr->hostname, MTC_LOG_ID__COMMAND_BM_DEPROVISIONED );
}
/* Send hmond updated bm info */
ilog ("%s sending board management info update to hwmond\n", node_ptr->hostname.c_str() );
if ( ( rc = send_hwmon_command(node_ptr->hostname,MTC_CMD_MOD_HOST) ) == PASS )
{
if ( node_ptr->bmc_provisioned == true )
{
rc = send_hwmon_command(node_ptr->hostname,MTC_CMD_START_HOST);
}
else
{
rc = send_hwmon_command(node_ptr->hostname,MTC_CMD_STOP_HOST);
}
if ( rc )
{
wlog ("%s failed to send START or STOP command to hwmond\n", node_ptr->hostname.c_str());
}
}
else
{
wlog ("%s failed to send MODIFY command to hwmond\n", node_ptr->hostname.c_str());
}
return (rc);
}
bool nodeLinkClass::is_bm_ip_already_used ( string bm_ip )
{
if ( hostUtil_is_valid_ip_addr ( bm_ip ) == true )
@ -4003,15 +4185,15 @@ bool nodeLinkClass::is_bm_ip_already_used ( string bm_ip )
int nodeLinkClass::set_bm_type ( string hostname , string bm_type )
{
int rc = FAIL_HOSTNAME_LOOKUP ;
nodeLinkClass::node* node_ptr ;
node_ptr = nodeLinkClass::getNode ( hostname );
if ( node_ptr != NULL )
{
node_ptr->bm_type = bm_type ;
dlog ("%s '%s' updated to '%s'\n",
hostname.c_str(),
MTC_JSON_INV_BMTYPE,
dlog ("%s '%s' updated to '%s'\n",
hostname.c_str(),
MTC_JSON_INV_BMTYPE,
node_ptr->bm_type.c_str());
rc = PASS ;
}
@ -4062,6 +4244,67 @@ int nodeLinkClass::set_bm_ip ( string hostname , string bm_ip )
return (rc);
}
void nodeLinkClass::bmc_load_protocol ( struct nodeLinkClass::node * node_ptr )
{
string bmc_protocol_in_database =
mtcInfo_get ( node_ptr, MTCE_INFO_KEY__BMC_PROTOCOL );
if ( node_ptr->bm_type == "ipmi" )
{
ilog ("%s BMC access method set to 'ipmi' (host)",
node_ptr->hostname.c_str());
node_ptr->bmc_protocol = BMC_PROTOCOL__IPMITOOL ;
mtcInfo_set ( node_ptr, MTCE_INFO_KEY__BMC_PROTOCOL, BMC_PROTOCOL__IPMI_STR );
}
else if ( node_ptr->bm_type == "redfish" )
{
ilog ("%s BMC access method set to 'redfish' (host)",
node_ptr->hostname.c_str());
node_ptr->bmc_protocol = BMC_PROTOCOL__REDFISHTOOL ;
mtcInfo_set ( node_ptr, MTCE_INFO_KEY__BMC_PROTOCOL, BMC_PROTOCOL__REDFISH_STR );
}
else if (( node_ptr->bm_type == "dynamic" ) ||
( node_ptr->bm_type == "bmc" ))
{
if ( ! bmc_protocol_in_database.empty() )
{
if ( bmc_protocol_in_database == BMC_PROTOCOL__IPMI_STR )
{
ilog ("%s BMC access method set to 'ipmi' (from sysinv)",
node_ptr->hostname.c_str());
node_ptr->bmc_protocol = BMC_PROTOCOL__IPMITOOL ;
}
else if ( bmc_protocol_in_database == BMC_PROTOCOL__REDFISH_STR )
{
ilog ("%s BMC access method set to 'redfish' (from sysinv)",
node_ptr->hostname.c_str());
node_ptr->bmc_protocol = BMC_PROTOCOL__REDFISHTOOL ;
}
else if ( bmc_protocol_in_database == BMC_PROTOCOL__DYNAMIC_STR )
{
ilog ("%s BMC method will be learned (from sysinv)",
node_ptr->hostname.c_str());
node_ptr->bmc_protocol = BMC_PROTOCOL__DYNAMIC ;
}
else
{
ilog ("%s BMC method will be learned (unexpected:%s)",
node_ptr->hostname.c_str(),
bmc_protocol_in_database.c_str());
node_ptr->bmc_protocol = BMC_PROTOCOL__DYNAMIC ;
}
}
else
{
ilog ("%s BMC method will be learned (%s)",
node_ptr->hostname.c_str(),
node_ptr->bm_type.c_str());
node_ptr->bmc_protocol = BMC_PROTOCOL__DYNAMIC ;
}
}
mtcInvApi_update_mtcInfo ( node_ptr );
}
void nodeLinkClass::bmc_access_data_init ( struct nodeLinkClass::node * node_ptr )
{
if ( node_ptr )
@ -4076,34 +4319,18 @@ void nodeLinkClass::bmc_access_data_init ( struct nodeLinkClass::node * node_ptr
node_ptr->power_status_query_active = false ;
node_ptr->power_status_query_done = false ;
node_ptr->bm_ping_info.stage = PINGUTIL_MONITOR_STAGE__OPEN ;
mtcTimer_reset ( node_ptr->bm_ping_info.timer );
node_ptr->bm_ping_info.timer_handler = &mtcTimer_handler ;
node_ptr->bm_ping_info.ip = node_ptr->bm_ip ;
node_ptr->bmc_protocol_learning = false ;
/* remove all the bmc related temporary files created
* for this host and process */
bmcUtil_remove_files ( node_ptr->hostname, node_ptr->bmc_protocol );
bmcUtil_remove_files ( node_ptr->hostname, BMC_PROTOCOL__IPMITOOL );
bmcUtil_remove_files ( node_ptr->hostname, BMC_PROTOCOL__REDFISHTOOL );
if ( this->bmc_access_method == "ipmi" )
{
blog2 ("%s BMC access method set to 'ipmi'",
node_ptr->hostname.c_str());
node_ptr->bmc_protocol = BMC_PROTOCOL__IPMITOOL ;
node_ptr->bmc_protocol_learning = false ;
node_ptr->bmc_protocol_learned = true ;
}
else if ( this->bmc_access_method == "redfish" )
{
blog2 ("%s BMC access method set to 'redfish'",
node_ptr->hostname.c_str());
node_ptr->bmc_protocol = BMC_PROTOCOL__REDFISHTOOL ;
node_ptr->bmc_protocol_learning = false ;
node_ptr->bmc_protocol_learned = true ;
}
else
{
blog2 ("%s BMC access method will be learned",
node_ptr->hostname.c_str());
node_ptr->bmc_protocol_learned = false ;
node_ptr->bmc_protocol_learning = false ;
node_ptr->bmc_protocol = BMC_PROTOCOL__IPMITOOL ;
}
bmcUtil_info_init ( node_ptr->bmc_info );
}
}
@ -4126,22 +4353,11 @@ int nodeLinkClass::set_bm_prov ( struct nodeLinkClass::node * node_ptr, bool sta
int rc = FAIL_HOSTNAME_LOOKUP ;
if ( node_ptr != NULL )
{
ilog ("%s bmc %sprovision request (provisioned:%s)\n",
node_ptr->hostname.c_str(),
state ? "" : "de",
node_ptr->bmc_provisioned ? "Yes" : "No" );
/* Clear the alarm if we are starting fresh from an unprovisioned state */
if (( node_ptr->bmc_provisioned == false ) && ( state == true ))
/* All Provisioning Cases */
if ( state == true )
{
ilog ("%s starting BM ping monitor to address '%s'\n",
node_ptr->hostname.c_str(),
node_ptr->bm_ip.c_str());
node_ptr->bm_ping_info.ip = node_ptr->bm_ip ;
node_ptr->bm_ping_info.stage = PINGUTIL_MONITOR_STAGE__OPEN ;
bmc_access_data_init ( node_ptr );
node_ptr->bm_ping_info.timer_handler = &mtcTimer_handler ;
bmc_load_protocol ( node_ptr );
barbicanSecret_type * secret = secretUtil_find_secret( node_ptr->uuid );
if ( secret )
@ -4149,46 +4365,44 @@ int nodeLinkClass::set_bm_prov ( struct nodeLinkClass::node * node_ptr, bool sta
secret->reference.clear() ;
secret->payload.clear() ;
secret->stage = MTC_SECRET__START ;
mtcTimer_reset ( node_ptr->bm_timer );
mtcTimer_start ( node_ptr->bm_timer, mtcTimer_handler, SECRET_START_DELAY );
}
send_hwmon_command(node_ptr->hostname, MTC_CMD_ADD_HOST);
send_hwmon_command(node_ptr->hostname, MTC_CMD_START_HOST);
/* start the connection timer - if it expires before we
* are 'accessible' then the BM Alarm is raised.
* Timer is further managed in mtcNodeHdlrs.cpp */
plog ("%s bmc access timer started (%d secs)\n", node_ptr->hostname.c_str(), MTC_MINS_2);
blog ("%s bmc access timer started (%d secs)\n", node_ptr->hostname.c_str(), MTC_MINS_2);
mtcTimer_reset ( node_ptr->bmc_access_timer );
mtcTimer_start ( node_ptr->bmc_access_timer, mtcTimer_handler, MTC_MINS_2 );
ilog ("%s bmc %sprovisioned",
node_ptr->hostname.c_str(),
node_ptr->bmc_provisioned ? "re":"");
}
/* handle the case going from provisioned to not provisioned */
else if (( node_ptr->bmc_provisioned == true ) && ( state == false ))
/* Deprovision Case */
else
{
/* remove the old BMC info file if present */
string bmc_info_path_n_filename = BMC_OUTPUT_DIR + node_ptr->hostname ;
daemon_remove_file ( bmc_info_path_n_filename.data() );
ilog ("%s deprovisioning bmc ; accessible:%s\n",
node_ptr->hostname.c_str(),
node_ptr->bmc_accessible ? "Yes" : "No" );
if ( node_ptr->bmc_provisioned == true )
{
ilog ("%s bmc deprovisioning", node_ptr->hostname.c_str());
}
pingUtil_fini ( node_ptr->bm_ping_info );
bmc_access_data_init ( node_ptr );
node_ptr->bmc_protocol = BMC_PROTOCOL__DYNAMIC ;
mtcTimer_reset ( node_ptr->bmc_audit_timer );
if ( !thread_idle( node_ptr->bmc_thread_ctrl ) )
{
thread_kill ( node_ptr->bmc_thread_ctrl , node_ptr->bmc_thread_info);
}
/* send a delete to hwmon if the provisioning data is NONE */
if ( hostUtil_is_valid_bm_type ( node_ptr->bm_type ) == false )
{
send_hwmon_command(node_ptr->hostname, MTC_CMD_DEL_HOST);
}
}
mtcInfo_clr ( node_ptr, MTCE_INFO_KEY__BMC_PROTOCOL);
mtcInvApi_update_mtcInfo ( node_ptr );
/* send a delete to hwmon if was provisioned */
send_hwmon_command(node_ptr->hostname, MTC_CMD_MOD_HOST);
}
node_ptr->bmc_provisioned = state ;
}
return (rc);
@ -4238,11 +4452,7 @@ string nodeLinkClass::get_hwmon_info ( string hostname )
{
string hwmon_info = "" ;
hwmon_info.append( "{ \"personality\":\"" ) ;
hwmon_info.append( node_ptr->type );
hwmon_info.append( "\"");
hwmon_info.append( ",\"hostname\":\"" ) ;
hwmon_info.append( "{ \"hostname\":\"" ) ;
hwmon_info.append( node_ptr->hostname );
hwmon_info.append( "\"");
@ -4250,14 +4460,26 @@ string nodeLinkClass::get_hwmon_info ( string hostname )
hwmon_info.append( node_ptr->bm_ip );
hwmon_info.append( "\"");
hwmon_info.append( ",\"bm_type\":\"");
hwmon_info.append( node_ptr->bm_type );
hwmon_info.append( "\"");
hwmon_info.append( ",\"bm_username\":\"");
hwmon_info.append( node_ptr->bm_un );
hwmon_info.append( "\"");
hwmon_info.append( ",\"bm_http\":\"");
hwmon_info.append( node_ptr->bm_http_mode );
hwmon_info.append( "\"");
hwmon_info.append( ",\"");
hwmon_info.append( MTCE_INFO_KEY__BMC_PROTOCOL );
hwmon_info.append( "\":\"");
if ( node_ptr->bmc_protocol == BMC_PROTOCOL__REDFISHTOOL )
hwmon_info.append( BMC_PROTOCOL__REDFISH_STR );
else if ( node_ptr->bmc_protocol == BMC_PROTOCOL__IPMITOOL )
hwmon_info.append( BMC_PROTOCOL__IPMI_STR );
else
hwmon_info.append(BMC_PROTOCOL__DYNAMIC_STR);
hwmon_info.append( "\"");
hwmon_info.append( ",\"uuid\":\"" ) ;
hwmon_info.append( node_ptr->uuid );
hwmon_info.append( "\" }");
@ -4268,13 +4490,11 @@ string nodeLinkClass::get_hwmon_info ( string hostname )
return ("");
}
int nodeLinkClass::manage_shadow_change ( string hostname )
{
int rc = FAIL ;
if ( ! hostname.empty() )
{
{
nodeLinkClass::node* node_ptr ;
node_ptr = nodeLinkClass::getNode ( hostname );
if ( node_ptr != NULL )
@ -4927,7 +5147,9 @@ int nodeLinkClass::declare_service_ready ( string & hostname,
plog ("%s %s ready event\n",
hostname.c_str(),
MTC_SERVICE_HWMOND_NAME);
if ( node_ptr->bmc_provisioned == true )
if (( node_ptr->bmc_accessible == true ) &&
(( node_ptr->bmc_protocol == BMC_PROTOCOL__IPMITOOL ) ||
( node_ptr->bmc_protocol == BMC_PROTOCOL__REDFISHTOOL )))
{
send_hwmon_command ( node_ptr->hostname, MTC_CMD_ADD_HOST );
send_hwmon_command ( node_ptr->hostname, MTC_CMD_START_HOST );
@ -8648,7 +8870,7 @@ void nodeLinkClass::mem_log_general_mtce_hosts ( void )
void nodeLinkClass::mem_log_bm ( struct nodeLinkClass::node * node_ptr )
{
char str[MAX_MEM_LOG_DATA] ;
snprintf (&str[0], MAX_MEM_LOG_DATA, "%s\tBMC %s %s:%s prov:%s acc:%s ping:%s learn:%s:%s Query:%s:%s Timer:%s:%s\n",
snprintf (&str[0], MAX_MEM_LOG_DATA, "%s\tBMC %s %s:%s prov:%s acc:%s ping:%s learning:%s Query:%s:%s Timer:%s:%s\n",
node_ptr->hostname.c_str(),
bmcUtil_getProtocol_str(node_ptr->bmc_protocol).c_str(),
node_ptr->bm_un.c_str(),
@ -8656,7 +8878,6 @@ void nodeLinkClass::mem_log_bm ( struct nodeLinkClass::node * node_ptr )
node_ptr->bmc_provisioned ? "Y" : "N",
node_ptr->bmc_accessible ? "Y" : "N",
node_ptr->bm_ping_info.ok ? "Y" : "N",
node_ptr->bmc_protocol_learned ? "Y" : "N",
node_ptr->bmc_protocol_learning ? "Y" : "N",
node_ptr->bmc_info_query_active ? "Y" : "N",
node_ptr->bmc_info_query_done ? "Y" : "N",
@ -8727,11 +8948,12 @@ void nodeLinkClass::mem_log_state2 ( struct nodeLinkClass::node * node_ptr )
char str[MAX_MEM_LOG_DATA] ;
string aa = adminAction_enum_to_str(node_ptr->adminAction) ;
snprintf (&str[0], MAX_MEM_LOG_DATA, "%s\tmtcAction:%s invAction:%s Task:%s\n",
snprintf (&str[0], MAX_MEM_LOG_DATA, "%s\tAction:%s:%s Task:%s Info:%s\n",
node_ptr->hostname.c_str(),
aa.c_str(),
node_ptr->action.c_str(),
node_ptr->task.c_str());
node_ptr->task.c_str(),
node_ptr->mtce_info.c_str());
mem_log (str);
}

44
mtce/src/common/nodeClass.h
View File

@ -134,6 +134,9 @@ private:
* taking on this host */
std::string task ;
/** String containing key value pairs acting like a dictionary */
std::string mtce_info ;
/** Administrative action from inventory */
std::string action ;
@ -580,6 +583,7 @@ private:
/** The password of the host's board management controller */
string bm_pw ;
int bm_pw_wait_log_throttle ;
/** A string label that represents the board management
* controller type for this host */
@ -588,6 +592,9 @@ private:
/** The operator provisioned board management hostname */
string bm_un ;
/** The security mode for BMC Access http requests */
string bm_http_mode ;
/** the command to use in the bmc thread.
* introduced for redfish reset sub command ; reset type */
string bm_cmd;
@ -605,10 +612,6 @@ private:
**/
bool bmc_accessible;
/* tell the host level bmc_handler that this hosts access
* method has changed */
bool bmc_access_method_changed ;
/** @} private_boad_management_variables */
/**
@ -671,11 +674,6 @@ private:
* defaults to 0 or BMC_PROTOCOL__IPMITOOL */
bmc_protocol_enum bmc_protocol ;
/* set true once the best BMC protocol has been learned
*
* looked at in the bmc_handler to decide learn or use bmc_protocol */
bool bmc_protocol_learned ;
/* set true while bmc protocol learning is in progress */
bool bmc_protocol_learning ;
@ -853,6 +851,13 @@ private:
void ctl_mtcAlive_gate ( struct nodeLinkClass::node * node_ptr, bool gate_state );
void set_mtcAlive ( struct nodeLinkClass::node * node_ptr, int interface );
int mtcInfo_set ( struct nodeLinkClass::node * node_ptr, string key, string value );
string mtcInfo_get ( struct nodeLinkClass::node * node_ptr, string key );
void mtcInfo_clr ( struct nodeLinkClass::node * node_ptr, string key );
void mtcInfo_log ( struct nodeLinkClass::node * node_ptr );
int set_mtcInfo ( struct nodeLinkClass::node * node_ptr, string & mtc_info );
/*****************************************************************************
*
* Name : bmc_command_send
@ -1132,6 +1137,7 @@ private:
int mtcInvApi_update_states ( struct nodeLinkClass::node * node_ptr, string admin, string oper, string avail );
int mtcInvApi_update_states_now ( struct nodeLinkClass::node * node_ptr, string admin, string oper, string avail, string oper_subf, string avail_subf);
int mtcInvApi_update_state ( struct nodeLinkClass::node * node_ptr, string state, string value );
int mtcInvApi_update_mtcInfo ( struct nodeLinkClass::node * node_ptr );
/* Private SM API */
int mtcSmgrApi_request ( struct nodeLinkClass::node * node_ptr, mtc_cmd_enum operation, int retries );
@ -1139,7 +1145,8 @@ private:
/* Private VIM API */
int mtcVimApi_state_change ( struct nodeLinkClass::node * node_ptr, libEvent_enum operation, int retries );
int set_bm_prov ( struct nodeLinkClass::node * node_ptr, bool state );
int set_bm_prov ( struct nodeLinkClass::node * node_ptr, bool state );
void bmc_load_protocol ( struct nodeLinkClass::node * node_ptr );
void set_uptime ( struct nodeLinkClass::node * node_ptr, unsigned int uptime, bool force );
@ -1375,8 +1382,6 @@ public:
/* the main fsm entrypoint to service all hosts */
void fsm ( void ) ;
void bmc_access_method_change_notifier ( void );
/** This controller's hostname set'er */
void set_my_hostname ( string hostname );
@ -1398,6 +1403,12 @@ public:
/** get ip address for any hostname */
string get_hostaddr ( string & hostname );
int mtcInfo_set ( string hostname, string key, string value );
string mtcInfo_get ( string hostname, string key );
void mtcInfo_clr ( string hostname, string key );
int set_mtcInfo ( string hostname, string & mtc_info );
/** get mac address for any hostname and specified interface */
string get_hostIfaceMac ( string & hostname, int iface );
@ -1500,13 +1511,6 @@ public:
std::list<string> mnfa_awol_list ;
void mnfa_timeout_handler ( void );
/* How to communicate with the BMCs in this lab.
* Options are: ipmi, redfish, learn */
string bmc_access_method ;
/* handle bmc access method change by service parameter */
bool bmc_access_method_changed ;
/** Return the number of inventoried hosts */
int num_hosts ( void );
@ -1780,8 +1784,6 @@ public:
bool is_bm_ip_already_used ( string bm_ip );
int manage_bmc_provisioning ( struct node * node_ptr );
string get_bm_ip ( string hostname );
string get_bm_un ( string hostname );
string get_bm_type ( string hostname );

7
mtce/src/heartbeat/hbsStubs.cpp
View File

@ -306,13 +306,18 @@ int nodeLinkClass::mtcVimApi_state_change ( struct nodeLinkClass::node * node_pt
libEvent_enum operation,
int retries )
{
UNUSED(node_ptr);
UNUSED(operation);
UNUSED(retries);
return(PASS);
}
int nodeLinkClass::mtcInvApi_update_mtcInfo (struct nodeLinkClass::node * node_ptr)
{
UNUSED(node_ptr);
return (PASS);
}
int nodeLinkClass::doneQueue_purge ( struct nodeLinkClass::node * node_ptr ) { node_ptr = node_ptr ; return (PASS) ; }
int nodeLinkClass::workQueue_purge ( struct nodeLinkClass::node * node_ptr ) { node_ptr = node_ptr ; return (PASS) ; }
int nodeLinkClass::mtcCmd_doneQ_purge ( struct nodeLinkClass::node * node_ptr ) { node_ptr = node_ptr ; return (PASS) ; }

175
mtce/src/hwmon/hwmonClass.cpp
View File

@ -31,6 +31,7 @@ hwmonHostClass::hwmonHostClass()
hosts = 0 ;
host_deleted = false ;
config_reload = false ;
hostlist.clear() ;
return ;
}
@ -91,10 +92,6 @@ void hwmonHostClass::bmc_data_init ( struct hwmonHostClass::hwmon_host * host_pt
host_ptr->addStage = HWMON_ADD__START;
host_ptr->sensor_query_count = 0 ;
/* remove all the bmc related temporary files created
* for this host and process */
bmcUtil_remove_files ( host_ptr->hostname, host_ptr->protocol );
}
/*
@ -514,9 +511,10 @@ int hwmonHostClass::set_bm_prov ( struct hwmonHostClass::hwmon_host * host_ptr,
if ( connect || reconnect )
{
ilog ("%s board management controller is being %sprovisioned\n",
ilog ("%s bmc %sprovisioning ; using %s",
host_ptr->hostname.c_str(),
host_ptr->bm_provisioned ? "re":"" );
host_ptr->bm_provisioned ? "re":"",
bmcUtil_getProtocol_str(host_ptr->protocol).c_str());
/* ---------------------------------------
* Init bmc data based on monitoring mode
@ -544,16 +542,27 @@ int hwmonHostClass::set_bm_prov ( struct hwmonHostClass::hwmon_host * host_ptr,
host_ptr->thread_extra_info.bm_pw.clear() ;
host_ptr->thread_extra_info.bm_ip = host_ptr->bm_ip ;
host_ptr->thread_extra_info.bm_un = host_ptr->bm_un ;
if ( reconnect )
{
host_ptr->bmc_thread_ctrl.retries = 0 ;
host_ptr->bmc_thread_ctrl.runcount = 0 ;
host_ptr->bmc_thread_ctrl.status = PASS ;
}
}
/* handle the case going from provisioned to not provisioned */
if (( host_ptr->bm_provisioned == true ) && ( state == false ))
{
ilog ("%s board management controller is being deprovisioned\n", host_ptr->hostname.c_str());
clear_bm_assertions ( host_ptr );
pingUtil_fini ( host_ptr->ping_info );
bmc_data_init ( host_ptr );
ilog ("%s bmc is deprovisioned\n", host_ptr->hostname.c_str());
}
/* remove all the bmc related temporary files created
* for this host and process */
bmcUtil_remove_files ( host_ptr->hostname, BMC_PROTOCOL__REDFISHTOOL );
bmcUtil_remove_files ( host_ptr->hostname, BMC_PROTOCOL__IPMITOOL );
host_ptr->bm_provisioned = state ;
}
return (rc);
@ -578,22 +587,53 @@ int hwmonHostClass::mod_host ( node_inv_type & inv )
{
rc = PASS ;
bool modify_bm = false ;
bool need_relearn = false ;
/* save the http mode */
host_ptr->bm_http_mode = inv.bm_http ;
/* Manage getting the bmc access protocol method */
bmc_protocol_enum protocol ;
if ( inv.bm_proto == BMC_PROTOCOL__REDFISH_STR )
protocol = BMC_PROTOCOL__REDFISHTOOL ;
else if ( inv.bm_proto == BMC_PROTOCOL__IPMI_STR )
protocol = BMC_PROTOCOL__IPMITOOL ;
else
protocol = BMC_PROTOCOL__DYNAMIC ;
if ( host_ptr->protocol != protocol )
{
ilog ("%s modify bmc protocol from %s to %s",
inv.name.c_str(),
bmcUtil_getProtocol_str(host_ptr->protocol).c_str(),
bmcUtil_getProtocol_str(protocol).c_str());
if ( hostUtil_is_valid_ip_addr ( inv.bm_ip ) )
need_relearn = true ;
host_ptr->protocol = protocol ;
modify_bm = true ;
}
if ( host_ptr->bm_ip.compare( inv.bm_ip ) )
{
ilog ("%s modify board management 'ip' from '%s' to '%s'\n",
ilog ("%s modify bmc 'ip' from '%s' to '%s'\n",
inv.name.c_str(),
host_ptr->bm_ip.c_str(),
inv.bm_ip.c_str());
host_ptr->bm_ip = inv.bm_ip ;
if ( hostUtil_is_valid_ip_addr ( inv.bm_ip ) )
need_relearn = true ;
host_ptr->bm_ip = inv.bm_ip ;
modify_bm = true ;
}
if ( host_ptr->bm_un.compare( inv.bm_un ) )
{
ilog ("%s modify board management 'username' from '%s' to '%s'\n",
ilog ("%s modify bmc 'username' from '%s' to '%s'\n",
inv.name.c_str(),
host_ptr->bm_un.c_str(),
inv.bm_un.c_str());
@ -603,50 +643,43 @@ int hwmonHostClass::mod_host ( node_inv_type & inv )
modify_bm = true ;
}
if ( host_ptr->bm_type.compare( inv.bm_type ) )
{
ilog ("%s modify board management 'type' from '%s' to '%s'\n",
inv.name.c_str(),
host_ptr->bm_type.c_str(),
inv.bm_type.c_str());
host_ptr->bm_type = inv.bm_type ;
modify_bm = true ;
}
/* force password relearn for all provisioning changes */
host_ptr->bm_pw.clear();
if ( modify_bm == true )
{
ilog ("%s modify summary %s %s@%s ... provisioned = %s\n",
ilog ("%s modify bmc summary %s %s@%s",
inv.name.c_str(),
host_ptr->bm_type.c_str(),
bmcUtil_getProtocol_str(host_ptr->protocol).c_str(),
host_ptr->bm_un.c_str(),
host_ptr->bm_ip.c_str(),
host_ptr->bm_provisioned ? "Yes" : "No" );
if ( host_ptr->bm_provisioned == true )
host_ptr->bm_ip.c_str());
if (( host_ptr->protocol != BMC_PROTOCOL__DYNAMIC ) &&
( hostUtil_is_valid_ip_addr (host_ptr->bm_ip) == true ) &&
( hostUtil_is_valid_username (host_ptr->bm_un) == true ))
{
/* if we have a credentials only change then disable the sensor
* model only to get re-enabled if sensor monitoring is
* successful with the new credentials */
if (( hostUtil_is_valid_bm_type (host_ptr->bm_type) == true ) &&
( host_ptr->bm_un.compare(NONE)))
{
bmc_set_group_state ( host_ptr, "disabled" );
bmc_disable_sensors ( host_ptr );
}
set_bm_prov ( host_ptr, true );
}
else
{
if ( host_ptr->groups )
bmc_delete_sensor_model ( host_ptr );
set_bm_prov ( host_ptr, false );
need_relearn = false ;
}
if (( hostUtil_is_valid_bm_type (host_ptr->bm_type) == true ) &&
( hostUtil_is_valid_ip_addr (host_ptr->bm_ip) == true ) &&
!host_ptr->bm_un.empty())
if (( need_relearn == true ) && ( host_ptr->groups ))
{
rc = set_bm_prov ( host_ptr, true );
ilog ("%s sensor model will be deleted and relearned", inv.name.c_str());
bmc_learn_sensor_model (hostBase.get_uuid( inv.name ));
}
}
else
{
/* Only reprovision if the provisioning data has changed */
dlog ("%s bmc provisioning unchanged\n", host_ptr->hostname.c_str());
return (rc);
}
}
else
@ -712,7 +745,14 @@ int hwmonHostClass::add_host ( node_inv_type & inv )
/* Add board management stuff */
host_ptr->bm_ip = inv.bm_ip ;
host_ptr->bm_un = inv.bm_un ;
host_ptr->bm_type = inv.bm_type ;
host_ptr->bm_http_mode= inv.bm_http ;
if ( inv.bm_proto == BMC_PROTOCOL__REDFISH_STR )
host_ptr->protocol = BMC_PROTOCOL__REDFISHTOOL ;
else if ( inv.bm_proto == BMC_PROTOCOL__IPMI_STR )
host_ptr->protocol = BMC_PROTOCOL__IPMITOOL ;
else
host_ptr->protocol = BMC_PROTOCOL__DYNAMIC ;
/* default the socket number to closed */
host_ptr->ping_info.sock = 0 ;
@ -739,6 +779,7 @@ int hwmonHostClass::add_host ( node_inv_type & inv )
host_ptr->monitor_ctrl.stage = HWMON_SENSOR_MONITOR__START ;
host_ptr->monitor_ctrl.last_sample_time = 0 ;
host_ptr->monitor_ctrl.this_sample_time = 0 ;
host_ptr->bmc_thread_ctrl.retries = 0 ;
host_ptr->sensor_query_count = 0 ;
/* Sensor Monitoring Thread 'Extra Request Information' */
@ -756,11 +797,9 @@ int hwmonHostClass::add_host ( node_inv_type & inv )
host_ptr->hostname,
THREAD_NAME__BMC);
/* TODO: create a is_bm_info_valid */
if ( ( hostUtil_is_valid_ip_addr (host_ptr->bm_ip) == true ) &&
( hostUtil_is_valid_bm_type (host_ptr->bm_type) == true ) &&
( !host_ptr->bm_un.empty() ) &&
( host_ptr->bm_un.compare(NONE)) )
if (( host_ptr->protocol != BMC_PROTOCOL__DYNAMIC ) &&
( hostUtil_is_valid_ip_addr (host_ptr->bm_ip) == true ) &&
( hostUtil_is_valid_username (host_ptr->bm_un) == true ))
{
set_bm_prov ( host_ptr, true );
}
@ -768,20 +807,15 @@ int hwmonHostClass::add_host ( node_inv_type & inv )
{
set_bm_prov ( host_ptr, false );
}
ilog ("%s BMC is %sprovisioned\n", host_ptr->hostname.c_str(), host_ptr->bm_provisioned ? "" : "not " );
host_ptr->bmc_fw_version.clear();
host_ptr->group_index = 0 ;
/* Set default BMC protocol */
host_ptr->protocol = bmcUtil_read_hwmond_protocol(host_ptr->hostname) ;
/* Init sensor model relearn controls, state and status */
host_ptr->relearn = false ;
host_ptr->relearn_request = false ;
host_ptr->relearn_retry_counter = 0 ;
host_ptr->relearn_done_date.clear();
init_model_attributes ( host_ptr->model_attributes_preserved );
/* Add to the end of inventory */
@ -983,19 +1017,6 @@ string hwmonHostClass::get_bm_ip ( string hostname )
return ("");
}
/** Get this hosts board management TYPE ilo3/ilo4/quanta/etc */
string hwmonHostClass::get_bm_type ( string hostname )
{
hwmonHostClass::hwmon_host * hwmon_host_ptr ;
hwmon_host_ptr = hwmonHostClass::getHost ( hostname );
if ( hwmon_host_ptr != NULL )
{
return (hwmon_host_ptr->bm_type);
}
elog ("%s bm type lookup failed\n", hostname.c_str() );
return ("");
}
/** Get this hosts board management user name */
string hwmonHostClass::get_bm_un ( string hostname )
{
@ -1016,24 +1037,6 @@ string hwmonHostClass::get_bm_un ( string hostname )
return ("");
}
string hwmonHostClass::get_relearn_done_date ( string hostname )
{
hwmonHostClass::hwmon_host * hwmon_host_ptr ;
hwmon_host_ptr = hwmonHostClass::getHost ( hostname );
if ( hwmon_host_ptr != NULL )
{
if ( !hwmon_host_ptr->relearn_done_date.empty())
{
return (hwmon_host_ptr->relearn_done_date);
}
}
elog ("%s relearn done date empty or hostname lookup failed\n", hostname.c_str());
return (pt());
}
struct hwmonHostClass::hwmon_host * hwmonHostClass::getHost_timer ( timer_t tid )
{
/* check for empty list condition */
@ -1689,15 +1692,11 @@ int hwmonHostClass::bmc_learn_sensor_model ( string uuid )
wlog ("%s sensor model relearn already in progress\n",
ptr->hostname.c_str());
wlog ("%s ... projected completion time: %s\n",
ptr->hostname.c_str(),
ptr->relearn_done_date.c_str());
rc = RETRY ;
}
else
{
ilog ("%s sensor model relearn request accepted\n",
blog ("%s sensor model relearn request accepted\n",
ptr->hostname.c_str());
ptr->bmc_fw_version.clear();
@ -2221,11 +2220,11 @@ void hwmonHostClass::mem_log_options ( struct hwmonHostClass::hwmon_host * hwmon
void hwmonHostClass::mem_log_bm ( struct hwmonHostClass::hwmon_host * hwmon_host_ptr )
{
char str[MAX_MEM_LOG_DATA] ;
snprintf (&str[0], MAX_MEM_LOG_DATA, "%s\tbm_ip:%s bm_un:%s bm_type:%s\n",
snprintf (&str[0], MAX_MEM_LOG_DATA, "%s\tbm_ip:%s bm_un:%s (%s)\n",
hwmon_host_ptr->hostname.c_str(),
hwmon_host_ptr->bm_ip.c_str(),
hwmon_host_ptr->bm_un.c_str(),
hwmon_host_ptr->bm_type.c_str());
bmcUtil_getProtocol_str(hwmon_host_ptr->protocol).c_str());
mem_log (str);
}

16
mtce/src/hwmon/hwmonClass.h
View File

@ -40,10 +40,6 @@ class hwmonHostClass
/** The PW of the host's board management controller */
string bm_pw ;
/** A string label that represents the board management
* controller type for this host */
string bm_type ;
/** The operator provisioned board management hostname */
string bm_un ;
@ -87,6 +83,9 @@ class hwmonHostClass
/** set to the protocol used to communicate with this server's BMC */
bmc_protocol_enum protocol ;
/** http or https security mode */
string bm_http_mode ;
/** Pointer to the previous host in the list */
struct hwmon_host * prev;
@ -207,12 +206,6 @@ class hwmonHostClass
* Used to avoid repeating some retry operations. */
int relearn_retry_counter ;
/* Store the date/time when learning mode will be disabled.
* Put into error message to tell the administrator when the
* next sensor relearn is permitted when the current request
* is rejected due to already being in relearn mode. */
string relearn_done_date ;
/* a structure used to preserved some key sensor model attributes
* so that they can be restored over/after the relearn action */
model_attr_type model_attributes_preserved ;
@ -465,12 +458,9 @@ class hwmonHostClass
bool is_bm_provisioned ( string hostname );
string get_bm_ip ( string hostname );
string get_bm_type ( string hostname );
string get_bm_un ( string hostname );
string get_hostname ( string uuid ); /**< lookup hostname from the host uuid */
string get_relearn_done_date ( string hostname );
int hosts ;
/* This bool is set in the daemon_configure case to inform the

2
mtce/src/hwmon/hwmonFsm.cpp
View File

@ -145,7 +145,7 @@ void hwmonHostClass::hwmon_fsm ( void )
if ( secret->stage == MTC_SECRET__GET_PWD_RECV )
{
host_ptr->bm_pw = secret->payload ;
host_ptr->bm_pw = host_ptr->thread_extra_info.bm_pw = secret->payload ;
ilog ("%s bmc credentials received",
hostname.c_str());
}

22
mtce/src/hwmon/hwmonGroup.cpp
View File

@ -267,6 +267,10 @@ canned_group_enum bmc_get_groupenum ( string & hostname,
canned_group_enum group_enum = HWMON_CANNED_GROUP__NULL ;
if ( !unittype.empty() )
{
/* convert sensorname and unittype to lower case for compares below */
string _unittype = tolowercase (unittype);
string _sensorname = tolowercase (sensorname);
/* search canned groups for one having units that match this
* sensor sample. */
for ( int canned_group = (HWMON_CANNED_GROUP__NULL+1) ; canned_group < HWMON_CANNED_GROUPS ; ++canned_group )
@ -291,7 +295,7 @@ canned_group_enum bmc_get_groupenum ( string & hostname,
/* handle some special cases */
/* 1. Quanta Power Sensors */
if (( unittype.compare("discrete") == 0 ) &&
if (( _unittype.compare("discrete") == 0 ) &&
((sensorname.find("PSU Redundancy") != std::string::npos ) ||
(sensorname.find("PSU1 Status") != std::string::npos ) ||
(sensorname.find("PSU2 Status") != std::string::npos )))
@ -303,7 +307,7 @@ canned_group_enum bmc_get_groupenum ( string & hostname,
canned_group_array[group_enum].group_name,
sensorname.c_str());
}
else if (( unittype.compare("discrete") == 0 ) &&
else if (( _unittype.compare("discrete") == 0 ) &&
((sensorname.find("MB Thermal Trip") != std::string::npos ) ||
(sensorname.find("PCH Thermal Trip") != std::string::npos )))
{
@ -315,10 +319,9 @@ canned_group_enum bmc_get_groupenum ( string & hostname,
sensorname.c_str());
}
/* 1. HP Fans show up as 'percent' sensor type with Fan in the name */
else if (( unittype.compare("percent") == 0 ) &&
((sensorname.find("Fan") != std::string::npos ) ||
(sensorname.find("fan") != std::string::npos )))
/* HP Fans show up as 'percent' sensor type with Fan in the name */
else if (( _unittype == "percent" ) &&
( _sensorname.find("fan") != std::string::npos ))
{
group_enum = HWMON_CANNED_GROUP__FANS ;
@ -327,10 +330,9 @@ canned_group_enum bmc_get_groupenum ( string & hostname,
canned_group_array[group_enum].group_name,
sensorname.c_str());
}
/* 1. HP Fans show up as 'percent' sensor type with Fan in the name */
else if (( unittype.compare("percent") == 0 ) &&
((sensorname.find("Usage") != std::string::npos ) ||
(sensorname.find("usage") != std::string::npos )))
/* HP sensor usage shows up as 'percent' sensor type with usage in the name */
else if (( _unittype == "percent") &&
( _sensorname.find("usage") != std::string::npos ))
{
group_enum = HWMON_CANNED_GROUP__USAGE ;

117
mtce/src/hwmon/hwmonHdlr.cpp
View File

@ -446,38 +446,12 @@ int hwmonHostClass::add_host_handler ( struct hwmonHostClass::hwmon_host * host_
mtcTimer_start ( host_ptr->addTimer, hwmonTimer_handler, delay );
break ;
}
/* get protocol used for last relearn from the file */
host_ptr->protocol = bmcUtil_read_hwmond_protocol ( host_ptr->hostname ) ;
}
else
{
string power_state ;
bmc_protocol_enum protocol ;
if ( bmcUtil_read_bmc_info( host_ptr->hostname, power_state, protocol ))
{
ilog ("%s no sensor model in database ; must be learned\n",
host_ptr->hostname.c_str());
if ( protocol != host_ptr->protocol)
{
ilog ("%s bmc protocol changed to %s",
host_ptr->hostname.c_str(),
bmcUtil_getProtocol_str(protocol).c_str());
}
host_ptr->protocol = protocol ;
bmcUtil_write_hwmond_protocol ( host_ptr->hostname, protocol ) ;
host_ptr->general_log_throttle = 0 ;
}
else
{
/* mtc has not yet determined bmc protocol */
mtcTimer_start ( host_ptr->addTimer, hwmonTimer_handler, MTC_SECS_5 );
/* log every minute ; 5*12 */
ilog_throttled (host_ptr->general_log_throttle, 12,
"%s waiting for bmc protocol from mtce ; %d seconds between retries\n",
host_ptr->hostname.c_str(), MTC_SECS_5);
break;
}
ilog ("%s no sensor model ; must be learned ; using %s\n",
host_ptr->hostname.c_str(),
bmcUtil_getProtocol_str(host_ptr->protocol).c_str());
}
addStageChange ( host_ptr , HWMON_ADD__DONE );
}
@ -713,41 +687,21 @@ int hwmonHostClass::bmc_sensor_monitor ( struct hwmonHostClass::hwmon_host * hos
return (RETRY);
}
relearn_time = MTC_MINS_5 ;
relearn_time = MTC_MINS_2;
/* enter relearn mode */
host_ptr->relearn = true ;
/* Update bmc protocol and hwmond_hostname_protocol file */
string power_state;
bmc_protocol_enum protocol;
bmcUtil_read_bmc_info ( host_ptr->hostname, power_state, protocol ) ;
if ( protocol != host_ptr->protocol)
{
ilog ("%s bmc protocol changed to %s",
host_ptr->hostname.c_str(),
bmcUtil_getProtocol_str(protocol).c_str());
}
host_ptr->protocol = protocol ;
bmcUtil_write_hwmond_protocol ( host_ptr->hostname, protocol ) ;
/* exit relearn request mode.
* allow the relearn operation to proceed */
host_ptr->relearn_request = false ;
host_ptr->relearn_done_date = future_time ( relearn_time );
ilog ("%s next relearn permitted after %s (%s)\n",
host_ptr->hostname.c_str(),
host_ptr->relearn_done_date.c_str(),
bmcUtil_getProtocol_str(host_ptr->protocol).c_str());
this->monitor_soon ( host_ptr );
this->monitor_soon ( host_ptr );
/* start the relearn timer */
mtcTimer_start ( host_ptr->relearnTimer,
hwmonTimer_handler,
relearn_time );
/* start the relearn timer */
mtcTimer_start ( host_ptr->relearnTimer,
hwmonTimer_handler,
relearn_time );
}
switch ( host_ptr->monitor_ctrl.stage )
@ -801,7 +755,6 @@ int hwmonHostClass::bmc_sensor_monitor ( struct hwmonHostClass::hwmon_host * hos
case HWMON_SENSOR_MONITOR__START:
{
mtcTimer_reset ( host_ptr->monitor_ctrl.timer );
if ( host_ptr->monitor )
{
/* Handle Audit Interval Change */
@ -880,7 +833,7 @@ int hwmonHostClass::bmc_sensor_monitor ( struct hwmonHostClass::hwmon_host * hos
host_ptr->bmc_thread_info.extra_info_ptr = (void*)&host_ptr->thread_extra_info ;
/* randomize the first audit a little so that over a swact we don't spike hwmond */
int r = (rand() % host_ptr->interval) + 1 ;
int r = (rand() % MTC_MINS_1) + 1 ;
/* poll all the sensors right away - between 1 and 10 seconds */
ilog ("%s sensor monitoring begins in %d seconds\n",
@ -935,6 +888,13 @@ int hwmonHostClass::bmc_sensor_monitor ( struct hwmonHostClass::hwmon_host * hos
host_ptr->hostname.c_str());
thread_kill ( host_ptr->bmc_thread_ctrl, host_ptr->bmc_thread_info );
mtcTimer_start ( host_ptr->monitor_ctrl.timer,
hwmonTimer_handler, MTC_MINS_1 );
_stage_change ( host_ptr->hostname,
host_ptr->monitor_ctrl.stage,
HWMON_SENSOR_MONITOR__RESTART );
}
/* check for 'thread done' completion */
@ -988,40 +948,33 @@ int hwmonHostClass::bmc_sensor_monitor ( struct hwmonHostClass::hwmon_host * hos
}
else if ( host_ptr->bmc_thread_info.data.find (BMC_POWER_ON_STATUS) == string::npos )
{
ilog ("%s %s\n", host_ptr->hostname.c_str(),
host_ptr->bmc_thread_info.data.c_str());
wlog ("%s power %s sensor learning delayed ; need power on\n",
host_ptr->hostname.c_str(),
host_ptr->bmc_thread_info.data.c_str());
host_ptr->poweron = false ;
wlog ("%s sensor model learning delayed ; need power on",
host_ptr->hostname.c_str());
}
else
{
/* OK, this is what we have been waiting for */
ilog ("%s power is on", host_ptr->hostname.c_str());
host_ptr->poweron = true ;
}
}
host_ptr->bmc_thread_ctrl.done = true ;
/* Start monitoring in 10 seconds */
int delay = MTC_SECS_10 ;
/* If power is off, retry in 2 minutes ; hold-off period */
if ( host_ptr->poweron == false )
{
mtcTimer_start ( host_ptr->monitor_ctrl.timer,
hwmonTimer_handler, MTC_MINS_1 );
delay = MTC_MINS_2 ;
_stage_change ( host_ptr->hostname,
host_ptr->monitor_ctrl.stage,
HWMON_SENSOR_MONITOR__RESTART );
}
else
{
mtcTimer_start ( host_ptr->monitor_ctrl.timer,
hwmonTimer_handler, MTC_MINS_2 );
mtcTimer_start ( host_ptr->monitor_ctrl.timer,
hwmonTimer_handler, delay );
_stage_change ( host_ptr->hostname,
host_ptr->monitor_ctrl.stage,
HWMON_SENSOR_MONITOR__RESTART );
}
_stage_change ( host_ptr->hostname,
host_ptr->monitor_ctrl.stage,
HWMON_SENSOR_MONITOR__RESTART );
}
break ;
}
@ -1929,7 +1882,6 @@ int hwmonHostClass::bmc_sensor_monitor ( struct hwmonHostClass::hwmon_host * hos
( host_ptr->sensors ) && ( host_ptr->groups ))
{
mtcTimer_reset ( host_ptr->relearnTimer );
host_ptr->relearn_done_date.clear();
host_ptr->relearn = false ;
plog ("%s sensor model relearn complete\n",
host_ptr->hostname.c_str());
@ -2036,11 +1988,6 @@ int hwmonHostClass::delete_handler ( struct hwmonHostClass::hwmon_host * host_pt
ilog ("%s Delete Operation Started\n", host_ptr->hostname.c_str());
host_ptr->retries = 0 ;
if ( host_ptr->bm_provisioned == true )
{
set_bm_prov ( host_ptr, false);
}
if ( host_ptr->bmc_thread_ctrl.stage != THREAD_STAGE__IDLE )
{
int delay = THREAD_POST_KILL_WAIT ;
@ -2545,7 +2492,7 @@ void hwmonHostClass::monitor_now ( struct hwmonHostClass::hwmon_host * host_ptr
*
* Name : monitor_soon
*
* Description: Force monitor to occur in 30 seconds.
* Description: Force monitor to occur in 5 seconds.
*
****************************************************************************/

7
mtce/src/hwmon/hwmonJson.cpp
View File

@ -75,8 +75,6 @@ int hwmonJson_load_inv ( char * json_str_ptr, node_inv_type & info )
node_inv_init ( info );
/* Get all required fields */
//info.mac = _get_key_value_string ( node_obj, MTC_JSON_INV_HOSTMAC);
//info.ip = _get_key_value_string ( node_obj, MTC_JSON_INV_HOSTIP );
info.name = _get_key_value_string ( node_obj, MTC_JSON_INV_NAME );
cluster_host_ip = _get_key_value_string ( node_obj, MTC_JSON_INV_CLSTRIP );
@ -85,11 +83,12 @@ int hwmonJson_load_inv ( char * json_str_ptr, node_inv_type & info )
dlog ("%s inventory has cluster_host_ip=%s\n", info.name.c_str(), cluster_host_ip.c_str());
info.clstr_ip = cluster_host_ip;
}
info.type = _get_key_value_string ( node_obj, MTC_JSON_INV_TYPE );
// info.type = _get_key_value_string ( node_obj, MTC_JSON_INV_TYPE );
info.uuid = _get_key_value_string ( node_obj, MTC_JSON_INV_UUID );
info.bm_ip = _get_key_value_string ( node_obj, MTC_JSON_INV_BMIP );
info.bm_un = _get_key_value_string ( node_obj, MTC_JSON_INV_BMUN );
info.bm_type = _get_key_value_string ( node_obj, MTC_JSON_INV_BMTYPE);
info.bm_proto= _get_key_value_string ( node_obj, MTCE_INFO_KEY__BMC_PROTOCOL);
info.bm_http = _get_key_value_string ( node_obj, MTC_JSON_INV_BMHTTP);
/* print the parsed info if debug level is 3 - mlog2 */
if ( daemon_get_cfg_ptr()->debug_msg == DEBUG_LEVEL3 )

7
mtce/src/hwmon/hwmonModel.cpp
View File

@ -58,9 +58,10 @@
int hwmonHostClass::bmc_create_sensor_model ( struct hwmonHostClass::hwmon_host * host_ptr )
{
int rc = PASS ;
ilog ("%s creating sensor model using %s\n",
ilog ("%s creating sensor model using %s:%s\n",
host_ptr->hostname.c_str(),
bmcUtil_getProtocol_str(host_ptr->protocol).c_str());
bmcUtil_getProtocol_str(host_ptr->protocol).c_str(),
host_ptr->bm_ip.c_str());
host_ptr->groups = 0 ;
@ -300,7 +301,7 @@ int hwmonHostClass::bmc_delete_sensor_model ( struct hwmonHostClass::hwmon_host
host_ptr->hostname.c_str());
this->clear_bm_assertions ( host_ptr );
ilog ("%s ... deleting sensor model\n",
blog ("%s ... deleting sensor model\n",
host_ptr->hostname.c_str());
}

24
mtce/src/hwmon/hwmonMsg.cpp
View File

@ -249,19 +249,16 @@ int hwmon_service_inbox ( void )
print_mtc_message ( inv.name, MTC_CMD_RX, msg, get_iface_name_str(MGMNT_IFACE) , false);
rc = PASS;
if ( msg.cmd == MTC_CMD_ADD_HOST )
if (( msg.cmd == MTC_CMD_ADD_HOST ) || ( msg.cmd == MTC_CMD_MOD_HOST ))
{
mlog ("%s %s host message\n", inv.name.c_str(), get_event_str(msg.cmd).c_str());
/* If the add returns a RETRY that means this host was already
* provisioned so turn around and run the modify */
if ( get_hwmonHostClass_ptr()->add_host ( inv ) == RETRY )
{
mlog ("%s modify host (from add ) message\n", inv.name.c_str());
get_hwmonHostClass_ptr()->mod_host ( inv );
}
else
{
mlog ("%s add host message\n", inv.name.c_str());
}
}
else if ( msg.cmd == MTC_CMD_DEL_HOST )
{
@ -279,21 +276,6 @@ int hwmon_service_inbox ( void )
mlog ("%s stop monitoring message\n", inv.name.c_str());
get_hwmonHostClass_ptr()->mon_host ( inv.name , false );
}
else if ( msg.cmd == MTC_CMD_MOD_HOST )
{
/* If the add returns a RETRY that means this host was already
* provisioned so turn around and run the modify otherwise
* default the modify to be an add */
if ( get_hwmonHostClass_ptr()->add_host ( inv ) == RETRY )
{
mlog ("%s modify host message\n", inv.name.c_str());
get_hwmonHostClass_ptr()->mod_host ( inv );
}
else
{
mlog ("%s add host (from modify) message\n", inv.name.c_str());
}
}
else if ( msg.cmd == MTC_CMD_QRY_HOST )
{
mlog ("%s query host message - NOT IMPLEMENTED YET !!!\n", inv.name.c_str());

138
mtce/src/hwmon/hwmonThreads.cpp
View File

@ -810,7 +810,8 @@ static void _set_default_unit_type_for_sensor( thread_info_type * info_ptr,
{
strcpy( _sample_list[samples].unit, BMC_SENSOR_DEFAULT_UNIT_TYPE_TEMP);
}
else if ( label == REDFISH_SENSOR_LABEL_POWER_CTRL )
else if (( label == REDFISH_SENSOR_LABEL_POWER_CTRL ) ||
( label == REDFISH_SENSOR_LABEL_POWER_SUPPLY ))
{
strcpy( _sample_list[samples].unit, BMC_SENSOR_DEFAULT_UNIT_TYPE_POWER);
}
@ -849,17 +850,36 @@ static int _parse_redfish_sensor_data( char * json_str_ptr, thread_info_type * i
string label, const char * reading_label, int & samples )
{
int rc = PASS ;
struct json_object *json_obj = NULL;
/*************************************************************************
*
* Gracefully handle a missing sensor group label.
* Return failure, that is ignored, if its not there.
*
* Calling jsonUtil_get_list directly results in noisy json error logs.
* If a server is not providing a canned group then so be it.
*
*************************************************************************
*
* Start by objectifying the output data followed by getting that
* sensor group key value */
struct json_object *json_obj = json_tokener_parse(json_str_ptr);
if ( !json_obj )
return (FAIL_JSON_PARSE);
string value = jsonUtil_get_key_value_string ( json_obj, label.data());
json_object_put(json_obj);
if (( value.empty() || value == NONE ))
return (FAIL_NO_DATA);
std::list<string> sensor_list ;
std::list<string>::iterator iter_curr_ptr ;
string status_str;
string temp_str;
sensor_list.clear();
rc = jsonUtil_get_list(json_str_ptr, label, sensor_list);
if ( rc == PASS )
{
string status_str;
string temp_str;
std::list<string>::iterator iter_curr_ptr ;
for ( iter_curr_ptr = sensor_list.begin();
iter_curr_ptr != sensor_list.end() ;
++iter_curr_ptr )
@ -870,17 +890,49 @@ static int _parse_redfish_sensor_data( char * json_str_ptr, thread_info_type * i
elog_t ("%s no or invalid sensor record\n", info_ptr->hostname.c_str());
return (FAIL_JSON_PARSE);
}
/* parse value from json string according to key, if value is none, return na
Put the value to _sample_list */
/* Name : GET_SENSOR_DATA_VALUE
*
* Purpose: Parse value from json string according to key.
*
* If value is none, return na.
* Put the value to _sample_list.
*
* Start with just 'Name' and 'Reading'
*/
GET_SENSOR_DATA_VALUE( temp_str, json_obj, "Name", name )
GET_SENSOR_DATA_VALUE( temp_str, json_obj, reading_label, value )
GET_SENSOR_DATA_VALUE( temp_str, json_obj, "ReadingUnits", unit )
/* Abort on this sensor if the sensor name is missing.
* A missing name is parsed as 'na' by GET_SENSOR_DATA_VALUE macro.
*
* This check was added after seeing a missing 'Name' and 'Status'
* on one of the integration servers this feature was tested against.
* Without this check the code will create a sensor with name = na */
if ( !strcmp(_sample_list[samples].name, "na") )
{
/* Another special case handling
*
* Its a Fan sensor if ReadingUnits is RPM */
if ( strcmp(_sample_list[samples].unit, "RPM"))
return (FAIL_NOT_FOUND);
/* So it might be a Fan sensor. Still need a sensor name.
* Some Dell servers that publish the fan sensor name key
* as 'FanName' rather than 'Name' like all other sensors. */
GET_SENSOR_DATA_VALUE( temp_str, json_obj, "FanName", name )
if ( !strcmp(_sample_list[samples].name,"na") )
return (FAIL_NOT_FOUND);
strcpy( _sample_list[samples].unit, BMC_SENSOR_DEFAULT_UNIT_TYPE_FANS);
}
GET_SENSOR_DATA_VALUE( temp_str, json_obj, "LowerThresholdNonRecoverable", lnr )
GET_SENSOR_DATA_VALUE( temp_str, json_obj, "LowerThresholdCritical", lcr )
GET_SENSOR_DATA_VALUE( temp_str, json_obj, "LowerThresholdNonCritical", lnc )
GET_SENSOR_DATA_VALUE( temp_str, json_obj, "UpperThresholdNonCritical", unc )
GET_SENSOR_DATA_VALUE( temp_str, json_obj, "UpperThresholdCritical", ucr )
GET_SENSOR_DATA_VALUE( temp_str, json_obj, "UpperThresholdNonRecoverable", unr )
GET_SENSOR_DATA_VALUE( temp_str, json_obj, "ReadingUnits", unit )
/* Set default unit type if can not get unit type from json string */
if ( !strcmp(_sample_list[samples].unit, "na") )
@ -1248,20 +1300,66 @@ void * hwmonThread_redfish ( void * arg )
}
case BMC_THREAD_CMD__POWER_STATUS:
{
string power_status = "" ;
bmc_protocol_enum protocol ;
blog2_t ("%s query power status info\n", info_ptr->log_prefix);
bmcUtil_read_bmc_info (info_ptr->hostname, power_status, protocol);
if (power_status.find (BMC_POWER_ON_STATUS) == string::npos)
blog2_t ("%s read power state\n", info_ptr->log_prefix);
if ( _redfishUtil_send_request( info_ptr, datafile,
BMC_POWER_STATUS_FILE_SUFFIX,
REDFISHTOOL_BMC_INFO_CMD ) != PASS )
{
info_ptr->data = BMC_POWER_OFF_STATUS ;
info_ptr->status_string = "failed to send request" ;
info_ptr->status = FAIL_OPERATION ;
goto redfishtool_thread_done ;
}
else
/* look for the output data file */
if( _wait_for_command_output(info_ptr, datafile) )
{
info_ptr->data = BMC_POWER_ON_STATUS ;
/* need to add one of the following 2 strings
* to info_ptr->data
* - Chassis Power is on
* - Chassis Power is off
*/
if ( datafile.empty() )
{
info_ptr->status_string = "bmc info filename empty" ;
info_ptr->status = FAIL_NO_DATA ;
goto redfishtool_thread_done ;
}
/* read the output data */
string json_bmc_info = daemon_read_file (datafile.data());
if ( json_bmc_info.empty() )
{
info_ptr->status_string = "bmc info file empty" ;
info_ptr->status = FAIL_STRING_EMPTY ;
goto redfishtool_thread_done ;
}
/* parse the output data */
struct json_object *json_obj =
json_tokener_parse((char*)json_bmc_info.data());
if ( !json_obj )
{
info_ptr->status_string = "bmc info data parse error" ;
info_ptr->status = FAIL_JSON_PARSE ;
goto redfishtool_thread_done ;
}
/* load the power state */
string power_state = tolowercase(
jsonUtil_get_key_value_string( json_obj, REDFISH_LABEL__POWER_STATE));
if ( power_state == "on" )
{
info_ptr->data = "Chassis Power is on" ;
}
else
{
info_ptr->data = "Chassis Power is off" ;
}
info_ptr->status_string = "pass" ;
info_ptr->status = PASS ;
ilog_t ("%s %s", info_ptr->hostname.c_str(),
info_ptr->data.c_str());
json_object_put( json_obj );
}
info_ptr->status_string = "pass" ;
info_ptr->status = PASS ;
break ;
}
default:

41
mtce/src/maintenance/mtcInvApi.cpp
View File

@ -543,6 +543,46 @@ int nodeLinkClass::mtcInvApi_update_value ( string hostname,
return (mtcInvApi_update_value ( node_ptr, key, value ));
}
/*****************************************************************************
*
* Name : mtcInvApi_update_mtcInfo
*
* Purpose : Update this host's mtce_info content in the sysinv database.
*
*****************************************************************************/
int nodeLinkClass::mtcInvApi_update_mtcInfo ( struct nodeLinkClass::node * node_ptr )
{
CHK_NODE_PTR(node_ptr);
int rc = mtcHttpUtil_event_init ( &node_ptr->httpReq,
node_ptr->hostname,
"mtcInvApi_update_mtce_info",
hostUtil_getServiceIp (SERVICE_SYSINV),
hostUtil_getServicePort(SERVICE_SYSINV));
if ( rc )
{
elog ("%s failed to allocate libEvent memory (%d)\n", node_ptr->hostname.c_str(), rc );
return (rc);
}
/* Set the host context */
node_ptr->httpReq.hostname = node_ptr->hostname ;
node_ptr->httpReq.uuid = node_ptr->uuid;
node_ptr->httpReq.request = SYSINV_UPDATE ;
node_ptr->httpReq.operation = SYSINV_OPER__UPDATE_VALUE ;
node_ptr->httpReq.max_retries = 3 ;
node_ptr->httpReq.cur_retries = 0 ;
node_ptr->httpReq.timeout = get_mtcInv_ptr()->sysinv_timeout ;
node_ptr->httpReq.payload = "[" ;
node_ptr->httpReq.payload.append ("{\"path\":\"/") ;
node_ptr->httpReq.payload.append (MTC_JSON_INV_MTCE_INFO);
node_ptr->httpReq.payload.append ("\",\"value\":\"");
node_ptr->httpReq.payload.append (node_ptr->mtce_info.data());
node_ptr->httpReq.payload.append ( "\",\"op\":\"replace\"}]");
hlog ("%s %s", node_ptr->hostname.c_str(), node_ptr->httpReq.payload.c_str());
return(this->workQueue_enqueue( node_ptr->httpReq));
}
/*****************************************************************************
*
* Name : mtcInvApi_update_uptime
@ -1479,6 +1519,7 @@ void nodeLinkClass::mtcInvApi_get_handler ( struct evhttp_request *req, void *ar
node.oper_subf = json_info.host[i].oper_subf ;
node.avail_subf = json_info.host[i].avail_subf ;
node.clstr_ip = json_info.host[i].clstr_ip ;
node.mtce_info = json_info.host[i].mtce_info ;
if (node.name.compare("none"))
{

39
mtce/src/maintenance/mtcNodeCtrl.cpp
View File

@ -432,15 +432,6 @@ static int mtc_ini_handler ( void * user,
}
}
}
else if (MATCH("agent", "bmc_access_method"))
{
string bmc_access_method_current = mtcInv.bmc_access_method ;
mtcInv.bmc_access_method = value ;
if ( mtcInv.bmc_access_method != bmc_access_method_current )
{
mtcInv.bmc_access_method_changed = true ;
}
}
return (PASS);
}
@ -682,8 +673,6 @@ int daemon_configure ( void )
ilog ("Controller : %s\n",
mtc_config.active ? "Active" : "In-Active" );
ilog ("BMC Access : %s", mtcInv.bmc_access_method.c_str());
/* remove any existing fit */
daemon_init_fit ();
@ -996,10 +985,6 @@ int daemon_init ( string iface, string nodetype )
return ( FAIL_DAEMON_CONFIG ) ;
}
/* bmc access method should not be considered changed if we
* are going through daemon_init ; i.e. process startup */
mtcInv.bmc_access_method_changed = false ;
return (rc);
}
@ -1095,10 +1080,10 @@ int _self_provision ( void )
mtcInv.set_bm_un ( my_identity.name, record_info.bm_un );
mtcInv.set_bm_ip ( my_identity.name, record_info.bm_ip );
mtcInv.set_bm_type ( my_identity.name, record_info.bm_type );
mtcInv.set_mtcInfo ( my_identity.name, record_info.mtce_info );
if ( my_identity.name == record_info.name )
{
/* If the active controller was 'locked' and is being auto-corrected
* to 'unlocked' then ensure that there is no locked alarm set for it */
if ( record_info.admin != "locked" )
@ -1220,23 +1205,6 @@ void nodeLinkClass::fsm ( void )
}
}
/* handle BMC access method change */
void nodeLinkClass::bmc_access_method_change_notifier ( void )
{
if ( head )
{
struct node * node_ptr ;
for ( node_ptr = head ;
node_ptr != NULL ;
node_ptr = node_ptr->next )
{
if ( node_ptr->bmc_provisioned )
node_ptr->bmc_access_method_changed = true ;
}
}
mtcInv.bmc_access_method_changed = false ;
}
void daemon_service_run ( void )
{
int rc ;
@ -1615,11 +1583,6 @@ void daemon_service_run ( void )
ilog ("DOR mode disable\n");
mtcInv.dor_mode_active = false ;
}
if ( mtcInv.bmc_access_method_changed == true )
{
mtcInv.bmc_access_method_change_notifier();
}
}
daemon_exit ();
}

184
mtce/src/maintenance/mtcNodeHdlrs.cpp
View File

@ -1302,13 +1302,6 @@ int nodeLinkClass::enable_handler ( struct nodeLinkClass::node * node_ptr )
else
{
mtcInvApi_update_task ( node_ptr, MTC_TASK_ENABLING );
/* Only run hardware monitor if board management is provisioned */
if ( node_ptr->bmc_provisioned == true )
{
send_hwmon_command ( node_ptr->hostname, MTC_CMD_START_HOST );
}
enableStageChange ( node_ptr, MTC_ENABLE__HOST_SERVICES_WAIT );
}
break ;
@ -2539,12 +2532,6 @@ int nodeLinkClass::recovery_handler ( struct nodeLinkClass::node * node_ptr )
MTC_AVAIL_STATUS__AVAILABLE );
}
/* Only run hardware monitor board management is provisioned */
if ( node_ptr->bmc_provisioned == true )
{
send_hwmon_command ( node_ptr->hostname, MTC_CMD_START_HOST );
}
/* Inform the VIM that this host is enabled */
mtcVimApi_state_change ( node_ptr, VIM_HOST_ENABLED, 3 );
@ -4715,7 +4702,7 @@ int nodeLinkClass::power_handler ( struct nodeLinkClass::node * node_ptr )
}
else
{
; // send_hwmon_command ( node_ptr->hostname, MTC_CMD_STOP_HOST );
;
}
node_ptr->power_action_retries = MTC_POWER_ACTION_RETRY_COUNT ;
@ -5079,8 +5066,6 @@ int nodeLinkClass::power_handler ( struct nodeLinkClass::node * node_ptr )
availStatusChange ( node_ptr, MTC_AVAIL_STATUS__OFFLINE );
// send_hwmon_command ( node_ptr->hostname, MTC_CMD_START_HOST );
powerStageChange ( node_ptr , MTC_POWER__DONE );
}
break ;
@ -5098,12 +5083,6 @@ int nodeLinkClass::power_handler ( struct nodeLinkClass::node * node_ptr )
ar_enable ( node_ptr );
/* tell the hardware monitor of the power state and protocol */
bmcUtil_hwmon_info ( node_ptr->hostname,
node_ptr->bmc_protocol,
node_ptr->power_on, "" );
send_hwmon_command ( node_ptr->hostname, MTC_CMD_MOD_HOST );
mtcInvApi_force_task ( node_ptr, "" );
break ;
}
@ -5148,14 +5127,6 @@ int nodeLinkClass::powercycle_handler ( struct nodeLinkClass::node * node_ptr )
node_ptr->hostname.c_str() ,
node_ptr->hwmon_powercycle.attempts );
/* Note: hwmon will continue to send powercycle requests to restart once it is accessible */
/* TODO: RELEASE NOTE: Node may be left in the disabled state
* - need to track power state and raise logs or alarms if host is stuck in power off state.
* - The bmc update does add tracking of the power state but does not introduce the alarm */
// send_hwmon_command ( node_ptr->hostname, MTC_CMD_START_HOST );
/* Let the next event perform anothe rpower-cycle retry */
adminActionChange ( node_ptr , MTC_ADMIN_ACTION__NONE );
powercycleStageChange ( node_ptr, MTC_POWERCYCLE__DONE );
@ -5225,7 +5196,6 @@ int nodeLinkClass::powercycle_handler ( struct nodeLinkClass::node * node_ptr )
node_ptr->hostname.c_str(),
node_ptr->hwmon_powercycle.attempts );
// send_hwmon_command ( node_ptr->hostname, MTC_CMD_STOP_HOST);
mtcInvApi_update_task ( node_ptr, MTC_TASK_POWERCYCLE_HOST, node_ptr->hwmon_powercycle.attempts );
node_ptr->hwmon_powercycle.retries = 0 ; /* remove for back to back power cycles */
@ -5695,12 +5665,6 @@ int nodeLinkClass::powercycle_handler ( struct nodeLinkClass::node * node_ptr )
recoveryStageChange ( node_ptr, MTC_RECOVERY__START); /* reset the fsm */
disableStageChange ( node_ptr, MTC_DISABLE__START); /* reset the fsm */
/* tell the hardware monitor of the power state and protocol */
bmcUtil_hwmon_info ( node_ptr->hostname,
node_ptr->bmc_protocol,
node_ptr->power_on, "" );
send_hwmon_command ( node_ptr->hostname, MTC_CMD_MOD_HOST );
plog ("%s Power-Cycle Completed (uptime:%d)\n", node_ptr->hostname.c_str(), node_ptr->uptime );
}
break ;
@ -5827,6 +5791,8 @@ int nodeLinkClass::add_handler ( struct nodeLinkClass::node * node_ptr )
node_ptr->uuid.length() ? node_ptr->uuid.c_str() : "" );
mtcInfo_log(node_ptr);
if (( CPE_SYSTEM ) && ( is_controller(node_ptr) == true ))
{
if ( daemon_is_file_present ( CONFIG_COMPLETE_WORKER ) == false )
@ -6088,11 +6054,6 @@ int nodeLinkClass::add_handler ( struct nodeLinkClass::node * node_ptr )
send_hbs_command ( node_ptr->hostname, MTC_CMD_ADD_HOST );
/* Add this host to other maintenance services */
if (( ! SIMPLEX_CPE_SYSTEM ) && ( node_ptr->bmc_provisioned ))
{
send_hwmon_command ( node_ptr->hostname, MTC_CMD_ADD_HOST );
}
if ( ( CPE_SYSTEM ) || ( is_worker (node_ptr) == true ))
{
send_guest_command ( node_ptr->hostname, MTC_CMD_ADD_HOST );
@ -6142,12 +6103,13 @@ int nodeLinkClass::add_handler ( struct nodeLinkClass::node * node_ptr )
}
}
}
/* Only run hardware monitor if the bm ip is provisioned */
if (( hostUtil_is_valid_bm_type ( node_ptr->bm_type )) &&
( hostUtil_is_valid_ip_addr ( node_ptr->bm_ip )))
( hostUtil_is_valid_ip_addr ( node_ptr->bm_ip )) &&
( hostUtil_is_valid_username ( node_ptr->bm_un )))
{
set_bm_prov ( node_ptr, true ) ;
send_hwmon_command ( node_ptr->hostname, MTC_CMD_START_HOST );
}
this->ctl_mtcAlive_gate(node_ptr, false) ;
@ -6210,38 +6172,10 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
( node_ptr->bm_ping_info.ok == true ) &&
( daemon_is_file_present ( MTC_CMD_FIT__JSON_LEAK_SOAK ) == true ))
{
node_ptr->bm_ping_info.stage = PINGUTIL_MONITOR_STAGE__FAIL ;
pingUtil_restart ( node_ptr->bm_ping_info );
}
#endif
/*****************************************************************
* Handle BMC access method changes
****************************************************************/
if ( node_ptr->bmc_access_method_changed )
{
node_ptr->bmc_access_method_changed = false ;
ilog ("%s bmc access method change ; force %s",
node_ptr->hostname.c_str(),
this->bmc_access_method.c_str());
thread_kill ( node_ptr->bmc_thread_ctrl, node_ptr->bmc_thread_info );
bmc_access_data_init ( node_ptr );
pingUtil_fini ( node_ptr->bm_ping_info );
node_ptr->bm_ping_info.stage = PINGUTIL_MONITOR_STAGE__OPEN ;
/* force re-fetch of the BMC password */
node_ptr->bm_pw.clear();
/* start a timer that will raise the BM Access alarm
* if we are not accessible by the time it expires */
mtcTimer_reset ( node_ptr->bm_timer );
mtcTimer_reset ( node_ptr->bmc_audit_timer );
mtcTimer_reset ( node_ptr->bmc_access_timer );
mtcTimer_start ( node_ptr->bmc_access_timer, mtcTimer_handler, MTC_MINS_2 );
}
/*****************************************************************
* Run the ping monitor if BMC provisioned and ip address is valid
*****************************************************************/
@ -6286,10 +6220,16 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
ilog ("%s bmc credentials received",
node_ptr->hostname.c_str());
}
else
{
ilog_throttled (node_ptr->bm_pw_wait_log_throttle, 10000,
"%s bmc handler is waiting on bmc password",
node_ptr->hostname.c_str());
}
}
if (( node_ptr->bmc_accessible == true ) &&
( node_ptr->bm_ping_info.ok == false ))
else if (( node_ptr->bmc_accessible == true ) &&
( node_ptr->bm_ping_info.ok == false ))
{
wlog ("%s bmc access lost\n", node_ptr->hostname.c_str());
@ -6312,7 +6252,7 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
bmc_access_data_init ( node_ptr );
node_ptr->bm_ping_info.stage = PINGUTIL_MONITOR_STAGE__FAIL ;
pingUtil_restart ( node_ptr->bm_ping_info );
/* start a timer that will raise the BM Access alarm
* if we are not accessible by the time it expires */
@ -6325,9 +6265,8 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
/* If the BMC protocol has not yet been learned then do so.
* Default is ipmi unless the target host responds to a
* redfish root query with a minimum version number ; 1.0 */
if (( node_ptr->bm_ping_info.ok == true ) &&
(!node_ptr->bm_pw.empty()) &&
( node_ptr->bmc_protocol_learned == false ))
else if (( node_ptr->bm_ping_info.ok == true ) &&
( node_ptr->bmc_protocol == BMC_PROTOCOL__DYNAMIC ))
{
if ( node_ptr->bmc_protocol_learning == false )
{
@ -6340,8 +6279,9 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
node_ptr->hostname.c_str(),
bmcUtil_getCmd_str(node_ptr->bmc_thread_info.command).c_str());
node_ptr->bmc_protocol_learning = false ;
node_ptr->bmc_protocol_learned = true ;
node_ptr->bmc_protocol = BMC_PROTOCOL__IPMITOOL ;
mtcInfo_set ( node_ptr, MTCE_INFO_KEY__BMC_PROTOCOL, BMC_PROTOCOL__IPMI_STR );
mtcInvApi_update_mtcInfo ( node_ptr );
}
else
{
@ -6385,8 +6325,9 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
node_ptr->bmc_thread_info.status_string.c_str());
}
node_ptr->bmc_protocol = BMC_PROTOCOL__IPMITOOL ;
mtcInfo_set ( node_ptr, MTCE_INFO_KEY__BMC_PROTOCOL, BMC_PROTOCOL__IPMI_STR );
mtcInvApi_update_mtcInfo ( node_ptr );
node_ptr->bmc_protocol_learning = false ;
node_ptr->bmc_protocol_learned = true ;
node_ptr->bmc_thread_ctrl.done = true ;
}
else
@ -6402,17 +6343,20 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
if ( redfishUtil_is_supported ( node_ptr->hostname,
node_ptr->bmc_thread_info.data) == true )
{
mtcInfo_set ( node_ptr, MTCE_INFO_KEY__BMC_PROTOCOL, BMC_PROTOCOL__REDFISH_STR );
node_ptr->bmc_protocol = BMC_PROTOCOL__REDFISHTOOL ;
}
else
{
mtcInfo_set ( node_ptr, MTCE_INFO_KEY__BMC_PROTOCOL, BMC_PROTOCOL__IPMI_STR );
node_ptr->bmc_protocol = BMC_PROTOCOL__IPMITOOL ;
}
node_ptr->bmc_protocol_learned = true ;
mtcInvApi_update_mtcInfo ( node_ptr );
blog ("%s bmc supports %s",
ilog ("%s bmc control using %s:%s",
node_ptr->hostname.c_str(),
bmcUtil_getProtocol_str(node_ptr->bmc_protocol).c_str());
bmcUtil_getProtocol_str(node_ptr->bmc_protocol).c_str(),
node_ptr->bm_ip.c_str());
node_ptr->bmc_thread_ctrl.done = true ;
}
@ -6430,7 +6374,6 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
( node_ptr->bmc_accessible == false ) &&
( node_ptr->bm_ping_info.ok == true ) &&
( node_ptr->bmc_info_query_done == false ) &&
( node_ptr->bmc_protocol_learned == true ) &&
( mtcTimer_expired (node_ptr->bm_timer ) == true ))
{
if ( node_ptr->bmc_info_query_active == false )
@ -6442,6 +6385,8 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
bmcUtil_getCmd_str(
node_ptr->bmc_thread_info.command).c_str());
node_ptr->bmc_protocol = BMC_PROTOCOL__IPMITOOL ;
mtcInfo_set ( node_ptr, MTCE_INFO_KEY__BMC_PROTOCOL, BMC_PROTOCOL__IPMI_STR );
mtcInvApi_update_mtcInfo ( node_ptr );
}
else
{
@ -6464,6 +6409,7 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
/* this error is reported by the receive driver */
node_ptr->bmc_info_query_active = false ;
node_ptr->bmc_thread_ctrl.done = true ;
mtcTimer_start ( node_ptr->bm_timer, mtcTimer_handler, MTC_BMC_REQUEST_DELAY );
}
else
{
@ -6475,7 +6421,7 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
node_ptr->bmc_thread_info.data,
node_ptr->bmc_info ) != PASS )
{
elog ("%s bmc %s failed ; defaulting to ipmitool",
elog ("%s bmc %s failed ; defaulting to ipmi",
node_ptr->hostname.c_str(),
bmcUtil_getCmd_str(
node_ptr->bmc_thread_info.command).c_str());
@ -6483,6 +6429,7 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
node_ptr->bmc_info_query_active = false ;
node_ptr->bmc_info_query_done = false ;
node_ptr->bmc_protocol = BMC_PROTOCOL__IPMITOOL ;
mtcInfo_set ( node_ptr, MTCE_INFO_KEY__BMC_PROTOCOL, BMC_PROTOCOL__IPMI_STR );
}
else
{
@ -6496,7 +6443,8 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
node_ptr->bmc_info_query_done = true ;
node_ptr->bmc_info_query_active = false ;
node_ptr->bmc_protocol_learning = false ;
node_ptr->bmc_protocol_learned = true ;
mtcInfo_set ( node_ptr, MTCE_INFO_KEY__BMC_PROTOCOL, BMC_PROTOCOL__REDFISH_STR );
mtcTimer_reset ( node_ptr->bmc_access_timer );
@ -6506,16 +6454,13 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
plog ("%s bmc is accessible using redfish",
node_ptr->hostname.c_str());
/* tell the hardware monitor of the power state and protocol */
bmcUtil_hwmon_info ( node_ptr->hostname,
node_ptr->bmc_protocol,
node_ptr->power_on, "" );
send_hwmon_command ( node_ptr->hostname, MTC_CMD_MOD_HOST );
node_ptr->bmc_thread_ctrl.done = true ;
node_ptr->bmc_thread_info.command = 0 ;
}
mtcInvApi_update_mtcInfo ( node_ptr );
send_hwmon_command ( node_ptr->hostname, MTC_CMD_ADD_HOST );
send_hwmon_command ( node_ptr->hostname, MTC_CMD_START_HOST );
}
}
}
@ -6572,7 +6517,7 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
/* this error is reported by the bmc receive driver */
node_ptr->bmc_info_query_active = false ;
node_ptr->bmc_thread_ctrl.done = true ;
mtcTimer_start ( node_ptr->bm_timer, mtcTimer_handler, MTC_POWER_ACTION_RETRY_DELAY );
mtcTimer_start ( node_ptr->bm_timer, mtcTimer_handler, MTC_BMC_REQUEST_DELAY );
}
else
{
@ -6621,7 +6566,7 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
bmcUtil_getCmd_str(
node_ptr->bmc_thread_info.command).c_str());
node_ptr->reset_cause_query_active = false ;
mtcTimer_start ( node_ptr->bm_timer, mtcTimer_handler, MTC_POWER_ACTION_RETRY_DELAY );
mtcTimer_start ( node_ptr->bm_timer, mtcTimer_handler, MTC_BMC_REQUEST_DELAY );
node_ptr->bmc_thread_ctrl.done = true ;
}
else
@ -6668,7 +6613,7 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
{
node_ptr->power_status_query_active = false ;
node_ptr->bmc_thread_ctrl.done = true ;
mtcTimer_start ( node_ptr->bm_timer, mtcTimer_handler, MTC_POWER_ACTION_RETRY_DELAY );
mtcTimer_start ( node_ptr->bm_timer, mtcTimer_handler, MTC_BMC_REQUEST_DELAY );
}
else
{
@ -6683,7 +6628,7 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
ilog ("%s %s\n", node_ptr->hostname.c_str(),
node_ptr->bmc_thread_info.data.c_str());
plog ("%s bmc is accessible\n", node_ptr->hostname.c_str());
plog ("%s bmc is accessible using ipmi\n", node_ptr->hostname.c_str());
/* set host power state ; on or off */
if ( node_ptr->bmc_thread_info.data.find (IPMITOOL_POWER_ON_STATUS) != std::string::npos )
@ -6703,7 +6648,8 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
}
} /* end power off detection handling */
bmcUtil_hwmon_info ( node_ptr->hostname, node_ptr->bmc_protocol, node_ptr->power_on, "" );
send_hwmon_command ( node_ptr->hostname, MTC_CMD_ADD_HOST );
send_hwmon_command ( node_ptr->hostname, MTC_CMD_START_HOST );
} /* end query handling success path */
} /* end power status query handling */
@ -6713,12 +6659,11 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
/* BMC Access Audit for Redfish.
* - used to refresh the host power state */
if (( node_ptr->bmc_protocol == BMC_PROTOCOL__REDFISHTOOL ) &&
( node_ptr->bmc_provisioned ) &&
( node_ptr->bmc_accessible ) &&
(!node_ptr->bm_pw.empty() ) &&
( mtcTimer_expired ( node_ptr->bmc_audit_timer ) == true ) &&
( mtcTimer_expired ( node_ptr->bm_timer ) == true ))
else if (( node_ptr->bmc_protocol == BMC_PROTOCOL__REDFISHTOOL ) &&
( node_ptr->bmc_provisioned ) &&
( node_ptr->bmc_accessible ) &&
( mtcTimer_expired ( node_ptr->bmc_audit_timer ) == true ) &&
( mtcTimer_expired ( node_ptr->bm_timer ) == true ))
{
if ( node_ptr->bmc_thread_ctrl.done )
{
@ -6729,8 +6674,7 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
node_ptr->hostname.c_str(),
bmcUtil_getCmd_str(
node_ptr->bmc_thread_info.command).c_str());
node_ptr->bm_ping_info.ok = false ;
node_ptr->bm_ping_info.stage = PINGUTIL_MONITOR_STAGE__FAIL ;
pingUtil_restart ( node_ptr->bm_ping_info );
}
else
{
@ -6751,15 +6695,13 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
{
wlog ("%s bmc audit failed receive (rc:%d)",
node_ptr->hostname.c_str(), rc );
node_ptr->bm_ping_info.ok = false ;
node_ptr->bm_ping_info.stage = PINGUTIL_MONITOR_STAGE__FAIL ;
pingUtil_restart ( node_ptr->bm_ping_info );
}
else if ( node_ptr->bmc_thread_info.data.empty())
{
wlog ("%s bmc audit failed get bmc query response data",
node_ptr->hostname.c_str());
node_ptr->bm_ping_info.ok = false ;
node_ptr->bm_ping_info.stage = PINGUTIL_MONITOR_STAGE__FAIL ;
pingUtil_restart ( node_ptr->bm_ping_info );
}
else
{
@ -6780,8 +6722,7 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
{
wlog ("%s bmc audit failed to get power state",
node_ptr->hostname.c_str());
node_ptr->bm_ping_info.ok = false ;
node_ptr->bm_ping_info.stage = PINGUTIL_MONITOR_STAGE__FAIL ;
pingUtil_restart ( node_ptr->bm_ping_info );
rc = FAIL_JSON_PARSE ;
}
if ( rc == PASS )
@ -6791,12 +6732,6 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
ilog ("%s power state changed to %s",
node_ptr->hostname.c_str(),
power_state.c_str());
/* tell the hardware monitor of the power state and protocol */
bmcUtil_hwmon_info ( node_ptr->hostname,
node_ptr->bmc_protocol,
power_on, "" );
send_hwmon_command ( node_ptr->hostname, MTC_CMD_MOD_HOST );
}
node_ptr->power_on = power_on ;
blog1 ("%s bmc audit timer re-started (%d secs)\n",
@ -6809,8 +6744,7 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
}
else
{
node_ptr->bm_ping_info.ok = false ;
node_ptr->bm_ping_info.stage = PINGUTIL_MONITOR_STAGE__FAIL ;
pingUtil_restart ( node_ptr->bm_ping_info );
wlog ("%s bmc audit failed parse bmc query response",
node_ptr->hostname.c_str());
}
@ -6824,9 +6758,7 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
if (( node_ptr->bmc_accessible == false ) &&
( mtcTimer_expired ( node_ptr->bmc_access_timer ) == true ))
{
node_ptr->bm_ping_info.ok = false ;
node_ptr->bm_ping_info.stage = PINGUTIL_MONITOR_STAGE__FAIL ;
pingUtil_restart ( node_ptr->bm_ping_info );
/* start a timer that will raise the BM Access alarm
* if we are not accessible by the time it expires */
@ -6853,7 +6785,7 @@ int nodeLinkClass::bmc_handler ( struct nodeLinkClass::node * node_ptr )
( node_ptr->reset_cause_query_done == true ) &&
( node_ptr->power_status_query_done == true )) ||
(( node_ptr->bmc_protocol == BMC_PROTOCOL__REDFISHTOOL ) &&
( node_ptr->bmc_protocol_learned == true ))))
( node_ptr->bmc_protocol_learning == false ))))
{
mtcAlarm_clear ( node_ptr->hostname, MTC_ALARM_ID__BM );
node_ptr->alarms[MTC_ALARM_ID__BM] = FM_ALARM_SEVERITY_CLEAR ;
@ -7066,7 +6998,7 @@ int nodeLinkClass::insv_test_handler ( struct nodeLinkClass::node * node_ptr )
if ( node_ptr->bm_ping_info.ok )
{
ilog ("%s FIT failing bmc ping monitor", node_ptr->hostname.c_str());
node_ptr->bm_ping_info.stage = PINGUTIL_MONITOR_STAGE__FAIL ;
pingUtil_restart ( node_ptr->bm_ping_info );
}
}

1
mtce/src/maintenance/mtcSubfHdlrs.cpp
View File

@ -113,6 +113,7 @@ int nodeLinkClass::enable_subf_handler ( struct nodeLinkClass::node * node_ptr )
plog ("%s Subf Configured OK\n", name.c_str());
enableStageChange ( node_ptr, MTC_ENABLE__GOENABLED_TIMER );
alarm_config_clear ( node_ptr );
break ;
}
if ((( !node_ptr->mtce_flags & MTC_FLAG__I_AM_CONFIGURED )) ||

6
mtce/src/maintenance/mtcThreads.cpp
View File

@ -322,7 +322,7 @@ void * mtcThread_bmc ( void * arg )
{
/* Log the command that failed unless ...
* - its the root query during learning
* - its not the typical falure to reach the BMC whose
* - its not the typical failure to reach the BMC whose
* error shows up as a ENOENT or
* 'No such file or directory'
*/
@ -337,8 +337,10 @@ void * mtcThread_bmc ( void * arg )
info_ptr->status = FAIL_SYSTEM_CALL ;
if ( daemon_is_file_present ( datafile.data() ))
{
/* load in the error. stdio is redirected to the datafile */
/* stdio is redirected to the datafile.
* load in the error and remove the file. */
info_ptr->status_string = daemon_read_file(datafile.data());
daemon_remove_file(datafile.data());
}
}
}