From 3b5d1182d4ae294649925c6fe83b27eb1903faf8 Mon Sep 17 00:00:00 2001 From: Eric MacDonald Date: Tue, 25 Jun 2024 14:21:29 +0000 Subject: [PATCH] Add support for any local LDAP user to run collect This update replaces the currently enforced sysadmin username, allowing any local LDAP user, that is part of the sudo and sys_protected groups, to run collect. This change introduced several challenging new failure modes that necessitated some refactoring of collect's existing fault reporting. Enhancements were made to the detection, handling and reporting of failures at 'all levels' ; i.e. local host, remote host, subcloud and subcloud remote host. Specific attention was put towards handling of most probable failure modes that include detection and handling of passwordless sudo, unsupported sudo, unknown usernames, unreachable hosts, invalid passwords, out of space errors, etc. 'at all levels' Generally, multi host collect handling continues in the presence of remote host collect errors or warnings. Whereas local host failures are typically treated as fatal. Additionally, reporting of collect timeout cases was improved by instead of printing only a timeout code number, a string based timeout cause is now included. Improvements were also made in the handling and packaging of the collect.log file. A collect.log is now included at the main bundle and subcloud collect levels. Some failures to collect remote hosts are now logged in these files at the appropriate level. If a user notices that a host or subcloud is missing from a bundle, say because it was unreachable or had sudo-less password enabled, the collect log will have a warning message to that effect. A 5 second yield was added to subcloud collect monitoring to reduce the cpu load subcloud collect monitoring was inducing. This lines up with the existing 5 second yield done for host collect monitoring. In attempt to improve the collect user experience the following additional improvements were made. - Attempt to source the openrc file and query system inventory is moved after the password prompt. This allows the various password checks to be handled early making the tool feel more responsive. - The global collect timeout now starts only after password is input and inventory is read so these operations don't contribute towards the collection time. - Improves how collect reports to the console and logs the hosts and or subclouds that 'will be' and 'were successfully' collected from. - Added expect segment debug logging tied to the --debug option. With debug enabled each function's expect segment logs its execution output to /tmp with files of the following form. /tmp/collect_expect___ - Added a --password option to simplify collect test automation. - Replaced the subcloud collect verbose option with debug. The verbose subcloud collect was known to cause issues. All the above changes warranted a collect tool up-version to 3.0 Test Plan: A full collect regression was performed PASS: Verify install and collect testing on the following systems - All-In-One SX - All-In-One DX - Standard DX with 1 worker and 1 storage - Simplex DC system with 2 subclouds ; 1 SX and 1 DX Success Path Handling: both sysadmin and any other username PASS: Verify collect handling at all levels PASS: Verify dated collect all for system and subcloud PASS: Verify all variations of collect host list handling PASS: Verify collect clean at all levels PASS: Verify system and subcloud collect --report handling PASS: Verify collect all --skip-mask PASS: Verify collect all --timeout PASS: Verify collect all --inline PASS: Verify collect all --subcloud PASS: Verify collect all --verbose PASS: Verify collect all --verbose --debug PASS: Verify collect all --version PASS: Verify collect all --subcloud --inline PASS: Verify collect all on SX/DX standard and DC systems PASS: verify new collect --password option PASS: Verify collect bundle content between sysadmin and other user. PASS: Verify bundle includes collect.log at bundle and host levels PASS: Verify collect.log content at each level. PASS: Verify collect from remote host that does not have this update PASS: Verify collect from subcloud that does not have this update PASS: Verify system and subcloud collect using account password with special character(s). Failure Path Handling: error response should clearly indicate the issue PASS: Verify all level collect handling of unknown username PASS: Verify all level collect handling with passwordless sudo enabled PASS: Verify all level collect handling of unsupported sudo PASS: Verify all level collect handling where hosts run out of scratch PASS: Verify all level collect handling of a host whose scratch space is filled to 75% or more PASS: Verify a successful collect following the cleanup of a previous out of space error. PASS: Verify collect handling of all non-active controller cases PASS: Verify collect handling of an invalid hostname PASS: Verify collect handling of unreachable remote hosts at all levels PASS: Verify collect handling of an invalid password at all levels PASS: Verify collect host and subcloud collect timeout handling PASS: Verify collect global timeout handling PASS: Verify collect handling of failure to get the remote tarball PASS: Verify collect debug option handling and data Story: 2010533 Task: 50419 Change-Id: Ibd827e1c72190bcdcf710b32ad7903cfa397c394 Signed-off-by: Eric MacDonald --- tools/collector/debian-scripts/collect | 1028 ++++++++++++----- .../collect_containerization.sh | 8 +- tools/collector/debian-scripts/collect_host | 25 +- tools/collector/debian-scripts/collect_utils | 124 +- 4 files changed, 831 insertions(+), 354 deletions(-) diff --git a/tools/collector/debian-scripts/collect b/tools/collector/debian-scripts/collect index df69f856..df8aee4b 100644 --- a/tools/collector/debian-scripts/collect +++ b/tools/collector/debian-scripts/collect @@ -1,7 +1,7 @@ #! /bin/bash ######################################################################## # -# Copyright (c) 2014-2023 Wind River Systems, Inc. +# Copyright (c) 2014-2024 Wind River Systems, Inc. # # SPDX-License-Identifier: Apache-2.0 # @@ -15,10 +15,11 @@ # expect scripts and collection commands, some that that require sudo # priviledge. # -# The collect tool can be run from any host to collect data for that host. +# The collect tool can be run by any sudo authorized ldap user from +# any host to collect data from that host or run from the active +# controller to collect data from its managed hosts or subclouds. # -# The collect tool must be run from an active controller to collect data -# from its managed hosts or subclouds. +# Note: Collect does NOT support 'passwordless' or 'NOPASSWD' sudo. # # Version 2.2 introduces the following behavioral changes. # @@ -198,11 +199,11 @@ TOOL_NAME="collect" -TOOL_VER=2 -TOOL_REV=2 +TOOL_VER=3 +TOOL_REV=0 # only supported username -UN="sysadmin" +UN=$(whoami) pw="" # pull in common utils and environment @@ -223,12 +224,8 @@ function collect_exit() if [ ${UID} -eq 0 ]; then elog "Cannot run collect as 'root' user" collect_exit -elif [ "${USER}" != "${UN}" ]; then - elog "Can only run collect as '${UN}' user" - collect_exit fi - -source_openrc_if_needed +ACTIVE=false # used to hold the name of the password file used to pass # the sudo password to a subcloud @@ -292,8 +289,13 @@ SCP_TIMEOUT="${SCP_TIMEOUT_DEFAULT}" SSH_CMD="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PreferredAuthentications=password -o PubkeyAuthentication=no" SSH_TIMEOUT=${SSH_TIMEOUT_DEFAULT} +# Get now date in the date format for the bundle file and dir names NOWDATE=$(date +"%Y%m%d.%H%M%S") +# Get now date in the format for filtering date from log files. +# Used for filtering the collect logs from user.log. +LOGDATE=$(date +"%Y-%m-%dT%H:%M:%S.%3N") + COLLECT_BASE_DIR="/scratch" collect_host="/usr/local/sbin/collect_host" collect="/usr/local/sbin/collect" @@ -305,10 +307,6 @@ ORCHESTRATED_COLLECT=false CURR_DIR=$(pwd) -# common permission error strings -pw_error="orry, try again" -ac_error="ermission denied" - function print_help() { echo "" @@ -319,6 +317,12 @@ function print_help() echo "StarlingX 'collect' is used to gather system logs, configuration" echo "and state data for off system analysis." echo "" + echo "Collect can be run by any sudo authorized ldap user from any" + echo "host to collect data from that host or run from the active" + echo "controller to collect data from its managed hosts or subclouds." + echo "" + echo "Note: Collect does NOT support 'passwordless' or 'NOPASSWD' sudo." + echo "" echo "Running collect will collect logs to /scratch/" echo "on the host collect is run from. Use host names to specify which" echo "hosts or subclouds to collect from." @@ -435,6 +439,7 @@ declare -a HOSTLIST=(${HOSTNAME}) declare -i SUBCLOUDS=0 declare -a SUBCLOUDLIST=() declare -i DONE_COUNT=0 +declare -i COLLECTED_COUNT=0 declare -i longest_name=0 PLEASE_STANDBY=false @@ -470,6 +475,12 @@ function report_error() { local string=${1} local code=${2} + local cause="" + local extra="" + local error_log=false # default to warning + local want_exit=false + local want_newline=false + if [[ "${PARALLEL_COLLECT_MODE}" = true && "${PLEASE_STANDBY}" = true && ${DONE_COUNT} -eq 0 ]] ; then DONE_COUNT=$((DONE_COUNT+1)) @@ -479,68 +490,149 @@ function report_error() fi if [ ${code} -eq ${FAIL_PASSWORD} ] ; then - elog "Invalid password" - collect_exit ${code} + cause="${FAIL_INVALID_PASSWORD_STR}" + want_exit=true + error_log=true elif [ ${code} -eq ${FAIL_CONTINUE} ] ; then - elog "${FAIL_CONTINUE_STR} ; ${string} (reason:${code})" - collect_exit ${code} + cause="${FAIL_CONTINUE_STR}" + want_exit=true + error_log=true elif [ ${code} -eq ${FAIL_INACTIVE} ] ; then - elog "${FAIL_INACTIVE_STR} ; ${string} (reason:${code})" - collect_exit ${code} + cause="${FAIL_INACTIVE_STR}" + want_exit=true + error_log=true elif [ ${code} -eq ${FAIL_PERMISSION} ] ; then - elog "Permission error ; exiting (${string})" + cause="${FAIL_PERMISSION_STR}" + error_log=true elif [ ${code} -eq ${FAIL_UNREACHABLE} ] ; then - wlog "${string} (reason:${code}:unreachable)" + cause="${FAIL_UNREACHABLE_STR}" - elif [ ${code} -eq ${FAIL_PERMISSION_SKIP} ] ; then - elog "${string} (reason:${code}:permission error)" + elif [ ${code} -eq ${FAIL_UNREACHABLE} ] ; then + cause="${FAIL_UNREACHABLE_STR}" + + elif [ ${code} -eq ${FAIL_PERMISSION_REMOTE} ] ; then + cause="${FAIL_PERMISSION_REMOTE_STR}" elif [ ${code} -eq ${FAIL_OUT_OF_SPACE} ] ; then - elog "${string} (reason:${code}:${FAIL_NOT_ENOUGH_SPACE_STR}) ; need to increase available space ${COLLECT_BASE_DIR}" + cause="${FAIL_NOT_ENOUGH_SPACE_STR}" + error_log=true + extra=" ; need to increase available ${COLLECT_BASE_DIR} space" + + elif [ ${code} -eq ${FAIL_OUT_OF_SPACE_REMOTE} ] ; then + cause="${FAIL_OUT_OF_SPACE_REMOTE_STR}" + extra=" ; need to increase available ${COLLECT_BASE_DIR} space" + + elif [ ${code} -eq ${FAIL_NOT_ENOUGH_SPACE_REMOTE} ] ; then + cause="${FAIL_NOT_ENOUGH_SPACE_REMOTE_STR}" + extra=" ; need to increase available ${COLLECT_BASE_DIR} space" elif [ ${code} -eq ${FAIL_INSUFFICIENT_SPACE} ] ; then - elog "${string} (reason:${code}:${FAIL_NOT_ENOUGH_SPACE_STR}) ; ${COLLECT_BASE_DIR} usage must be below ${MIN_PERCENT_SPACE_REQUIRED}%" + cause="${FAIL_INSUFFICIENT_SPACE_STR}" + error_log=true + extra=" ; ${COLLECT_BASE_DIR} usage must be below ${MIN_PERCENT_SPACE_REQUIRED}%" - elif [ ${code} -ge ${FAIL_TIMEOUT} -a ${code} -le ${FAIL_TIMEOUT9} ] ; then - wlog "${FAIL_TIMEOUT_STR} ; ${string} (reason:${code})" + elif [ ${code} -eq ${FAIL_INSUFFICIENT_SPACE_REMOTE} ] ; then + cause="${FAIL_INSUFFICIENT_SPACE_REMOTE_STR}" + extra=" ; ${COLLECT_BASE_DIR} usage must be below ${MIN_PERCENT_SPACE_REQUIRED}%" - elif [ ${code} -eq ${FAIL_SUBCLOUD_TIMEOUT} ] ; then - elog "${FAIL_SUBCLOUD_TIMEOUT_STR} ; ${string} (reason:${code})" + elif [ ${code} -eq ${FAIL_TIMEOUT_GLOBAL} ] ; then + cause="${FAIL_TIMEOUT_GLOBAL_STR}" - elif [ ${code} -eq ${FAIL_PASSWORD_PROMPT} ] ; then - elog "${string} (reason:${code}:failed to learn password)" + elif [ ${code} -eq ${FAIL_TIMEOUT_PW} ] ; then + cause="${FAIL_TIMEOUT_PW_STR}" + + elif [ ${code} -eq ${FAIL_TIMEOUT_OPERATION} ] ; then + cause="${FAIL_TIMEOUT_OPERATION_STR}" + + elif [ ${code} -eq ${FAIL_TIMEOUT_OPERATION_SSH} ] ; then + cause="${FAIL_TIMEOUT_OPERATION_SSH_STR}" + + elif [ ${code} -eq ${FAIL_TIMEOUT_OPERATION_SCP} ] ; then + cause="${FAIL_TIMEOUT_OPERATION_SCP_STR}" + + elif [ ${code} -eq ${FAIL_TIMEOUT_HOST_ACCESS} ] ; then + cause="${FAIL_TIMEOUT_HOST_ACCESS_STR}" + + elif [ ${code} -eq ${FAIL_TIMEOUT_SUBCLOUD_ACCESS} ] ; then + cause="${FAIL_TIMEOUT_SUBCLOUD_ACCESS_STR}" + + elif [ ${code} -eq ${FAIL_TIMEOUT_SCP} ] ; then + cause="${FAIL_TIMEOUT_SCP_STR}" + + elif [ ${code} -eq ${FAIL_TIMEOUT_SSH} ] ; then + cause="${FAIL_TIMEOUT_SSH_STR}" + + elif [ ${code} -eq ${FAIL_TIMEOUT_HOST} ] ; then + cause="${FAIL_TIMEOUT_HOST_STR}" + + elif [ ${code} -eq ${FAIL_TIMEOUT_SUBCLOUD} ] ; then + cause="${FAIL_TIMEOUT_SUBCLOUD_STR}" + + elif [ ${code} -eq ${FAIL_PASSWORDLESS} ] ; then + cause="${FAIL_PASSWORDLESS_STR}" + extra=" ; User '${UN}' may have passwordless sudo enabled. Please disable and retry" + error_log=true + + elif [ ${code} -eq ${FAIL_PASSWORDLESS_REMOTE} ] ; then + cause="${FAIL_PASSWORDLESS_REMOTE_STR}" + extra=" ; User '${UN}' may have passwordless sudo enabled. Please disable and retry" + + elif [ ${code} -eq ${FAIL_NOT_SUDOER} ] ; then + cause="${FAIL_NOT_SUDOER_STR}" + error_log=true + + elif [ ${code} -eq ${FAIL_NOT_SUDOER_REMOTE} ] ; then + cause="${FAIL_NOT_SUDOER_REMOTE_STR}" + extra=" ; User '${UN}' may not be in the sudoers file" elif [ ${code} -eq ${FAIL_DATE_FORMAT} ] ; then - elog "${FAIL_DATE_FORMAT_STR} ; ${string} (reason:${code})" + cause="${FAIL_DATE_FORMAT_STR}" + error_log=true elif [ ${code} -eq ${FAIL_NO_FILE_SPECIFIED} ] ; then - elog "${FAIL_NO_FILE_SPECIFIED_STR} ; ${string} (reason:${code})" + cause="${FAIL_NO_FILE_SPECIFIED_STR}" + error_log=true elif [ ${code} -eq ${FAIL_FILE_NOT_FOUND} ] ; then - elog "${FAIL_FILE_NOT_FOUND_STR} ; ${string} (reason:${code})" + cause="${FAIL_FILE_NOT_FOUND_STR}" + error_log=true elif [ ${code} -eq ${FAIL_FILE_EMPTY} ] ; then - elog "${FAIL_FILE_EMPTY_STR} ; ${string} (reason:${code})" + cause="${FAIL_FILE_EMPTY_STR}" + error_log=true elif [ ${code} -eq ${FAIL_NO_HOSTS} ] ; then - elog "${FAIL_NO_HOSTS_STR} ; ${string} (reason:${code})" + cause="${FAIL_NO_HOSTS_STR}" + error_log=true elif [ ${code} -eq ${FAIL_NO_SUBCLOUDS} ] ; then - elog "${FAIL_NO_SUBCLOUDS_STR} ; ${string} (reason:${code})" + cause="${FAIL_NO_SUBCLOUDS_STR}" + error_log=true elif [ ${code} -eq ${FAIL_MISSING_PARAMETER} ] ; then - elog "${FAIL_MISSING_PARAMETER_STR} ; ${string} (reason:${code})" + cause="${FAIL_MISSING_PARAMETER_STR}" + error_log=true elif [ ${code} -eq ${FAIL_TIMEOUT_ARG} ] ; then - wlog "${FAIL_TIMEOUT_ARG_STR} ; ${string} (reason:${code})" + cause="${FAIL_TIMEOUT_ARG_STR}" else - elog "${string} (reason:${code})" + cause="${FAIL_UNSPECIFIED_CAUSE_STR}" + error_log=true fi + + [ "${want_newline}" = true ] && echo "" + + if [ "${error_log}" = true ] ; then + elog "${cause} ; ${string}${extra} (reason:${code})" + else + wlog "${cause} ; ${string}${extra} (reason:${code})" + fi + [ "${want_exit}" = true ] && collect_exit ${code} } ########################################################################### @@ -648,6 +740,11 @@ while [[ ${#} -gt 0 ]] ; do -v|--verbose) USER_LOG_MODE=1 VERBOSE=true + redirect="/dev/stdout" + ;; + + --version) + ilog "Collect version: ${TOOL_VER}.${TOOL_REV}" ;; -r| --report) @@ -672,10 +769,6 @@ while [[ ${#} -gt 0 ]] ; do report_error "collect exit" ${FAIL_NO_HOSTS} collect_exit ${FAIL_NO_HOSTS} fi - if [ "${ACTIVE}" = false ] ; then - report_error "can only run collect for remote hosts on active controller" ${FAIL_INACTIVE} - collect_exit ${FAIL_INACTIVE} - fi HOSTLIST=(${2}) HOSTS=1 LISTING=true @@ -684,12 +777,7 @@ while [[ ${#} -gt 0 ]] ; do ;; -a|--all|all) - if [ "${ACTIVE}" = false ] ; then - wlog "collect with '${key}' option is only supported on an active controller ; defaulting to local collect" - else - ALLHOSTS=true - fi - + ALLHOSTS=true HOSTLIST=(${HOSTNAME}) HOSTS=1 clear_variable_args @@ -727,6 +815,7 @@ while [[ ${#} -gt 0 ]] ; do -d|--debug) DEBUG=true + expect_debug="-d" clear_variable_args ;; @@ -778,6 +867,13 @@ while [[ ${#} -gt 0 ]] ; do fi ;; + -pw|--password) + if [ ! -z "${2}" ] ; then + pw="${2}" + shift + fi + ;; + *) if [ "${LISTING}" = true ] ; then HOSTS=$((HOSTS+1)) @@ -813,10 +909,9 @@ elif (( STARTDATE > ENDDATE )) && [ "$STARTDATE" != "any" -a "$ENDDATE" != "any" fi -# startup state debug logs +# startup state debug logs for options dlog "${TOOL_NAME} ver ${TOOL_REV}.${TOOL_REV} (pid:$$)" dlog "USERNAME = ${USER}" -dlog "ACTIVE = ${ACTIVE}" dlog "HOSTNAME = ${HOSTNAME}" dlog "INVENTORY = ${INVENTORY}" dlog "STARTDATE = ${STARTDATE}" @@ -826,9 +921,150 @@ dlog "OMITCERTS = ${OMIT_CERTS}" dlog "ALLHOSTS = ${ALLHOSTS}" dlog "LISTING = ${LISTING}" dlog "CLEAN = ${CLEAN}" -dlog "TIMEOUT = ${TIMEOUT}" -dlog "SECONDS = ${SECONDS}" -dlog "UNTIL = ${UNTIL}" + + +############################################################################ +# +# Password handling +# +# If the password is not learned by other means by this time +# then prompt the user to enter it. +# The password is used for expect driven requests. +# +############################################################################ +# dlog "password coming in is:$pw" + +if [ -z "${pw}" ] ; then + read -s -p "[sudo] password for ${USER}:" pw + echo "" +fi + +# Save the original unmodified password so it can be used in a subcloud +# collect which calls collect directly again. +# In that case we don't want to do a double special character replacement. +PW=${pw} + +# When the pw is used locally for expect requests ... +# +# Although bash 'read' will handle sanitizing the password +# input for the purposes of storing it in ${pw}, expect +# will need certain special characters to be backslash +# delimited + +pw=$(echo "${pw}" | sed 's/\\/\\\\/g') # replace all '\' with '\\' +pw=$(echo "${pw}" | sed 's/\]/\\]/g') # replace all ']' with '\]' +pw=$(echo "${pw}" | sed 's/\[/\\[/g') # replace all '[' with '\[' +pw=$(echo "${pw}" | sed 's/\$/\\$/g') # replace all '$' with '\$' +pw=$(echo "${pw}" | sed 's/\"/\\"/g') # replace all '"' with '\"' + +########################################################################### +# +# Name : passwordless_sudo_test +# +# Purpose : Verify sudo is working for this user. +# Verify that passwordless sudo is not enabled. +# +# Description: cat the content of the /usr/local/sbin/expect_done as sudo +# +########################################################################### + +function passwordless_sudo_test() +{ +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${HOSTNAME}_${FUNCNAME[0]} } + log_user ${USER_LOG_MODE} + spawn bash -i + set timeout ${SUDO_TIMEOUT} + expect -re $ + send "sudo cat /usr/local/sbin/expect_done\n" + expect { + "assword:" { + send "${pw}\r" + expect { + "${cmd_done_sig}" { exit ${PASS} } + "${pw_error}" { exit ${FAIL_PASSWORD} } + "${su_error}" { exit ${FAIL_NOT_SUDOER}} + timeout { exit ${FAIL_TIMEOUT_OPERATION}} + } + } + "${pw_error}" { exit ${FAIL_PASSWORD} } + timeout { exit ${FAIL_PASSWORDLESS} } + } +EOF + local rc=${?} + if [ ${rc} -ne ${PASS} ] ; then + if [ ${rc} -eq ${FAIL_NOT_SUDOER} ] ; then + report_error "User '${UN}' is not in the sudoers file" ${rc} + elif [ ${rc} -eq ${FAIL_PASSWORD} ] ; then + report_error "Supplied password appears invalid" ${rc} + elif [ ${rc} -eq ${FAIL_PASSWORDLESS} ] ; then + report_error "Password test failed for ${HOSTNAME}" ${rc} + elif [ ${rc} -eq ${FAIL_TIMEOUT_OPERATION} ] ; then + report_error "sudo cat /usr/local/sbin/expect_done file failed" ${rc} + else + report_error "Unexpected error code" ${rc} + fi + collect_exit ${rc} + fi +} + +########################################################################## +# +# Name : remove_debug_files_local +# +# Purpose : Remove all collect expect debug files from /tmp +# +########################################################################### + +function remove_debug_files_local() +{ +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 +trap exit {SIGINT SIGTERM} +if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${HOSTNAME}_${FUNCNAME[0]} } +log_user ${USER_LOG_MODE} +spawn bash -i +set timeout ${SUDO_TIMEOUT} +expect -re $ +send -- "sudo rm -f ${EXPECT_LOG_FILE}_* ; cat ${cmd_done_file}\n" +expect { + "assword:" { send -- "${pw}\r" ; exp_continue } + "${cmd_done_sig}" { exit ${PASS} } + "annot remove" { exit ${FAIL_CLEANUP} } + "${pw_error}" { exit ${FAIL_PASSWORD} } + "${ac_error}" { exit ${FAIL_PERMISSION} } + timeout { exit ${FAIL_TIMEOUT_OPERATION} } +} +EOF +rc=${?} +if [ ${rc} -ne ${PASS} ] ; then + wlog "unable to remove /tmp/${EXPECT_LOG_FILE}_ debug files" +fi +return ${rc} +} + +# This call will update ACTIVE to true if this host is the active controller +source_openrc_if_needed + +# run the passwordless sudo test +passwordless_sudo_test + +# cleanup any existing debug logs if in debug mode +if [ "${DEBUG}" = true ] ; then + remove_debug_files_local +fi + +# Handle cases where the user requested to collect remote hosts while not on the active controller +if [ "${ACTIVE}" = false ] ; then + if [ "${ALLHOSTS}" = true ] ; then + wlog "collect with 'all' option is only supported on the active controller ; defaulting to local collect" + ALLHOSTS=false + elif [ ${HOSTS} -gt 1 ] ; then + report_error "can only run collect for remote hosts on the active controller" ${FAIL_INACTIVE} + elif [ "${HOSTLIST[0]}" != "${HOSTNAME}" ] ; then + report_error "can only run collect for remote hosts on the active controller" ${FAIL_INACTIVE} + fi +fi # the continue option is only supported for subcloud collect if [[ "${SUBCLOUD_COLLECT_CONTINUE}" = true && "${SUBCLOUD_COLLECT}" = false ]] ; then @@ -1097,90 +1333,19 @@ fi if [ "${DCROLE}" != "" ] ; then dlog "DCROLE = ${DCROLE}" fi + +dlog "ACTIVE = ${ACTIVE}" +dlog "TIMEOUT = ${TIMEOUT}" +dlog "SECONDS = ${SECONDS}" +dlog "UNTIL = ${UNTIL}" dlog "PARALLEL = ${PARALLEL_COLLECT_MODE}" dlog "COLLECT_TYPE = ${COLLECT_TYPE}" dlog "COLLECT_NAME = ${COLLECT_NAME}" dlog "COLLECT_DIR = ${COLLECT_DIR}" dlog "TARBALL_NAME = ${TARBALL_NAME}" -############################################################################ -# -# Password handling -# -# If the password is not learned by other means by this time -# then prompt the user to enter it. -# -# The password is used for expect driven requests. -# -############################################################################ -# dlog "password coming in is:$pw" - -if [ -z "${pw}" ] ; then - read -s -p "[sudo] password for ${USER}:" pw - echo "" -fi - -# Save the original unmodified password so it can be used in a subcloud -# collect which calls collect directly again. -# In that case we don't want to do a double special character replacement. -PW=${pw} - -# When the pw is used locally for expect requests ... -# -# Although bash 'read' will handle sanitizing the password -# input for the purposes of storing it in ${pw}, expect -# will need certain special characters to be backslash -# delimited -pw=$(echo "${pw}" | sed 's/\\/\\\\/g') # replace all '\' with '\\' -pw=$(echo "${pw}" | sed 's/\]/\\]/g') # replace all ']' with '\]' -pw=$(echo "${pw}" | sed 's/\[/\\[/g') # replace all '[' with '\[' -pw=$(echo "${pw}" | sed 's/\$/\\$/g') # replace all '$' with '\$' -pw=$(echo "${pw}" | sed 's/\"/\\"/g') # replace all '"' with '\"' ilog "collect bundle timeout set to $((${TIMEOUT}/60)) minutes" - -########################################################################### -# -# Name : passwordless_sudo_test -# -# Purpose : Verify to detect passwordless sudo for getting password promptand -# fails the collect if there is no password prompt -# -# Description: cat the content of the /usr/local/sbin/expect_done -# -########################################################################### - -function passwordless_sudo_test() -{ - -/usr/bin/expect << EOF - log_user ${USER_LOG_MODE} - spawn bash -i - set timeout ${SUDO_TIMEOUT} - expect -re $ - send "sudo cat /usr/local/sbin/expect_done\n" - expect { - "assword:" { - send "${pw}\r" - expect { - "${cmd_done_sig}" { exit ${PASS} } - "${pw_error}" { exit ${FAIL_PASSWORD} } - timeout { exit ${FAIL_TIMEOUT1} } - } - } - "${pw_error}" { exit ${FAIL_PASSWORD} } - timeout { exit ${FAIL_TIMEOUT} } - } -EOF - local rc=${?} - if [ ${rc} -ne ${PASS} ] ; then - report_error "Timeout waiting for password prompt. Passwordless sudo may be enabled. Please disable and retry." ${rc} - collect_exit ${rc} - fi -} - -passwordless_sudo_test - ########################################################################### # # Name : check_host_reachable @@ -1201,7 +1366,9 @@ function check_host_reachable() return ${PASS} fi -/usr/bin/expect << EOF +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${hostname}_${FUNCNAME[0]} } log_user ${USER_LOG_MODE} spawn bash -i expect -re $ @@ -1216,8 +1383,8 @@ function check_host_reachable() "${cmd_done_sig}" { exit ${PASS} } "No such file or directory" { exit ${FAIL_FILE_NOT_FOUND} } "${pw_error}" { exit ${FAIL_PASSWORD} } - "${ac_error}" { exit ${FAIL_PERMISSION_SKIP}} - timeout { exit ${FAIL_TIMEOUT3} } + "${ac_error}" { exit ${FAIL_PERMISSION_REMOTE}} + timeout { exit ${FAIL_TIMEOUT_SSH}} } } "(yes/no)?" { @@ -1239,12 +1406,78 @@ function check_host_reachable() "Connection timed out" { exit ${FAIL_UNREACHABLE} } - timeout { exit ${FAIL_TIMEOUT} } + timeout { exit ${FAIL_TIMEOUT_PW} } } EOF - return ${?} + rc=$? + if [ ${rc} -eq ${FAIL_PERMISSION_REMOTE} ] ; then + cause="permissions" + elif [ ${rc} -eq ${FAIL_TIMEOUT_SSH} ] ; then + cause="ssh timeout" + elif [ ${rc} -eq ${FAIL_PASSWORD} ] ; then + cause="password error" + elif [ ${rc} -eq ${FAIL_FILE_NOT_FOUND} ] ; then + cause="file not found" + elif [ ${rc} -eq ${FAIL_TIMEOUT_PW} ] ; then + cause="password timeout" + else + cause="access" + fi + + if [ ${rc} -ne 0 ] ; then + rc=${FAIL_UNREACHABLE} + report_error "cannot collect from ${hostname} (${cause})" ${rc} + fi + return ${rc} } +########################################################################### +# +# Name : create_collect_log +# +# Purpose : Get the last few collect logs from user.log and +# put them in /collect.log +# +########################################################################### +function create_collect_log () +{ + local logs=100 + local temp_file=$(mktemp /tmp/collect_log.XXXXXX) + +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${HOSTNAME}_${FUNCNAME[0]} } + log_user ${USER_LOG_MODE} + spawn bash -i + set timeout ${SUDO_TIMEOUT} + expect -re $ + send -- "sudo tail -${logs} /var/log/user.log | grep 'COLLECT:' | grep -v '${FAIL_OUT_OF_SPACE_STR}' > ${temp_file} ; cat ${cmd_done_file}\n" + expect { + "assword:" { send "${pw}\r" ; exp_continue } + "${cmd_done_sig}" { exit ${PASS} } + "${pw_error}" { exit ${FAIL_PASSWORD} } + "${ac_error}" { exit ${FAIL_PERMISSION} } + timeout { exit ${FAIL_TIMEOUT_OPERATION} } + } +EOF + local rc=${?} + if [ ${rc} -ne ${PASS} ] ; then + report_error "create_collect_log ${HOSTNAME} failed" ${rc} + fi + + # get rid of older collect logs ; that may be from a different run + while IFS= read -r line; do + # get the timestamp from the log entry + log_timestamp=$(echo "$line" | awk '{print $1}' | cut -d. -f1) + + # compare the log timestamp with the timestamp we got at the beginning of collect + if [[ "${log_timestamp}" > "${LOGDATE}" ]]; then + echo "${line}" >> ${COLLECT_DIR}/${COLLECT_LOG} + fi + done < "${temp_file}" + rm -f "${temp_file}" + return ${rc} +} ########################################################################### # @@ -1262,7 +1495,9 @@ function clean_scratch_dir_local () local this_hostname=${1} local directory=${2} -/usr/bin/expect << EOF +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${this_hostname}_${FUNCNAME[0]} } log_user ${USER_LOG_MODE} spawn bash -i set timeout ${SUDO_TIMEOUT} @@ -1274,7 +1509,7 @@ function clean_scratch_dir_local () "annot remove" { exit ${FAIL_CLEANUP} } "${pw_error}" { exit ${FAIL_PASSWORD} } "${ac_error}" { exit ${FAIL_PERMISSION} } - timeout { exit ${FAIL_TIMEOUT} } + timeout { exit ${FAIL_TIMEOUT_OPERATION}} } EOF local rc=${?} @@ -1300,7 +1535,9 @@ function clean_scratch_dir_remote() local this_hostname=${1} local directory=${2} -/usr/bin/expect << EOF +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${this_hostname}_${FUNCNAME[0]} } log_user ${USER_LOG_MODE} spawn bash -i expect -re $ @@ -1320,11 +1557,11 @@ function clean_scratch_dir_remote() "${cmd_done_file}: No such file or directory" { exit ${PASS} } "annot remove" { exit ${FAIL_CLEANUP} } "${pw_error}" { exit ${FAIL_PASSWORD} } - "${ac_error}" { exit ${FAIL_PERMISSION_SKIP}} - timeout { exit ${FAIL_TIMEOUT3} } + "${ac_error}" { exit ${FAIL_PERMISSION_REMOTE}} + timeout { exit ${FAIL_TIMEOUT_HOST}} } } - timeout { exit ${FAIL_TIMEOUT1} } + timeout { exit ${FAIL_TIMEOUT_OPERATION_SSH}} } } "(yes/no)?" { @@ -1337,7 +1574,7 @@ function clean_scratch_dir_remote() "Could not resolve hostname" { exit ${FAIL_UNREACHABLE} } - timeout { exit ${FAIL_TIMEOUT} } + timeout { exit ${FAIL_TIMEOUT_PW} } } EOF local rc=${?} @@ -1376,7 +1613,9 @@ function delete_remote_dir_or_file() alt_login_prompt=${login_prompt}; fi -/usr/bin/expect << EOF +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${remote_hostname}_${FUNCNAME[0]} } log_user ${USER_LOG_MODE} spawn bash -i expect -re $ @@ -1386,7 +1625,7 @@ function delete_remote_dir_or_file() "assword:" { send "${pw}\r" expect { - timeout { exit ${FAIL_TIMEOUT1} } + timeout { exit ${FAIL_TIMEOUT_SSH} } "${login_prompt}" {} "${alt_login_prompt}" {} } @@ -1399,8 +1638,8 @@ function delete_remote_dir_or_file() "${cmd_done_file}: No such file or directory" { exit ${PASS} } "annot remove" { exit ${FAIL_CLEANUP} } "${pw_error}" { exit ${FAIL_PASSWORD} } - "${ac_error}" { exit ${FAIL_PERMISSION_SKIP}} - timeout { exit ${FAIL_TIMEOUT3} } + "${ac_error}" { exit ${FAIL_PERMISSION_REMOTE}} + timeout { exit ${FAIL_TIMEOUT_OPERATION}} } } "(yes/no)?" { @@ -1413,7 +1652,7 @@ function delete_remote_dir_or_file() "Could not resolve hostname" { exit ${FAIL_UNREACHABLE} } - timeout { exit ${FAIL_TIMEOUT} } + timeout { exit ${FAIL_TIMEOUT_PW} } } EOF local rc=${?} @@ -1446,7 +1685,9 @@ function get_file_from_host() dlog "get_file_from_host: ${UN}@${remote_hostname}:${COLLECT_BASE_DIR}/${remote_file} ${local_dest}" -/usr/bin/expect << EOF +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${remote_hostname}_${FUNCNAME[0]} } log_user ${USER_LOG_MODE} spawn bash -i set timeout ${SCP_TIMEOUT} @@ -1458,17 +1699,21 @@ function get_file_from_host() expect { "100%" { exit ${PASS} } "${pw_error}" { exit ${FAIL_PASSWORD} } - "${ac_error}" { exit ${FAIL_PERMISSION_SKIP}} - timeout { exit ${FAIL_TIMEOUT1} } + "${ac_error}" { exit ${FAIL_PERMISSION_REMOTE}} + timeout { exit ${FAIL_TIMEOUT_SCP} } } } + "(yes/no)?" { + send "yes\r" + exp_continue + } "No route to host" { exit ${FAIL_UNREACHABLE} } "Could not resolve hostname" { exit ${FAIL_UNREACHABLE} } - timeout { exit ${FAIL_TIMEOUT} } + timeout { exit ${FAIL_TIMEOUT_PW} } } EOF local rc=${?} @@ -1480,6 +1725,9 @@ EOF if [ "$?" == "0" ] ; then remove_file_local "${local_dest}/${remote_file}" rc=${FAIL_OUT_OF_SPACE} + else + chown_file_or_dir_local ${UN} ${local_dest} + chown_file_or_dir_local ${UN} ${local_dest}/${remote_file} fi fi @@ -1506,7 +1754,9 @@ function copy_file_to_host() local remote_hostname="${2}" local remote_dir="${3}" -/usr/bin/expect << EOF +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${remote_hostname}_${FUNCNAME[0]} } log_user ${USER_LOG_MODE} spawn bash -i set timeout ${SCP_TIMEOUT} @@ -1518,8 +1768,8 @@ function copy_file_to_host() expect { "100%" { exit ${PASS} } "${pw_error}" { exit ${FAIL_PASSWORD} } - "${ac_error}" { exit ${FAIL_PERMISSION_SKIP}} - timeout { exit ${FAIL_TIMEOUT1} } + "${ac_error}" { exit ${FAIL_PERMISSION_REMOTE}} + timeout { exit ${FAIL_TIMEOUT_SCP} } } } "No route to host" { @@ -1528,7 +1778,7 @@ function copy_file_to_host() "Could not resolve hostname" { exit ${FAIL_UNREACHABLE} } - timeout { exit ${FAIL_TIMEOUT} } + timeout { exit ${FAIL_TIMEOUT_PW} } } EOF local rc=${?} @@ -1538,25 +1788,25 @@ EOF return ${rc} } + ########################################################################### # -# Name : create_collect_dir_local +# Name : create_dir_local # -# Purpose : Create the local dated collect dir where all -# the tarballs for this collect will get put. +# Purpose : Create a local directory using sudo and then change +# the owner from root to the current username. # -# Assumptions: Permissions are set to make it easy to copy -# tarballs from remote host into -# -# Parameters: $1 - the fill dir +# Parameters: $1 - the dir to create # ########################################################################### -function create_collect_dir_local() +function create_dir_local() { local dir=${1} -/usr/bin/expect << EOF +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${HOSTNAME}_${FUNCNAME[0]} } log_user ${USER_LOG_MODE} spawn bash -i set timeout ${SUDO_TIMEOUT} @@ -1579,21 +1829,121 @@ function create_collect_dir_local() EOF local rc=${?} if [ ${rc} -ne ${PASS} ] ; then - report_error "failed to create_collect_dir_local for ${dir}" ${rc} + report_error "failed to create_dir_local for ${dir}" ${rc} collect_exit ${rc} fi + + chown_file_or_dir_local $(whoami) ${dir} return ${rc} } -############################################################################ +########################################################################### # -# Create the local collect dir where the tarball(s) will temporarily stored +# Name : chown_file_or_dir_local # -# Note: Catches the password error case +# Purpose : Change the ownership of a file or directory on +# the local machine using sudo. # -############################################################################ +# Warning : change of ownership is bypassed for sysadmin users. +# sysadmin is an invalid group for chown. +# +# Parameters: $1 - the file or dir +# +########################################################################### + +function chown_file_or_dir_local() +{ + local user=${1} + local object=${2} + + # sysadmin is an invalid group for chown + [ "${user}" == "sysadmin" ] && return + + # change the ownership to the current user +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${HOSTNAME}_${FUNCNAME[0]} } + log_user ${USER_LOG_MODE} + spawn bash -i + set timeout ${SUDO_TIMEOUT} + expect -re $ + send "sudo chown ${user}:${user} ${object} ; cat ${cmd_done_file}\n" + expect { + "assword:" { + send "${pw}\r" + expect { + "${pw_error}" { exit ${FAIL_PASSWORD} } + "${ac_error}" { exit ${FAIL_PERMISSION}} + "${cmd_done_sig}" { exit ${PASS} } + timeout { exit ${FAIL_TIMEOUT1} } + } + } + "${cmd_done_sig}" { exit ${PASS}} + "${ac_error}" { exit ${FAIL_PERMISSION}} + timeout { exit ${FAIL_TIMEOUT}} + } +EOF + local rc=${?} + if [ ${rc} -ne ${PASS} ] ; then + report_error "failed to change ownership of ${object} to ${user} in chown_file_or_dir_local" ${rc} + fi + + return ${rc} +} + +########################################################################### +# +# Name : create_collect_dir_local +# +# Purpose : Create the local dated collect dir where all +# the tarballs for this collect will get put. +# Then change the owner from root to the current username. +# +# Assumptions: Permissions are set to make it easy to copy +# tarballs from remote host into +# +# Parameters: $1 - the fill dir +# +########################################################################### + +function create_collect_dir_local() +{ + local dir=${1} + +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${HOSTNAME}_${FUNCNAME[0]} } + log_user ${USER_LOG_MODE} + spawn bash -i + set timeout ${SUDO_TIMEOUT} + expect -re $ + send "sudo mkdir -m 775 -p ${dir} ; cat ${cmd_done_file}\n" + expect { + "assword:" { + send "${pw}\r" + expect { + "${cmd_done_sig}" { exit ${PASS} } + "${pw_error}" { exit ${FAIL_PASSWORD} } + "${ac_error}" { exit ${FAIL_PERMISSION}} + timeout { exit ${FAIL_TIMEOUT_OPERATION} } + } + } + "${cmd_done_sig}" { exit ${PASS} } + "${ac_error}" { exit ${FAIL_PERMISSION}} + timeout { exit ${FAIL_TIMEOUT_PW} } + } +EOF + local rc=${?} + if [ ${rc} -ne ${PASS} ] ; then + report_error "failed to create_collect_dir_local for ${dir}" ${rc} + collect_exit ${rc} + fi + + chown_file_or_dir_local $(whoami) ${dir} + + return ${rc} +} -create_collect_dir_local "${COLLECT_DIR}" ########################################################################## # @@ -1612,7 +1962,9 @@ function remove_file_local() if [ -e ${local_file} ] ; then -/usr/bin/expect << EOF +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${HOSTNAME}_${FUNCNAME[0]} } log_user ${USER_LOG_MODE} spawn bash -i set timeout ${SUDO_TIMEOUT} @@ -1624,7 +1976,7 @@ function remove_file_local() "annot remove" { exit ${FAIL_CLEANUP} } "${pw_error}" { exit ${FAIL_PASSWORD} } "${ac_error}" { exit ${FAIL_PERMISSION} } - timeout { exit ${FAIL_TIMEOUT} } + timeout { exit ${FAIL_TIMEOUT_OPERATION} } } EOF rc=${?} @@ -1649,7 +2001,9 @@ function remove_dir_local() { local dir=${1} -/usr/bin/expect << EOF +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${HOSTNAME}_${FUNCNAME[0]} } log_user ${USER_LOG_MODE} spawn bash -i set timeout ${SUDO_TIMEOUT} @@ -1661,7 +2015,7 @@ function remove_dir_local() "annot remove" { exit ${FAIL_CLEANUP} } "${pw_error}" { exit ${FAIL_PASSWORD} } "${ac_error}" { exit ${FAIL_PERMISSION} } - timeout { exit ${FAIL_TIMEOUT} } + timeout { exit ${FAIL_TIMEOUT_OPERATION} } } EOF local rc=${?} @@ -1688,7 +2042,9 @@ function move_file_local() local src=${1} local dst=${2} -/usr/bin/expect << EOF +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${HOSTNAME}_${FUNCNAME[0]} } log_user ${USER_LOG_MODE} spawn bash -i set timeout ${SUDO_TIMEOUT} @@ -1700,7 +2056,7 @@ function move_file_local() "annot remove" { exit ${FAIL_CLEANUP} } "${pw_error}" { exit ${FAIL_PASSWORD} } "${ac_error}" { exit ${FAIL_PERMISSION} } - timeout { exit ${FAIL_TIMEOUT} } + timeout { exit ${FAIL_TIMEOUT_OPERATION} } } EOF local rc=${?} @@ -1710,10 +2066,6 @@ EOF return ${rc} } - -########################################################################### - - ########################################################################### function scratch_full() { @@ -1766,7 +2118,8 @@ function echo_stats() if [ -e ${file} ] ; then avail=$(df -h --output=pcent ${COLLECT_BASE_DIR} | grep -v Use) size=$(du -h ${file} | cut -f 1 2>/dev/null) - if [ $? -eq 0 ] ; then + rc=$? + if [ ${rc} -eq 0 ] ; then if [ "${label}" == "stats-only" ] ; then printf "%s %5s %3s)\n" "${stats}" "${size}" "${avail}" log "${MSG} $stats ${size} ${avail})" @@ -1779,9 +2132,12 @@ function echo_stats() log "${MSG} $stats ${size} ${avail})" fi return + else + printf "\nstats error: size query error ; rc:%s\n" "${rc}" fi + else + printf "\nstats error: file '%s' does not exist\n" "${file}" fi - printf "stats error)\n" } ############################################################################ @@ -1820,26 +2176,35 @@ function collect_host_run() if [ "${host}" == "${HOSTNAME}" ] ; then -/usr/bin/expect << EOF + collect_cmd="sudo REMOTE_HOST=false OMIT_CERTS=${OMIT_CERTS} SKIP_MASK=${SKIP_MASK} ${collect_host} ${TARNAME} ${STARTDATE_OPTION} ${STARTDATE} ${STARTTIME} ${ENDDATE_OPTION} ${ENDDATE} ${ENDTIME} ${DEBUG} ${INVENTORY}" + log "${collect_cmd}" + +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${host}_${FUNCNAME[0]} } log_user ${USER_LOG_MODE} spawn bash -i set timeout ${TIMEOUT} - send "sudo OMIT_CERTS=${OMIT_CERTS} SKIP_MASK=${SKIP_MASK} ${collect_host} ${TARNAME} ${STARTDATE_OPTION} ${STARTDATE} ${STARTTIME} ${ENDDATE_OPTION} ${ENDDATE} ${ENDTIME} ${VERBOSE} ${INVENTORY}\n" + send "${collect_cmd}\n" expect { "assword:" { send "${pw}\r" expect { "${FAIL_INSUFFICIENT_SPACE_STR}" { exit ${FAIL_INSUFFICIENT_SPACE}} - "${FAIL_OUT_OF_SPACE_STR}" { exit ${FAIL_OUT_OF_SPACE}} - "${collect_done}" { exit ${PASS} } - "${pw_error}" { exit ${FAIL_PASSWORD} } - "${ac_error}" { exit ${FAIL_PERMISSION}} - timeout { exit ${FAIL_TIMEOUT} } + "${FAIL_OUT_OF_SPACE_STR}" { exit ${FAIL_OUT_OF_SPACE}} + "${FAIL_TIMEOUT_OPERATION_STR}" { exit ${FAIL_TIMEOUT_OPERATION}} + "${FAIL_INVALID_PASSWORD_STR}" { exit ${FAIL_INVALID_PASSWORD}} + "${FAIL_PASSWORDLESS_STR}" { exit ${FAIL_PASSWORDLESS}} + "${FAIL_NOT_SUDOER_STR}" { exit ${FAIL_NOT_SUDOER}} + "${su_error}" { exit ${FAIL_NOT_SUDOER}} + "${pw_error}" { exit ${FAIL_PASSWORD}} + "${ac_error}" { exit ${FAIL_PERMISSION}} + "${collect_done}" { exit ${PASS}} + timeout { exit ${FAIL_TIMEOUT_HOST}} } } - timeout { exit ${FAIL_TIMEOUT} } + timeout { exit ${FAIL_TIMEOUT_PW} } } exit { ${FAIL} } EOF @@ -1847,47 +2212,86 @@ EOF # otherwise the host is remote else -/usr/bin/expect << EOF + ssh_cmd="${SSH_CMD} ${UN}@${host}" + log "${ssh_cmd}" + collect_cmd="sudo REMOTE_HOST=true OMIT_CERTS=${OMIT_CERTS} SKIP_MASK=${SKIP_MASK} ${collect_host} ${TARNAME} ${STARTDATE_OPTION} ${STARTDATE} ${STARTTIME} ${ENDDATE_OPTION} ${ENDDATE} ${ENDTIME} ${DEBUG} ${INVENTORY}" + log "${collect_cmd}" + +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 trap exit {SIGINT SIGTERM} log_user ${USER_LOG_MODE} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${host}_${FUNCNAME[0]} } spawn bash -i set timeout ${SSH_TIMEOUT} expect -re $ - send "${SSH_CMD} ${UN}@${host}\n" + send "${ssh_cmd}\n" expect { "assword:" { send "${pw}\r" expect { "${host}:" { - set timeout ${COLLECT_HOST_TIMEOUT} - send "sudo OMIT_CERTS=${OMIT_CERTS} SKIP_MASK=${SKIP_MASK} ${collect_host} ${TARNAME} ${STARTDATE_OPTION} ${STARTDATE} ${STARTTIME} ${ENDDATE_OPTION} ${ENDDATE} ${ENDTIME} ${VERBOSE} ${INVENTORY}\n" + set timeout ${SUDO_TIMEOUT} + send "${collect_cmd}\n" expect { "assword:" { + set timeout ${COLLECT_HOST_TIMEOUT} send "${pw}\r" expect { "${FAIL_INSUFFICIENT_SPACE_STR}" { send "exit\r" exit ${FAIL_INSUFFICIENT_SPACE} } + "${FAIL_INSUFFICIENT_SPACE_REMOTE_STR}" { + send "exit\r" + exit ${FAIL_INSUFFICIENT_SPACE_REMOTE} + } + "${FAIL_OUT_OF_SPACE_REMOTE_STR}" { + send "exit\r" + exit ${FAIL_OUT_OF_SPACE_REMOTE} + } + "${FAIL_NOT_ENOUGH_SPACE_REMOTE_STR}" { + send "exit\r" + exit ${FAIL_NOT_ENOUGH_SPACE_REMOTE} + } + "${FAIL_INVALID_PASSWORD_STR}" { + send "exit\r" + exit ${FAIL_INVALID_PASSWORD} + } + "${FAIL_TIMEOUT_OPERATION_STR}" { + send "exit\r" + exit ${FAIL_TIMEOUT_OPERATION} + } "${FAIL_OUT_OF_SPACE_STR}" { send "exit\r" exit ${FAIL_OUT_OF_SPACE} } + "${FAIL_NOT_SUDOER_REMOTE_STR}" { + send "exit\r" + exit ${FAIL_NOT_SUDOER_REMOTE} + } + "${su_error}" { + send "exit\r" + exit ${FAIL_NOT_SUDOER_REMOTE} + } "${collect_done}" { send "exit\r" exit ${PASS} } "${pw_error}" { exit ${FAIL_PASSWORD} } - "${ac_error}" { exit ${FAIL_PERMISSION_SKIP}} - timeout { exit ${FAIL_TIMEOUT5} } + "${ac_error}" { exit ${FAIL_PERMISSION_REMOTE}} + timeout { exit ${FAIL_TIMEOUT_HOST} } } } - timeout { exit ${FAIL_TIMEOUT4} } + "${su_error}" { + send "exit\r" + exit ${FAIL_NOT_SUDOER_REMOTE} + } + timeout { exit ${FAIL_PASSWORDLESS_REMOTE} } } } "${pw_error}" { exit ${FAIL_PASSWORD} } - "${ac_error}" { exit ${FAIL_PERMISSION_SKIP}} - timeout { exit ${FAIL_TIMEOUT3} } + "${ac_error}" { exit ${FAIL_PERMISSION_REMOTE}} + timeout { exit ${FAIL_TIMEOUT_HOST_ACCESS} } } } "(yes/no)?" { @@ -1904,7 +2308,7 @@ EOF send "rm -f /home/${UN}/.ssh/known_hosts\n" exit ${FAIL} } - timeout { exit ${FAIL_TIMEOUT} } + timeout { exit ${FAIL_TIMEOUT_PW} } } exit { $FAIL } EOF @@ -1938,33 +2342,29 @@ function collect_subcloud_run() fi # build the command starting with the final tarball name prefix - collect_cmd=("-n ${tarname}") + collect_cmd="-n ${tarname}" # all hosts in a subcloud are collected - collect_cmd+=("-a") + collect_cmd+=" -a" # pass the report option to the subcloud if specified - if [ "${REPORT}" = true ] ; then - collect_cmd+=("-r") - fi + [ "${REPORT}" = true ] && collect_cmd+=" -r" + + # pass security options if specified + [ "${SKIP_MASK}" = true ] && collect_cmd+=" --skip-mask" + [ "${OMIT_CERTS}" = true ] && collect_cmd+=" --omit-certs" # all subcloud hosts are collected in parallel unless legacy more is specified - if [ "${PARALLEL_COLLECT_MODE}" = false ] ; then - collect_cmd+=("-in") - fi - if [ "${DEBUG}" = true ] ; then - collect_cmd+=("-d") - fi - if [ "${VERBOSE}" = true ] ; then - collect_cmd+=("-v") - fi + [ "${PARALLEL_COLLECT_MODE}" = false ] && collect_cmd+=" -in" + + [ "${DEBUG}" = true ] && collect_cmd+=" -d" # pass the timeout to the subcloud - collect_cmd+=("-t $((${TIMEOUT}/60))") + collect_cmd+=" -t $((${TIMEOUT}/60))" # pass the date range to the subcloud - collect_cmd+=("--start-date ${STARTDATE}") - collect_cmd+=("--end-date $ENDDATE") + collect_cmd+=" --start-date ${STARTDATE}" + collect_cmd+=" --end-date ${ENDDATE}" # copy the pw file to the subcloud and then cleanup TEMPFILE=$(mktemp) @@ -1980,7 +2380,7 @@ function collect_subcloud_run() fi # tell the remote subcloud the name of the password file - collect_cmd+=("-f ${TEMPFILE}") + collect_cmd+=" -f ${TEMPFILE}" # Save current user log level local save=${USER_LOG_MODE} @@ -1988,31 +2388,51 @@ function collect_subcloud_run() USER_LOG_MODE=1 fi - # echo "Subcloud Collect: ${subcloud} ${collect_cmd[@]}" -/usr/bin/expect << EOF + log "${collect} ${collect_cmd}" + ssh_cmd="${SSH_CMD} ${UN}@${subcloud}" + log "${ssh_cmd}" + +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${subcloud}_${FUNCNAME[0]} } log_user ${USER_LOG_MODE} spawn bash -i set timeout ${SSH_TIMEOUT} expect -re $ - send "${SSH_CMD} ${UN}@${subcloud}\n" + send "${ssh_cmd}\n" expect { "assword:" { send "${pw}\r" expect { "${pw_error}" { exit ${FAIL_PASSWORD} } - "${ac_error}" { exit ${FAIL_PERMISSION_SKIP}} - timeout { exit ${FAIL_TIMEOUT3} } + "${ac_error}" { exit ${FAIL_PERMISSION_REMOTE}} + timeout { exit ${FAIL_TIMEOUT_SUBCLOUD_ACCESS} } "${SUBCLOUD_LOGIN_PROMPT}" {} "${subcloud}:" {} } set timeout ${TIMEOUT} - send "${collect} ${collect_cmd[@]}\n" + send "${collect} ${collect_cmd}\n" expect { "${collect_done}" { send "exit\r" exit ${PASS} } + "${FAIL_NOT_SUDOER_STR}" { + send "exit\r" + exit ${FAIL_NOT_SUDOER} + } + "${FAIL_INVALID_PASSWORD_STR}" { + send "exit\r" + exit ${FAIL_PASSWORD} + } + "${FAIL_PASSWORDLESS_STR}" { + send "exit\r" + exit ${FAIL_PASSWORDLESS} + } + "${FAIL_NOT_ENOUGH_SPACE_STR}" { + send "exit\r" + exit ${FAIL_OUT_OF_SPACE} + } "${FAIL_INSUFFICIENT_SPACE_STR}" { send "exit\r" exit ${FAIL_INSUFFICIENT_SPACE} @@ -2021,10 +2441,6 @@ function collect_subcloud_run() send "exit\r" exit ${FAIL_OUT_OF_SPACE} } - "${FAIL_PASSWORD_PROMPT_STR}" { - send "exit\r" - exit ${FAIL_PASSWORD_PROMPT} - } "${COLLECT_ERROR} ${FAIL_FILE_EMPTY_STR}" { send "exit\r" exit ${FAIL_FILE_EMPTY} @@ -2057,15 +2473,15 @@ function collect_subcloud_run() send "exit\r" exit ${FAIL_NO_FILE_SPECIFIED} } - "${COLLECT_ERROR} ${FAIL_SUBCLOUD_TIMEOUT_STR}" { + "${COLLECT_ERROR} ${FAIL_TIMEOUT_SUBCLOUD_STR}" { send "exit\r" - exit ${FAIL_SUBCLOUD_TIMEOUT} + exit ${FAIL_TIMEOUT_SUBCLOUD} } "${COLLECT_ERROR}" { send "exit\r" exit ${FAIL} } - timeout { exit ${FAIL_TIMEOUT5} } + timeout { exit ${FAIL_TIMEOUT_SUBCLOUD} } } } "(yes/no)?" { @@ -2082,7 +2498,7 @@ function collect_subcloud_run() send "rm -f /home/${UN}/.ssh/known_hosts\n" exit ${FAIL} } - timeout { exit ${FAIL_TIMEOUT} } + timeout { exit ${FAIL_TIMEOUT_PW} } } exit { $FAIL } EOF @@ -2125,11 +2541,8 @@ function collect_host_complete_local() else if [ ${rc} -eq ${FAIL_INSUFFICIENT_SPACE} ] ; then - report_error "${FAIL_INSUFFICIENT_SPACE_STR}" ${rc} + report_error "failed to collect from ${HOSTNAME}" ${rc} - echo "" - wlog "Increase available space in ${host}:${COLLECT_BASE_DIR} and retry operation." - echo "" remove_dir_local ${COLLECT_DIR} @@ -2137,11 +2550,8 @@ function collect_host_complete_local() elif [ ${rc} -eq ${FAIL_OUT_OF_SPACE} ] ; then - report_error "${FAIL_OUT_OF_SPACE_STR}" ${rc} + report_error "failed to collect from ${HOSTNAME}" ${rc} - echo "" - wlog "Increase available space in ${host}:${COLLECT_BASE_DIR} and retry operation." - echo "" # Remove the corrupt file and exit remove_file_local ${COLLECT_ERROR_LOG} @@ -2179,7 +2589,7 @@ function collect_host_complete_remote () local host="${1}" local tarname="${2}" - if [ "${SUBCLOUD_COLLECT}" == true ] ; then + if [ "${SUBCLOUD_COLLECT}" = true ] ; then SUFFIX="tar" else SUFFIX="tgz" @@ -2188,7 +2598,7 @@ function collect_host_complete_remote () local rc=${?} if [ ${rc} -eq ${PASS} ] ; then - if [ "${SUBCLOUD_COLLECT}" == true ] ; then + if [ "${SUBCLOUD_COLLECT}" = true ] ; then # login to subclouds does not show the subcloud name # in the login prompt. It will always be one of the controllers # so set login prompt to SUBCLOUD_LOGIN_PROMPT @@ -2205,7 +2615,7 @@ function collect_host_complete_remote () rc=${PASS} fi else - wlog "failed to collect from ${host} (reason:${rc}:get file:${COLLECT_DIR}:${tarname}.${SUFFIX})" + report_error "failed to fetch ${tarname}.${SUFFIX} from ${host}" ${rc} dlog "get_file_from_host failure: ${host}:${tarname}.${SUFFIX}:${COLLECT_DIR}" fi return ${rc} @@ -2382,14 +2792,6 @@ collect_host_ctrl_list_index_print() if [ "${DEBUG}" = true ] ; then HOST=${collect_host_ctrl_list[${index}]} info=(${HOST//:/ }) - printf "%s Debug: %-12s %7s [%6s] | Secs:%3s | %3s | %s\n" \ - "$(date)" \ - "${info[${INDEX_HOST}]}" \ - "${info[${INDEX_STAGE}]}" \ - "${info[${INDEX_PID}]}" \ - "${info[${INDEX_SECONDS}]}" \ - "${info[${INDEX_STATUS}]}" \ - "${info[${INDEX_TARBALL}]}" dlog "${info[${INDEX_HOST}]} ${info[${INDEX_STAGE}]} [${info[${INDEX_PID}]}] | Secs:${info[${INDEX_SECONDS}]} | ${info[${INDEX_STATUS}]} | ${info[${INDEX_TARBALL}]}" fi } @@ -2444,7 +2846,6 @@ function collect_subcloud_clean() check_host_reachable "${subcloud}" if [ ${?} -ne ${PASS} ] ; then - wlog "cannot clean unreachable subcloud ${subcloud}" return ${FAIL_UNREACHABLE} fi @@ -2469,6 +2870,7 @@ function collect_subcloud_clean() report_error "failed to copy '${TEMPFILE}' to ${subcloud}/tmp" ${FAIL_FILE_COPY} collect_exit ${FAIL_FILE_COPY} fi + collect_cmd+=("-f ${TEMPFILE}") if [ "${DEBUG}" = true ] ; then @@ -2480,8 +2882,9 @@ function collect_subcloud_clean() # echo "Subcloud Collect Clean: ${subcloud} ${collect_cmd[@]}" -/usr/bin/expect << EOF +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 trap exit {SIGINT SIGTERM} + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_clean_${subcloud}_${FUNCNAME[0]} } log_user ${USER_LOG_MODE} spawn bash -i set timeout ${SSH_TIMEOUT} @@ -2492,8 +2895,8 @@ function collect_subcloud_clean() send "${pw}\r" expect { "${pw_error}" { exit ${FAIL_PASSWORD} } - "${ac_error}" { exit ${FAIL_PERMISSION_SKIP}} - timeout { exit ${FAIL_TIMEOUT3} } + "${ac_error}" { exit ${FAIL_PERMISSION_REMOTE}} + timeout { exit ${FAIL_TIMEOUT_SSH} } "${SUBCLOUD_LOGIN_PROMPT}" {} "${subcloud}:" {} } @@ -2524,7 +2927,7 @@ function collect_subcloud_clean() exit ${FAIL} } timeout { - exit ${FAIL_TIMEOUT5} + exit ${FAIL_TIMEOUT_SUBCLOUD} } } } @@ -2542,7 +2945,7 @@ function collect_subcloud_clean() send "rm -f /home/${UN}/.ssh/known_hosts\n" exit ${FAIL} } - timeout { exit ${FAIL_TIMEOUT} } + timeout { exit ${FAIL_TIMEOUT_PW} } } exit { $FAIL } EOF @@ -2597,6 +3000,16 @@ fi # ############################################################################ +############################################################################ +# +# Create the local collect dir where the tarball(s) will temporarily stored +# +# Note: Catches the password error case +# +############################################################################ + +create_collect_dir_local "${COLLECT_DIR}" + declare COLLECT_START_TIME=${SECONDS} if [ "${SUBCLOUD_COLLECT}" = true ] ; then @@ -2641,7 +3054,7 @@ function collect_hosts() { dlog "collect_hosts: [${HOSTS}] ${HOSTLIST[@]}" let NEXT_PROGRESS_TIME=${SECONDS}+${PROGRESS_INTERVAL} - + local report_collected_hosts=false for host in "${HOSTLIST[@]}" ; do if [ "${host}" != " " ] ; then @@ -2653,7 +3066,6 @@ function collect_hosts() check_host_reachable "${host}" if [ ${?} -ne ${PASS} ] ; then - wlog "cannot collect from ${host} (reason:${FAIL_UNREACHABLE}:${FAIL_UNREACHABLE_STR})" continue fi @@ -2709,11 +3121,10 @@ function collect_hosts() secs=$((SECONDS-HOST_START_TIME)) echo -n "done" echo_stats $secs "${TARNAME}" "${COLLECT_DIR}/${TARNAME}.tgz" + COLLECTED_COUNT=$((COLLECTED_COUNT+1)) fi - elif [ ${rc} -ge ${FAIL_TIMEOUT} -a ${rc} -le ${FAIL_TIMEOUT9} -a "${DCROLE}" == "${DCROLE_SUBCLOUD}" ] ; then - wlog "failed to collect from ${host} (reason:${FAIL_SUBCLOUD_TIMEOUT}:subcloud host run timeout)" else - wlog "failed to collect from ${host} (reason:${rc}:host)" + report_error "failed to collect from ${host}" ${rc} fi fi fi @@ -2726,8 +3137,11 @@ function collect_hosts() ############################################# monitoring=false if [ "${PARALLEL_COLLECT_MODE}" = true ] ; then - - echo -n "monitoring host collect ; please standby " + if [ "${DEBUG}" = true ] ; then + echo "monitoring host collect ; please standby " + else + echo -n "monitoring host collect ; please standby " + fi PLEASE_STANDBY=true # All hosts collected overall timeout @@ -2761,11 +3175,12 @@ function collect_hosts() collect_host_done ${index} ${rc} if [ ${rc} -eq ${PASS} ] ; then collect_host_stats ${index} ${rc} + COLLECTED_COUNT=$((COLLECTED_COUNT+1)) fi DONE_COUNT=$((DONE_COUNT+1)) else collect_host_done ${index} ${rc} - wlog "failed to collect from ${info[${INDEX_HOST}]} (reason:${rc}:target)" + report_error "failed to collect from ${info[${INDEX_HOST}]}" ${rc} fi else if [ ${DONE_COUNT} -eq 0 ] ; then @@ -2787,15 +3202,22 @@ function collect_hosts() done if [ "${monitoring}" = false ] ; then - ilog "collected from ${DONE_COUNT} hosts" + report_collected_hosts=true break fi done + else + report_collected_hosts=true fi # Report that the overall collect timed-out if [ "$monitoring" = true ]; then - report_error "collect operation timeout after $TIMEOUT secs" ${FAIL_TIMEOUT} + report_error "collect operation timeout after $TIMEOUT secs" ${FAIL_TIMEOUT_GLOBAL} + fi + if [ "${report_collected_hosts}" = true ] ; then + if [ ${COLLECTED_COUNT} -ne 0 ] ; then + ilog "collected from ${COLLECTED_COUNT} host$( [ ${COLLECTED_COUNT} -gt 1 ] && echo 's')" + fi fi } @@ -2828,7 +3250,7 @@ collect_subclouds() { dlog "collect_subclouds: [${SUBCLOUDS}] ${SUBCLOUDLIST[@]}" let NEXT_PROGRESS_TIME=${SECONDS}+${PROGRESS_INTERVAL} - + local report_collected_subclouds=false local -a DONE_LIST=() for subcloud in "${SUBCLOUDLIST[@]}" ; do if [ "${subcloud}" != " " ] ; then @@ -2841,7 +3263,6 @@ collect_subclouds() check_host_reachable "${subcloud}" if [ ${?} -ne ${PASS} ] ; then - wlog "cannot collect from ${subcloud} (reason:${FAIL_UNREACHABLE}:${FAIL_UNREACHABLE_STR})" continue fi @@ -2902,6 +3323,7 @@ collect_subclouds() SUFFIX="tgz" fi echo_stats $secs "${COLLECT_NAME}" "${COLLECT_DIR}/${subcloud}_${NOWDATE}.${SUFFIX}" + COLLECTED_COUNT=$((COLLECTED_COUNT+1)) fi else report_error "failed to collect from ${subcloud} [subcloud run]" ${rc} @@ -2953,7 +3375,11 @@ collect_subclouds() if [ "${PARALLEL_COLLECT_MODE}" = true ] ; then - echo -n "monitoring subcloud collect ; please standby " + if [ "${DEBUG}" = true ] ; then + echo "monitoring subcloud collect ; please standby " + else + echo -n "monitoring subcloud collect ; please standby " + fi PLEASE_STANDBY=true # All hosts collected overall timeout @@ -2961,6 +3387,7 @@ collect_subclouds() index=0 monitoring=false for subcloud in "${collect_host_ctrl_list[@]}" ; do + sleep 5 info=(${subcloud//:/ }) # collect_host_ctrl_list_index_print ${index} @@ -2987,6 +3414,7 @@ collect_subclouds() collect_host_done ${index} ${rc} if [ ${rc} -eq ${PASS} ] ; then collect_host_stats ${index} ${rc} + COLLECTED_COUNT=$((COLLECTED_COUNT+1)) fi DONE_COUNT=$((DONE_COUNT+1)) @@ -3019,7 +3447,7 @@ collect_subclouds() fi else collect_host_done ${index} ${rc} - report_error "failed to collect from ${info[${INDEX_HOST}]} [remote]" ${rc} + report_error "failed to collect from ${info[${INDEX_HOST}]}" ${rc} fi else if [ ${DONE_COUNT} -eq 0 ] ; then @@ -3041,17 +3469,24 @@ collect_subclouds() done if [ "${monitoring}" = false ] ; then - ilog "collected from ${DONE_COUNT} subclouds" + report_collected_subclouds=true break fi done + else + report_collected_subclouds=true fi # Report that the overall collect timed-out if [ "$monitoring" = true ]; then if [ "${ORCHESTRATED_COLLECT}" = true ] ; then - report_error "collect operation timeout after $TIMEOUT secs" ${FAIL_SUBCLOUD_TIMEOUT} + report_error "collect operation timeout after $TIMEOUT secs" ${FAIL_TIMEOUT_SUBCLOUD} else - report_error "collect operation timeout after $TIMEOUT secs" ${FAIL_TIMEOUT} + report_error "collect operation timeout after $TIMEOUT secs" ${FAIL_TIMEOUT_GLOBAL} + fi + fi + if [ "${report_collected_subclouds}" = true ] ; then + if [ ${COLLECTED_COUNT} -ne 0 ] ; then + ilog "collected from ${COLLECTED_COUNT} subcloud$( [ ${COLLECTED_COUNT} -gt 1 ] && echo 's')" fi fi } @@ -3071,7 +3506,8 @@ function get_report_tool() local local_dest="${1}" local local_path="/usr/local/bin/report" - mkdir -p "${local_dest}" + create_dir_local "${local_dest}" + chown_file_or_dir_local $(whoami) "${local_dest}" cp -a "${local_path}" "${local_dest}" local rc=${?} @@ -3095,7 +3531,8 @@ function get_report_plugins() local local_dest="${1}" local local_path="/etc/collect/plugins" - mkdir -p "${local_dest}" + create_dir_local "${local_dest}" + chown_file_or_dir_local $(whoami) "${local_dest}" cp -a "${local_path}" "${local_dest}" local rc=${?} @@ -3116,6 +3553,7 @@ declare -i HOSTS_TIMEOUT_BOOST=10 declare -i MAX_LIST_PRINT=6 if [ "${SUBCLOUD_COLLECT}" = true ] ; then + plural=$( [ ${SUBCLOUDS} -gt 1 ] && echo 's') if [ ${SUBCLOUDS} -eq 0 ] ; then report_error "no valid subclouds to collect" ${FAIL_NO_SUBCLOUDS} collect_exit ${FAIL_NO_SUBCLOUDS} @@ -3127,14 +3565,14 @@ if [ "${SUBCLOUD_COLLECT}" = true ] ; then fi if [ "${ALLHOSTS}" = true ] ; then if [ ${SUBCLOUDS} -gt ${MAX_LIST_PRINT} ] ; then - ilog "collecting data from all ${SUBCLOUDS} subcloud(s)" + ilog "collecting data from all ${SUBCLOUDS} subclouds" else - ilog "collecting data from ${SUBCLOUDS} subcloud(s)" + ilog "collecting data from ${SUBCLOUDS} subcloud${plural}: ${SUBCLOUDLIST[@]}" fi elif [ ${SUBCLOUDS} -gt ${MAX_LIST_PRINT} ] ; then - ilog "collecting data from ${SUBCLOUDS} subcloud(s)" + ilog "collecting data from ${SUBCLOUDS} subclouds" else - ilog "collecting data from ${SUBCLOUDS} subcloud(s): ${SUBCLOUDLIST[@]}" + ilog "collecting data from ${SUBCLOUDS} subcloud${plural}: ${SUBCLOUDLIST[@]}" fi collect_subclouds "$@" else @@ -3148,15 +3586,16 @@ else ilog "adjusted hosts collect timout from ${TIMEOUT} to ${UNTIL} secs to account for ${HOSTS} hosts" fi if [ "${ALLHOSTS}" = true ] ; then + plural=$( [ ${HOSTS} -gt 1 ] && echo 's') if [ ${HOSTS} -gt ${MAX_LIST_PRINT} ] ; then - ilog "collecting data from all ${HOSTS} host(s)" + ilog "collecting data from all ${HOSTS} hosts" else - ilog "collecting data from ${HOSTS} host(s)" + ilog "collecting data from ${HOSTS} host${plural}: ${HOSTLIST[@]}" fi elif [ ${HOSTS} -gt ${MAX_LIST_PRINT} ] ; then - ilog "collecting data from ${HOSTS} host(s)" + ilog "collecting data from ${HOSTS} hosts" else - ilog "collecting data from ${HOSTS} host(s): ${HOSTLIST[@]}" + ilog "collecting data from ${HOSTS} host${plural}: ${HOSTLIST[@]}" fi collect_hosts "$@" fi @@ -3205,12 +3644,10 @@ if [ "${SUBCLOUD_COLLECT}" = false ] ; then if [ ${?} -eq 0 -a -e ./report ] ; then if [ "${REPORT}" = true ] ; then - echo "" - echo "------------- Collect Report ------------------" + ilog "running collect report on ${COLLECT_DIR}" + # run the report tool ${COLLECT_DIR}/report/report.py -b ${COLLECT_DIR} - echo "-----------------------------------------------" - echo "" # cleanup and tar the report tool and analysis rm -rf ${COLLECT_DIR}/report/plugin_algs/__pycache__ @@ -3232,9 +3669,11 @@ if [ "${SUBCLOUD_COLLECT}" = false ] ; then [ -d ./$d ] && remove_dir_local "./$d" done fi +create_collect_log echo -n "creating ${COLLECT_TYPE} tarball ${TARBALL_NAME} ... " -/usr/bin/expect << EOF +/usr/bin/expect ${expect_debug} << EOF > ${redirect} 2>&1 + if { "${expect_debug}" != "" } { log_file ${EXPECT_LOG_FILE}_${UN}_${HOSTNAME}_create_tarball } log_user ${USER_LOG_MODE} spawn bash -i expect -re $ @@ -3250,7 +3689,7 @@ echo -n "creating ${COLLECT_TYPE} tarball ${TARBALL_NAME} ... " timeout { exit ${FAIL_TIMEOUT1} } } } - timeout { exit ${FAIL_TIMEOUT} } + timeout { exit ${FAIL_TIMEOUT_PW}} } EOF rc=${?} @@ -3264,6 +3703,7 @@ else secs=$((SECONDS-COLLECT_START_TIME)) echo -n "done" echo_stats $secs "stats-only" "${TARBALL_NAME}" + chown_file_or_dir_local ${UN} ${TARBALL_NAME} log "created ${COLLECT_TYPE} tarball ${TARBALL_NAME}" if [ "${ORCHESTRATED_COLLECT}" = true ] ; then diff --git a/tools/collector/debian-scripts/collect_containerization.sh b/tools/collector/debian-scripts/collect_containerization.sh index e41023fb..74cbfa2d 100755 --- a/tools/collector/debian-scripts/collect_containerization.sh +++ b/tools/collector/debian-scripts/collect_containerization.sh @@ -143,7 +143,7 @@ if [ "$nodetype" = "controller" -a "${ACTIVE}" = true ] ; then echo >>${LOGFILE_HELM} # NOTE: helm environment not configured for root user - CMD="sudo -u sysadmin KUBECONFIG=${KUBECONFIG} helm list --all --all-namespaces" + CMD="sudo -u $(whoami) KUBECONFIG=${KUBECONFIG} helm list --all --all-namespaces" delimiter ${LOGFILE_HELM} "${CMD}" ${CMD} 2>>${COLLECT_ERROR_LOG} >>${LOGFILE_HELM} @@ -152,16 +152,16 @@ if [ "$nodetype" = "controller" -a "${ACTIVE}" = true ] ; then for RELEASE in "${RELEASES[@]:1}"; do NAME=$(echo ${RELEASE} | awk '{print $1}') NAMESPACE=$(echo ${RELEASE} | awk '{print $2}') - CMD="sudo -u sysadmin KUBECONFIG=${KUBECONFIG} helm history -n ${NAMESPACE} ${NAME}" + CMD="sudo -u $(whoami) KUBECONFIG=${KUBECONFIG} helm history -n ${NAMESPACE} ${NAME}" delimiter ${HELM_DIR}/helm-history.info "${CMD}" ${CMD} >> ${HELM_DIR}/helm-history.info 2>>${COLLECT_ERROR_LOG} done - CMD="sudo -u sysadmin KUBECONFIG=${KUBECONFIG} helm search repo" + CMD="sudo -u $(whoami) KUBECONFIG=${KUBECONFIG} helm search repo" delimiter ${LOGFILE_HELM} "${CMD}" ${CMD} 2>>${COLLECT_ERROR_LOG} >>${LOGFILE_HELM} - CMD="sudo -u sysadmin KUBECONFIG=${KUBECONFIG} helm repo list" + CMD="sudo -u $(whoami) KUBECONFIG=${KUBECONFIG} helm repo list" delimiter ${LOGFILE_HELM} "${CMD}" ${CMD} 2>>${COLLECT_ERROR_LOG} >>${LOGFILE_HELM} diff --git a/tools/collector/debian-scripts/collect_host b/tools/collector/debian-scripts/collect_host index a06d7b4b..ef68ecee 100755 --- a/tools/collector/debian-scripts/collect_host +++ b/tools/collector/debian-scripts/collect_host @@ -1,7 +1,7 @@ #! /bin/bash ######################################################################## # -# Copyright (c) 2016-2024 Wind River Systems, Inc. +# Copyright (c) 2016-2022, 2024 Wind River Systems, Inc. # # SPDX-License-Identifier: Apache-2.0 # @@ -36,6 +36,7 @@ COLLECT_NAME="${1}" DEBUG=${8} INVENTORY=${9} set_debug_mode ${DEBUG} +expect_debug=$([ "${DEBUG}" = true ] && echo "-d" || echo "") # Calling parms # @@ -478,7 +479,7 @@ if [ "${OMIT_CERTS}" != "true" ]; then dlog "running /usr/local/sbin/collect_certificates ${EXTRA_DIR}" COLLECT_ERROR_LOG="$COLLECT_ERROR_LOG" \ /usr/local/sbin/collect_certificates ${EXTRA_DIR} - log_space "after collecting certificates :" + log_space "after certificates ..:" fi (cd ${COLLECT_BASE_DIR} ; ${IONICE_CMD} ${NICE_CMD} ${TAR_ZIP_CMD} ${COLLECT_NAME_DIR}.tgz ${COLLECT_NAME} 2>/dev/null 1>/dev/null ) @@ -489,12 +490,9 @@ mkdir -p ${COLLECT_NAME_DIR}/${FLIGHT_RECORDER_PATH} (cd /${FLIGHT_RECORDER_PATH} ; ${TAR_ZIP_CMD} ${COLLECT_NAME_DIR}/${FLIGHT_RECORDER_PATH}/${FLIGHT_RECORDER_FILE}.tgz ./${FLIGHT_RECORDER_FILE} 2>>${COLLECT_ERROR_LOG} 1>>${COLLECT_ERROR_LOG}) -# Pull in an updated user.log which contains the most recent collect logs -# ... be sure to exclude any out of space logs -tail -30 /var/log/user.log | grep "COLLECT:" | grep -v "${FAIL_OUT_OF_SPACE_STR}" >> ${COLLECT_ERROR_LOG} -cp -a ${COLLECT_LOG} ${COLLECT_LOG}.last -cp -a ${COLLECT_ERROR_LOG} ${COLLECT_LOG} -cp -a ${COLLECT_LOG} ${COLLECT_NAME_DIR}/var/log + +# save the collect.log file to this host's tarball +cp -a ${COLLECT_ERROR_LOG} ${COLLECT_NAME_DIR}/${COLLECT_LOG} log_space "with flight data ....:" @@ -509,15 +507,22 @@ log_space "after cleanup .......:" # Check for collect errors # Only out of space error is enough to fail this hosts's collect -collect_errors ${HOSTNAME} +collect_errors "${HOSTNAME}" RC=${?} rm -f ${COLLECT_ERROR_LOG} if [ ${RC} -ne 0 ] ; then rm -f ${COLLECT_NAME_DIR}.tgz - ilog "${FAIL_OUT_OF_SPACE_STR} ${COLLECT_BASE_DIR}" + if [ "${REMOTE_HOST}" = true ] ; then + ilog "${FAIL_OUT_OF_SPACE_REMOTE_STR} ${COLLECT_BASE_DIR}" + else + ilog "${FAIL_OUT_OF_SPACE_STR} ${COLLECT_BASE_DIR}" + fi else ilog "collect of ${COLLECT_NAME_DIR}.tgz succeeded" echo "${collect_done}" fi + +dlog "collect_host exit code: ${rc}" +exit ${rc} \ No newline at end of file diff --git a/tools/collector/debian-scripts/collect_utils b/tools/collector/debian-scripts/collect_utils index d2c8da07..b6ba13ba 100755 --- a/tools/collector/debian-scripts/collect_utils +++ b/tools/collector/debian-scripts/collect_utils @@ -1,6 +1,6 @@ #! /bin/bash # -# Copyright (c) 2013-2019 Wind River Systems, Inc. +# Copyright (c) 2013-2019, 2024 Wind River Systems, Inc. # # SPDX-License-Identifier: Apache-2.0 # @@ -8,6 +8,7 @@ ########################################################################################## DEBUG=false +redirect="/dev/null" # Fail Codes PASS=0 @@ -16,18 +17,17 @@ RETRY=2 FAIL_NODETYPE=3 -FAIL_TIMEOUT=10 -FAIL_TIMEOUT1=11 -FAIL_TIMEOUT2=12 -FAIL_TIMEOUT3=13 -FAIL_TIMEOUT4=14 -FAIL_TIMEOUT5=15 -FAIL_TIMEOUT6=16 -FAIL_TIMEOUT7=17 -FAIL_TIMEOUT8=18 -FAIL_TIMEOUT9=19 - -FAIL_SUBCLOUD_TIMEOUT=20 +FAIL_TIMEOUT_GLOBAL=10 +FAIL_TIMEOUT_OPERATION=11 +FAIL_TIMEOUT_OPERATION_SCP=12 +FAIL_TIMEOUT_OPERATION_SSH=13 +FAIL_TIMEOUT_HOST_ACCESS=14 +FAIL_TIMEOUT_HOST=15 +FAIL_TIMEOUT_PW=16 +FAIL_TIMEOUT_SCP=17 +FAIL_TIMEOUT_SSH=18 +FAIL_TIMEOUT_SUBCLOUD_ACCESS=19 +FAIL_TIMEOUT_SUBCLOUD=20 FAIL_PASSWORD=30 FAIL_PERMISSION=31 @@ -35,7 +35,7 @@ FAIL_CLEANUP=32 FAIL_UNREACHABLE=33 FAIL_HOSTNAME=34 FAIL_INACTIVE=35 -FAIL_PERMISSION_SKIP=36 +FAIL_PERMISSION_REMOTE=36 FAIL_OUT_OF_SPACE=37 FAIL_INSUFFICIENT_SPACE=38 FAIL_INTERNAL=39 @@ -59,6 +59,13 @@ FAIL_INVALID_START_DATE=56 FAIL_INVALID_END_DATE=57 FAIL_INVALID_DATE_RANGE=58 FAIL_TIMEOUT_ARG=59 +FAIL_NOT_SUDOER=60 +FAIL_NOT_SUDOER_REMOTE=61 +FAIL_PASSWORDLESS=65 +FAIL_PASSWORDLESS_REMOTE=66 +FAIL_INSUFFICIENT_SPACE_REMOTE=70 +FAIL_NOT_ENOUGH_SPACE_REMOTE=71 +FAIL_OUT_OF_SPACE_REMOTE=72 # Warnings are above 200 WARN_WARNING=200 @@ -69,23 +76,48 @@ COLLECT_ERROR="Error:" COLLECT_DEBUG="Debug:" COLLECT_WARN="Warning:" +# common permission error strings +pw_error="orry, try again" +ac_error="ermission denied" +su_error="not in the sudoers" + # Failure Strings FAIL_NOT_ENOUGH_SPACE_STR="Not enough /scratch filesystem space" +FAIL_NOT_ENOUGH_SPACE_REMOTE_STR="Not enough remote /scratch filesystem space" FAIL_OUT_OF_SPACE_STR="No space left on device" +FAIL_OUT_OF_SPACE_REMOTE_STR="No space left on remote device" FAIL_TAR_OUT_OF_SPACE_STR="tar: Error is not recoverable" FAIL_INSUFFICIENT_SPACE_STR="Not enough space on device" -FAIL_UNREACHABLE_STR="Unreachable" +FAIL_INSUFFICIENT_SPACE_REMOTE_STR="Not enough space on remote device" FAIL_TIMEOUT_STR="operation timeout" FAIL_TIMEOUT_ARG_STR="out-of-range timeout" -FAIL_SUBCLOUD_TIMEOUT_STR="subcloud collect timeout" +# Operational timeouts +FAIL_TIMEOUT_GLOBAL_STR="global collect timeout" +FAIL_TIMEOUT_PW_STR="password prompt timeout" +FAIL_TIMEOUT_SCP_STR="scp timeout" +FAIL_TIMEOUT_SSH_STR="ssh timeout" +FAIL_TIMEOUT_OPERATION_STR="linux operation timeout" +FAIL_TIMEOUT_OPERATION_SSH_STR="ssh operation timeout" +FAIL_TIMEOUT_OPERATION_SCP_STR="linux operation timeout" + +# host and subcloud timeouts +FAIL_TIMEOUT_HOST_ACCESS_STR="host access timeout" +FAIL_TIMEOUT_HOST_STR="host collect timeout" +FAIL_TIMEOUT_SUBCLOUD_ACCESS_STR="subcloud access timeout" +FAIL_TIMEOUT_SUBCLOUD_STR="subcloud collect timeout" FAIL_NO_FILE_SPECIFIED_STR="no file specified" FAIL_FILE_NOT_FOUND_STR="no such file or directory" FAIL_FILE_EMPTY_STR="file is empty" FAIL_PASSWORD_PROMPT_STR="password for" - +FAIL_PASSWORDLESS_STR="timeout waiting for password prompt" +FAIL_PASSWORDLESS_REMOTE_STR="timeout waiting for remote password prompt" +FAIL_NOT_SUDOER_STR="collect requires sudo on host" +FAIL_NOT_SUDOER_REMOTE_STR="collect requires sudo on remote host" +FAIL_INVALID_PASSWORD_STR="invalid password" +FAIL_PERMISSION_STR="permission error" FAIL_DATE_FORMAT_STR="date format" FAIL_INACTIVE_STR="not active" FAIL_NO_HOSTS_STR="empty host list" @@ -93,6 +125,9 @@ FAIL_NO_SUBCLOUDS_STR="empty subcloud list" FAIL_MISSING_PARAMETER_STR="missing parameter" FAIL_FILE_COPY_STR="failed to copy" FAIL_CONTINUE_STR="cannot continue" +FAIL_UNREACHABLE_STR="Unreachable" +FAIL_PERMISSION_REMOTE_STR="remote permission error" +FAIL_UNSPECIFIED_CAUSE_STR="unspecified cause" # The minimum amount of % free space on /scratch to allow collect to proceed MIN_PERCENT_SPACE_REQUIRED=75 @@ -104,9 +139,9 @@ MIN_PERCENT_SPACE_REQUIRED=75 declare -i COLLECT_BASE_DIR_FULL_THRESHOLD=2147484 # 2Gib in K blocks rounded up # Log file path/names -COLLECT_LOG=/var/log/collect.log -COLLECT_ERROR_LOG=/tmp/collect_error.log -HOST_COLLECT_ERROR_LOG="/tmp/host_collect_error.log" +COLLECT_LOG=collect.log +COLLECT_ERROR_LOG=/tmp/$(whoami)_collect_error.log +HOST_COLLECT_ERROR_LOG="/tmp/$(whoami)_host_collect_error.log" DCROLE_SYSTEMCONTROLLER="systemcontroller" DCROLE_SUBCLOUD="subcloud" @@ -132,7 +167,7 @@ function source_openrc_if_needed fi if [ "${nodetype}" != "controller" -a "${nodetype}" != "worker" -a "${nodetype}" != "storage" ] ; then - logger -t ${COLLECT_TAG} "could not identify nodetype ($nodetype)" + wlog "could not identify nodetype ($nodetype)" exit $FAIL_NODETYPE fi @@ -150,13 +185,13 @@ function source_openrc_if_needed fi } - # Setup an expect command completion file. # This is used to force serialization of expect # sequences and highlight command completion collect_done="collect done" cmd_done_sig="expect done" cmd_done_file="/usr/local/sbin/expect_done" +EXPECT_LOG_FILE="/tmp/collect_expect" # Compression Commands TAR_ZIP_CMD="tar -cvzf" @@ -181,25 +216,25 @@ BUILD_INFO_CMD="cat /etc/build.info" ################################################################################ function log { - logger -t ${COLLECT_TAG} $@ + logger -t ${COLLECT_TAG} "$(whoami) $@" } function ilog { echo "$@" - logger -t ${COLLECT_TAG} $@ + logger -t ${COLLECT_TAG} "$(whoami) $@" } function elog { echo "${COLLECT_ERROR} $@" - logger -t ${COLLECT_TAG} "${COLLECT_ERROR} $@" + logger -t ${COLLECT_TAG} "$(whoami) ${COLLECT_ERROR} $@" } function wlog { echo "${COLLECT_WARN} $@" - logger -t ${COLLECT_TAG} "${COLLECT_WARN} $@" + logger -t ${COLLECT_TAG} "$(whoami) ${COLLECT_WARN} $@" } function set_debug_mode() @@ -210,7 +245,7 @@ function set_debug_mode() function dlog() { if [ "$DEBUG" == true ] ; then - logger -t ${COLLECT_TAG} "${COLLECT_DEBUG} $@" + logger -t ${COLLECT_TAG} $(whoami) "${COLLECT_DEBUG} $@" echo "$(date) ${COLLECT_DEBUG} $@" fi } @@ -271,8 +306,8 @@ listOfOutOfSpaceErrors=( function collect_errors() { - local host=${1} - local RC=0 + local host="${1}" + local rc=0 if [ -e "${COLLECT_ERROR_LOG}" ] ; then @@ -281,24 +316,19 @@ function collect_errors() while [ "x${listOfOutOfSpaceErrors[index]}" != "x" ] ; do grep -q "${listOfOutOfSpaceErrors[index]}" ${COLLECT_ERROR_LOG} if [ "$?" == "0" ] ; then - - string="failed to collect from ${host} (reason:${FAIL_OUT_OF_SPACE}:${FAIL_OUT_OF_SPACE_STR})" - - # /var/log/user.log it - logger -t ${COLLECT_TAG} "${string}" - - # logs that show up in the foreground - echo "${string}" - echo "Increase available space in ${host}:${COLLECT_BASE_DIR} and retry operation." - + wlog "Out of space error(s) found in ${host}:${COLLECT_ERROR_LOG}" + if [ "${REMOTE_HOST}" = false ] ; then + rc=${FAIL_OUT_OF_SPACE} + else + rc=${FAIL_OUT_OF_SPACE_REMOTE} + fi # return error code - RC=1 break fi index=$(($index+1)) done fi - return ${RC} + return ${rc} } ############################################################################ @@ -320,11 +350,13 @@ function space_precheck() size=`echo ${space1} | cut -f 1 -d '%'` if [ ${size} -ge 0 -a ${size} -le 100 ] ; then if [ ${size} -ge ${MIN_PERCENT_SPACE_REQUIRED} ] ; then - ilog "${COLLECT_BASE_DIR} is $size% full" - echo "${FAIL_INSUFFICIENT_SPACE_STR}" - wlog "${HOSTNAME}:${COLLECT_BASE_DIR} does not have enough available space in to perform collect" - wlog "${HOSTNAME}:${COLLECT_BASE_DIR} must be below ${MIN_PERCENT_SPACE_REQUIRED}% to perform collect" - wlog "Increase available space in ${HOSTNAME}:${COLLECT_BASE_DIR} and retry operation." + if [ "${REMOTE_HOST}" = false ] ; then + elog "${HOSTNAME}:${COLLECT_BASE_DIR} ${FAIL_INSUFFICIENT_SPACE_STR}" + else + wlog "${HOSTNAME}:${COLLECT_BASE_DIR} ${FAIL_INSUFFICIENT_SPACE_REMOTE_STR}" + fi + wlog "${HOSTNAME}:${COLLECT_BASE_DIR} at ${size}% ; usage must be below ${MIN_PERCENT_SPACE_REQUIRED}%" + wlog "Increase available space in ${HOSTNAME}:${COLLECT_BASE_DIR} and retry operation" exit ${FAIL_INSUFFICIENT_SPACE} fi else