starlingx/metal
Code Issues Proposed changes

Add new configuration parameters to crashDumpMgr

Browse Source 
This commmit updates crashDumpMgr in order to add three new parameters
and enhance the existing one.

1. Maximum Files: Added 'max-files' parameter to specify the maximum
   number of saved crash dump files. The default value is 4.
2. Maximum Size: Updated the 'max-size' parameter to support
   the 'unlimited' value. The default value is 5GiB.
3. Maximum Used: Included 'max-used' parameter to limit the maximum
   storage used by saved crash dump files. It supports 'unlimited'
   and has a default value of unlimited.
4. Minimum Available: Implemented 'min-available' parameter, enabling
   the definition of a minimum available storage threshold on the
   crash dump file system. The value is restricted to a minimum of
   1GB and defaults to 10%.

These enhancements refine the crash dump management process and
offer more control over storage usage and crash dump file retention.

Story: 2010893
Task: 48676

Test Plan:
1) max-files parameter:
  PASS: don't set max-files param. Ensure the default value is used.
  Create 5 directories inside /var/crash. Each of them contains
  dmesg.<date> and dump.<date>. run the crashDumpMgr script.
  Verify:
    PASS: the vmcore_first.tar.1.gz is created when the first
          directory is read.
    PASS: 4 more vmcore_<date>.tar files are created.
    PASS: There will be 1 vmcore_first.tar.1.gz and 4
          vmcore_<date>.tar inside /var/log/crash.
    PASS: There will be one summary file for each direcory:
          <date>_dmesg.<date> inside /var/crash
2) max-size parameter
  PASS: don't set max-size param. Ensure the default value is used
        (5GiB).
  PASS: Set a fixed max-size param. Create a dump.<date> file greater
        that the max-size param. Run the crashDumpMgr script. Verify
        that the crash dump file is not generated and a log
        message is displayed.
3) max-used parameter:
  PASS: don't set max-used param. Ensure the default value is used
        (unlimited).
  PASS: Set a fixed max-used param. Create a dump.<date> file that
        will generate that the used space is greater that the
        max-used param. Run the crashDumpMgr script. Verify that
        the crash dump file is not generated, a log message is
        displayed and the directory is deleted.
4) min-available parameter:
  PASS: don't set min-available param. Ensure the default value is
        used (10% of /var/log/crash).
  PASS: Set a fixed 'min-available' param. Generate a 'dump.<date>'
        file to simulate a situation where the remaining space is
        less than the 'min-available' parameter. Run the crashDumpMgr
        script and ensure that it does not create the crashdump file,
        displays a log message, and deletes the entry.
5) PASS: Since the crashDumpMgr.service file is not being modified,
         verify that the script takes the default values.

Note: All tests have also been conducted by generating a kernel panic
and ensuring the crashDumpMgr script follows the correct workflow.

Change-Id: I8948593469dae01f190fd1ea21da3d0852bd7814
Signed-off-by: Enzo Candotti <enzo.candotti@windriver.com>
This commit is contained in:
Enzo Candotti 2023-08-25 17:52:55 -03:00
parent 005544b651
commit a120cc5fea
1 changed files with 154 additions and 50 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

204
mtce/src/scripts/crashDumpMgr
View File

@ -1,18 +1,21 @@
#!/bin/bash
#
# Copyright (c) 2022 Wind River Systems, Inc.
# Copyright (c) 2020-2023 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
# Modify it that is to support the debian coredump file.
# coredump files are dmesg.202206101633 and dump.202206101633 in Debian.
CRASHDUMPMGR_TAG=${CRASHDUMPMGR_TAG:-"crashDumpMgr"}
RETVAL=0
max_size=3221225472 # "3GiB"
min_remainder=1073741824 # "1GiB"
# Default values and constants
DEFAULT_MAX_SIZE=5368709120 # "5GiB"
DEFAULT_MAX_FILES=4
UNLIMITED="unlimited"
DEFAULT_MAX_USED="${UNLIMITED}" # Assign UNLIMITED to DEFAULT_MAX_USED
DEFAULT_MIN_REMAINDER_PERCENT=10
MIN_REMAINDER_MINIMUM=1073741824 # 1GiB in bytes
# number format to/from human readable commands.
NUMFMT_TO_HR="/usr/bin/numfmt --to=iec"
@ -38,51 +41,81 @@ function log()
# bundles are quite large and, if too many occur,
# can fill up its target filesystem.
#
# This function nicely tars a crash bundle found in /var/crash
# to /var/log/crash.
# This function manages the crash dump bundles, creating tar archives for
# storage, handling maximum file count and storage limits, and preserving
# summaries of the crash dumps.
#
# The first bundle is tar'ed as vmcore_first.tar and preserved.
# Subsequent crash bundles are nicely tar'ed as vmcore.tar
# Subsequent crash bundles are nicely tar'ed as vmcore_<date>.tar.
#
# Save the crash dump vmcore summary for all crash dumps.
#
# Assumptions: logration is used to compress these bundles in the background
# Assumptions: log rotation is used to compress these bundles in the background
#
# Parameters : $1 = max_size ; maximum vmcore size to keep
# Parameters:
# $1 = max_size ; maximum vmcore size to keep (human-readable size or "unlimited")
# $2 = max_files ; maximum number of crash dump files to keep
# $3 = max_used ; maximum used storage size (human-readable size or "unlimited")
# $4 = min_available ; minimum available storage size (human-readable size)
#
############################################################################
function manage_crash_dumps()
{
if [ "${1}" != "" ] ; then
max_size=${1}
log "max_size=$max_size"
fi
CRASH_DIR="/var/crash"
CRASH_BUNDLE_DIR="/var/log/crash"
OTHER_BUNDLE="${CRASH_BUNDLE_DIR}/vmcore.tar"
OTHER_BUNDLE="${CRASH_BUNDLE_DIR}/vmcore"
FIRST_BUNDLE="${CRASH_BUNDLE_DIR}/vmcore_first.tar"
FIRST_BUNDLE_ROTATED="${CRASH_BUNDLE_DIR}/vmcore_first.tar.1.gz"
CRASH_BUNDLE_SUMMARY="vmcore-dmesg.txt"
CRASH_BUNDLE_SUMMARY_DEB="dmesg."
if [ "${4}" = "" ] ; then
# Get the size of the filesystem assigned to /var/log/crash in bytes
fs_size=$(df -B1 ${CRASH_BUNDLE_DIR} | awk 'NR==2 {print $2}')
# Calculate min_available as the percentage of the filesystem size
min_available=$((${fs_size} * ${DEFAULT_MIN_REMAINDER_PERCENT} / 100))
else
min_available=${4}
fi
# Set a minimum value for min_available
if [ ${min_available} -lt ${MIN_REMAINDER_MINIMUM} ]; then
min_available=${MIN_REMAINDER_MINIMUM}
fi
max_size_hr=""
max_used_hr=""
if [ "${max_size}" != "${UNLIMITED}" ]; then
max_size_hr="($(${NUMFMT_TO_HR} ${max_size}))"
fi
if [ "${max_used}" != "${UNLIMITED}" ]; then
max_used_hr="($(${NUMFMT_TO_HR} ${max_used}))"
fi
min_available_hr=$(${NUMFMT_TO_HR} ${min_available})
log "max crash dump files set to ${max_files}"
log "max crash dump vmcore size is ${max_size} ${max_size_hr}"
log "max used storage size is ${max_used} ${max_used_hr}"
log "minimum available storage size is ${min_available} (${min_available_hr})"
# tar command and nice levels
TAR_CMD="tar -cf"
NICE_CMD="/usr/bin/nice -n19"
IONICE_CMD="/usr/bin/ionice -c2 -n7"
log "managing ${CRASH_DIR}"
cleanup=false
# create dir if it does not exist
if [ ! -d ${CRASH_BUNDLE_DIR} ] ; then
mkdir ${CRASH_BUNDLE_DIR}
fi
for entry in ${CRASH_DIR}/*
for entry in $(ls -rt ${CRASH_DIR}/);
do
entry=${CRASH_DIR}/${entry}
remove_entry=false
max_files_saved=false
if [ -d "${entry}" ] ; then
time=${entry##*/}
if [ -e "${entry}/${CRASH_BUNDLE_SUMMARY_DEB}${time}" ] ; then
@ -107,10 +140,28 @@ function manage_crash_dumps()
fi
vmcore_size_hr=$(${NUMFMT_TO_HR} ${vmcore_size})
# get available ${CRASH_BUNDLE_DIR} fs space in 1k blocks and convert that to bytes
available=$(($(df -k ${CRASH_BUNDLE_DIR} | grep -v Available | awk '{ print $4 }')*1000))
# Manage max number of files
if [ "$(ls -A ${CRASH_BUNDLE_DIR} | wc -l)" -ge "${max_files}" ]; then
oldest_vmcore=$(ls -t ${CRASH_BUNDLE_DIR} | tail -2 | head -1)
oldest_vmcore_size=$(stat --format='%s' ${oldest_vmcore})
max_files_saved=true
fi
# get available ${CRASH_BUNDLE_DIR} fs space in bytes
available=$(df -B1 ${CRASH_BUNDLE_DIR} | grep -v Available | awk '{ print $4 }')
# get the current used space in the ${CRASH_BUNDLE_DIR} fs
used_space=$(du -sb ${CRASH_BUNDLE_DIR} | awk '{print $1}')
# if the ${CRASH_BUNDLE_DIR} contains the maximum number of files, the available and used_space
# need to be updated to the value after deleting the oldest crash dump file.
if [ "${max_files_saved}" = true ] ; then
available=$((available+oldest_vmcore_size))
used_space=$((used_space-oldest_vmcore_size))
fi
available_hr=$(${NUMFMT_TO_HR} ${available})
log "new vmcore detected (size:${vmcore_size}:${vmcore_size_hr}) ; ${CRASH_BUNDLE_DIR} avail:${available}:${available_hr}"
log "new vmcore detected (size:${vmcore_size}:${vmcore_size_hr}) ;" \
"${CRASH_BUNDLE_DIR} avail:${available}:${available_hr}"
# Don't save this crash dump if it would leave the
# ${CRASH_BUNDLE_DIR} filesystem with less than 1GiB.
@ -120,27 +171,35 @@ function manage_crash_dumps()
remaining=0
fi
if [ "${cleanup}" = true ] ; then
log "... remove ${entry} ; cleanup"
# Check if adding the file would exceed the maximum used space limit
total_used_space=$((used_space + file_size))
if [ "${max_used}" != "$UNLIMITED" ] && [ ${total_used_space} -gt ${max_used} ]; then
log "The last crash dump is not saved because it would exceed the maximum" \
"used space limit specified in the max_used parameter (${max_used} bytes)."
remove_entry=true
# check for min required 'remaining' ${CRASH_BUNDLE_DIR} filesystem space
elif [ ${remaining} -lt ${min_remainder} ] ; then
log "insufficient space in ${CRASH_BUNDLE_DIR} for ${vmcore_size_hr} ${entry}; would leave only ${remaining} bytes"
elif [ ${remaining} -lt ${min_available} ] ; then
log "insufficient space in ${CRASH_BUNDLE_DIR} for ${vmcore_size_hr} ${entry};" \
"would leave only ${remaining} bytes"
remove_entry=true
# create a new crash bundle if the vmcore file isn't oversized
elif [ ${vmcore_size} -lt ${max_size} ] ; then
if [ -e ${FIRST_BUNDLE} -o -e ${FIRST_BUNDLE_ROTATED} ] ; then
if [ ! -e ${OTHER_BUNDLE} ] ; then
log "creating bundle from ${entry}"
${IONICE_CMD} ${NICE_CMD} ${TAR_CMD} ${OTHER_BUNDLE} -C ${CRASH_DIR} $(basename ${entry})
cleanup=true
fi
else
elif [ ${vmcore_size} -lt ${max_size} ] || [ "${max_size}" = "${UNLIMITED}" ] ; then
if [ ! -e "${FIRST_BUNDLE}" ] && [ ! -e "${FIRST_BUNDLE_ROTATED}" ]; then
log "creating first bundle from ${entry}"
${IONICE_CMD} ${NICE_CMD} ${TAR_CMD} ${FIRST_BUNDLE} -C ${CRASH_DIR} $(basename ${entry})
cleanup=true
else
if [ "${max_files_saved}" = true ] ; then
# delete oldest vmcore file
log "removing oldest vmcore file: ${oldest_vmcore}"
rm -rf "${CRASH_BUNDLE_DIR}/${oldest_vmcore}"
fi
log "creating bundle from ${entry}"
${IONICE_CMD} ${NICE_CMD} ${TAR_CMD} ${OTHER_BUNDLE}_${time}.tar -C ${CRASH_DIR} $(basename ${entry})
fi
remove_entry=true
else
@ -166,36 +225,81 @@ function manage_crash_dumps()
done
}
function print_help()
{
echo "$(basename $0) { --max-size <human-readable-size> }"
function print_help() {
echo "Usage: $(basename "$0") [OPTIONS]"
echo "Options:"
echo " --max-size <size> Set maximum vmcore size (human-readable size or \"$UNLIMITED\")"
echo " --max-files <number> Set maximum number of crash dump files"
echo " --max-used <size> Set maximum used storage size (human-readable size or \"$UNLIMITED\")"
echo " --min-available <size> Set minimum available storage size (human-readable size)"
}
normalize_size_format() {
echo "$1" | tr ',' '.'
}
# Initialize default values
max_size="${DEFAULT_MAX_SIZE}"
max_files="${DEFAULT_MAX_FILES}"
max_used="${DEFAULT_MAX_USED}"
min_available=""
# Initialize a flag to indicate if help has been shown
help_shown=false
# Parse the command line
while [[ ${#} -gt 0 ]] ; do
key="${1}"
case $key in
while [[ $# -gt 0 ]]; do
case "${1}" in
-h|--help)
print_help
help_shown=true
shift
;;
--max-size)
max_size=$(echo "$2" | ${NUMFMT_FROM_HR})
log "max crash dump vmcore size is ${2} (${max_size})"
shift
if [ "${1}" = "${UNLIMITED}" ]; then
max_size=${UNLIMITED}
else
max_size=$(${NUMFMT_FROM_HR} "$(normalize_size_format "${1}")")
fi
shift
;;
--max-files)
shift
max_files="${1}"
shift
;;
--max-used)
shift
if [ "${1}" = "${UNLIMITED}" ]; then
max_used=${UNLIMITED}
else
max_used=$(${NUMFMT_FROM_HR} "$(normalize_size_format "${1}")")
fi
shift
;;
--min-available)
shift
min_available=$(${NUMFMT_FROM_HR} "$(normalize_size_format "${1}")")
shift
;;
*)
print_help
if [ "${help_shown}" = false ]; then
print_help
help_shown=true
else
# Handle unknown options or arguments
shift
fi
;;
esac
shift
done
manage_crash_dumps $max_size
manage_crash_dumps ${max_size} ${max_files} ${max_used} ${min_available}
exit $RETVAL