docs/doc/source/security/kubernetes/auditd-support-339a51d8ce16.rst

Linux Auditing System

This section describes the Linux Auditing System containerized solution for . The container-based solution aligns with the modular architecture approach of the product.

The Linux Auditing System helps system administrators track security violation events based on preconfigured audit rules. The events are recorded in a log file and the information in the log entries helps to detect misuse or unauthorized activities. Some examples of auditable events are:

• file or directory access (Such as files/directories that were accessed, modified, executed, or attributes changed)
• system calls (For example, useradd, time-related system calls)
• commands run by a user (For example, a rule can be defined for every executable in the /bin directory and tracked per user.)
• security events, such as failed login attempts
• network access (The iptables and ebtables utilities can be configured to trigger audit events.)

The Linux Audit daemon, auditd, is the main component of the Linux Auditing System, and is responsible for writing the audit logs. For more information on auditd daemon configuration, see https://man7.org/linux/man-pages/man8/auditd.8.html.

To run auditd on , you must enable auditd in the kernel of hosts and then upload and apply the auditd system application.

Enable Auditd in the Kernel

The Linux Auditing System is disabled in the kernel by default.

To enable auditd in the kernel of all hosts in the system, set the system service parameter audit to '1' and apply the service-parameter change, using the following commands, executed on the active controller.

~(keystone_admin)]$ system service-parameter-modify platform kernel audit=1
~(keystone_admin)]$ system service-parameter-apply platform

To persist the service parameter change, all hosts need to be locked and unlocked, using the following commands for each host depending on the deployed configuration:

For deployments:

~(keystone_admin)$ system host-lock controller-0
~(keystone_admin)$ system host-unlock controller-0

For and Standards deployments, after controller-1 is locked/unlocked swact controller-0 to make controller-1 the active node. The next set of commands are executed on controller-0 node:

~(keystone_admin)$ system host-lock controller-1
~(keystone_admin)$ system host-unlock controller-1
~(keystone_admin)$ system host-swact controller-0

On controller-1, after controller-0 is locked/unlocked swact controller-1 to go back to controller-0 as the active node. The next set of commands are executed on controller-1 node:

~(keystone_admin)$ system host-lock controller-0
~(keystone_admin)$ system host-unlock controller-0
~(keystone_admin)$ system host-swact controller-1

For each worker node in the configuration execute the commands from controller-0:

~(keystone_admin)$ system host-lock worker-0
~(keystone_admin)$ system host-unlock worker-0

To verify if the grub kernel parameter audit was updated to '1', for a particular host, ssh to the host, and check the cmdline parameters, for example:

~(keystone_admin)]$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-5.10.57-200.185.tis.el7.x86_64 root=UUID=e11d78a2-7e1c-4613-84c7-002647b1cf8d ro security_profile=standard module_blacklist=integrity,ima tboot=false crashkernel=512M biosdevname=0 console=ttyS0,115200 iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=panic,1 softlockup_panic=1 softdog.soft_panic=1 intel_iommu=on user_namespace.enable=1 nopti nospectre_v2 nospectre_v1 hugepagesz=2M hugepages=0 default_hugepagesz=2M irqaffinity=2-3 rcu_nocbs=2-3 kthread_cpus=0-1 audit=1 audit_backlog_limit=8192

Note

Enabling auditd should only be done if the purpose is to start auditd in the container/pod using the process described in Start Auditd System Application <start-auditd-system-application>. Otherwise, there will be unnecessary performance impact and the backlog events queue limit will eventually exceed, causing audit: kauditd hold queue overflow messages to be displayed.

Start Auditd System Application

• Set the audit grub kernel parameter to '1'.
• The auditd container that runs the auditd daemon must be started by uploading and applying the audit-armada-app.

The auditd system application is installed as part of the software install or upgrade.

The auditd system application tarball can be found after installation in the /usr/local/share/applications/helm directory. The name of the tarball is auditd-<version>.tgz, for example, auditd-1.0-2.tgz.

Use the following commands to upload and apply the auditd system application:

~(keystone_admin)]$ system application-upload /usr/local/share/applications/helm/auditd-1.0-2.tgz
# check the app was uploaded

~(keystone_admin)]$ system application-show auditd
# if status is "uploaded" proceed with app apply

~(keystone_admin)]$ system application-apply auditd
# check the app was applied

~(keystone_admin)]$ system application-show auditd
# if successful, status will be "applied"

To check that auditd container/pod is created and running on each master and worker node, use the following command:

~(keystone_admin)]$ kubectl get pods -n kube-system -o wide | grep auditd

ns-auditd-9hgq5   1/1  Running 0 2m46s face::e95d:7b0:368d:55f8   compute-0     <none> <none>
ns-auditd-btww5   1/1  Running 1 2m46s face::2d8f:b75d:d511:81ef  compute-1     <none> <none>
ns-auditd-czsdf   1/1  Running 1 2m46s face::977:4894:111d:5bf0   compute-2     <none> <none>
ns-auditd-hs62t   1/1  Running 0 2m46s face::3                    controller-1  <none> <none>
ns-auditd-nn8jw   1/1  Running 0 2m46s face::2                    controller-0  <none> <none>

Auditd Configuration Overrides

The Auditd daemon specific configuration is available in the /etc/audit/auditd.conf file. For more information, see, https://man7.org/linux/man-pages/man5/auditd.conf.5.html.

Besides the auditd main configuration file auditd.conf, auditd uses audit rules configuration that is available in the /etc/audit/audit.rules file which defines what audit events are logged. For more information on how audit rules are configured, see https://linux.die.net/man/7/audit.rules.

In the containerized auditd solution, both configuration files have default settings that can be overwritten using Helm chart overrides.

The Helm chart overrides are applied using the following command:

~(keystone_admin)]$ system helm-override-update auditd auditd kube-system --reuse-values --values /home/sysadmin/<user_specific_config>.yaml

The <user_specific_config>.yaml defines the overrides that will apply either to the auditd.conf and/or to the audit.rules files.

Note

The default values for auditd.conf should be sufficient and you do not need to update them.

In rare cases, the following example describes how to update the default value with the desired value.

Example of user defined overrides file for auditd.conf:

auditdconf: |-
  ##########################################################################
  #
  # auditd.conf
  #
  ##########################################################################
  local_events = yes
  write_logs = yes
  log_file = /var/log/audit/audit.log
  log_group = root
  log_format = RAW
  flush = INCREMENTAL_ASYNC
  freq = 50
  max_log_file = 8
  num_logs = **8**
  priority_boost = 4
  disp_qos = lossy
  dispatcher = /sbin/audispd
  name_format = NONE
  ##name = mydomain
  max_log_file_action = IGNORE
  space_left = 75
  space_left_action = SYSLOG
  ##verify_email = yes
  ##action_mail_acct = root
  admin_space_left = 50
  admin_space_left_action = SYSLOG
  disk_full_action = SYSLOG
  disk_error_action = SYSLOG
  use_libwrap = yes
  ##tcp_listen_port = 60
  ##tcp_listen_queue = 5
  ##tcp_max_per_addr = 1
  ##tcp_client_ports = 1024-65535
  ##tcp_client_max_idle = 0
  enable_krb5 = no
  krb5_principal = auditd
  distribute_network = no

Example of user defined overrides file for audit.rules:

auditdrules: |-
  ## First rule - delete all
  -D

  ## Increase the buffers to survive stress events.
  ## Make this bigger for busy systems
  -b 8192

  ## Set failure mode to syslog
  -f 1

  -a always,exit -F arch=b32 -S init_module,finit_module,delete_module -F key=modules
  -a always,exit -F arch=b64 -S init_module,finit_module,delete_module -F key=modules

Note

The log rotation configuration in auditd.conf file must not be updated, and must use the default value, max_log_file_action = IGNORE, since the logrotate linux utility is used to manage auditd log rotation.

Apply the audit rules overrides using the following command:

~(keystone_admin)]$ system application-apply auditd

Check that application apply has completed successfully:

~(keystone_admin)]$ system application-show auditd

The Helm chart overrides system helm-override-update command, automatically applies the additional rules from the user provided yaml file to the audit.rules in the auditd container.

Similarly, configuration overrides can be applied to update the default configuration of auditd.conf using the system helm-override-update command.

Auditd logs

auditd logs can be viewed on the host in the /var/log/audit directory. Logs are generated by the auditd daemon running in the container and the logs record auditable events configured using the audit.rules file. Log rotation is automatically configured by the system.

Disable Auditd

You may decide to disable auditd for performance reasons. First, you must remove the auditd application. Then, you must set the kernel service parameter audit to '0'. These steps removes the auditd containers on all hosts and the auditd application.

Use the following system commands to disable auditd:

To remove the auditd application:

~(keystone_admin)]$ system application-remove auditd
~(keystone_admin)]$ system application-delete auditd

To verify that the application does not exist in the system:

~(keystone_admin)]$ system application-list |grep auditd

To set the kernel service parameter audit to '0':

~(keystone_admin)]$ system service-parameter-modify platform kernel audit=0
~(keystone_admin)]$ system service-parameter-apply platform

To persist the kernel parameter change, all hosts need to be locked and unlocked:

~(keystone_admin)]$ system host-lock controller-0
~(keystone_admin)]$ system host-unlock controller-0