AppArmor support for StarlingX
This specification describes the introduction of AppArmor into the StarlingX solution. Story: 2010310 Task: 46355 Signed-off-by: jmishra <jagatguruprasad.mishra@windriver.com> Change-Id: Id76a3da52c148c77bfd78cafbe583b94345e491a
This commit is contained in:
jmishra
parent
f618ca6764
commit
a09b153e2f
@ -0,0 +1,357 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License. http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
========================================
|
||||||
|
StarlingX AppArmor Support
|
||||||
|
========================================
|
||||||
|
|
||||||
|
Storyboard:
|
||||||
|
https://storyboard.openstack.org/#!/story/2010310
|
||||||
|
|
||||||
|
This specification describes the introduction of AppArmor into the
|
||||||
|
StarlingX solution, initially for the purposes of running end user's
|
||||||
|
hosted containers under AppArmor profiles.
|
||||||
|
|
||||||
|
AppArmor is a Mandatory Access Control (MAC) system built on Linux's
|
||||||
|
LSM (Linux Security Modules) interface. In practice, the kernel queries
|
||||||
|
AppArmor before each system call to know whether the process is authorized
|
||||||
|
to do the given operation. Through this mechanism, AppArmor confines programs
|
||||||
|
to a limited set of resources.
|
||||||
|
|
||||||
|
AppArmor applies a set of rules (known as a “profile”) on each program. The
|
||||||
|
profile applied by the kernel depends on the installation path of the program
|
||||||
|
being executed, the rules applied do not depend on the user. All users face
|
||||||
|
the same set of rules when they are executing the same program but traditional
|
||||||
|
user permissions still apply and might result in different behavior.
|
||||||
|
|
||||||
|
AppArmor profiles contain a list of access control rules on resources that each
|
||||||
|
program can make use of. Each profile can be loaded either in enforcing or
|
||||||
|
complaining mode. The former enforces the policy and reports violation attempts,
|
||||||
|
while the latter does not enforce the policy but still logs the system calls
|
||||||
|
that would have been denied.
|
||||||
|
|
||||||
|
AppArmor helps administrators in running a more secure kubernetes deployment by
|
||||||
|
restricting what containers/pods are allowed to do, and/or provide better
|
||||||
|
auditing through system logs.The access needed by a container/pod is configured
|
||||||
|
through profiles tuned to allow the access such as Linux capabilities, network
|
||||||
|
access, file permissions, etc.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
Need a mechanism for restricting application pods run by the end users. AppArmor does
|
||||||
|
this by allowing the system administrator to restrict programs' capabilities
|
||||||
|
with per-program profiles.Then Kubernetes' container.apparmor.security.beta.kubernetes.io
|
||||||
|
annotation can be used in a pod spec to assign an AppArmor profile to a pod.
|
||||||
|
|
||||||
|
Management of AppArmor profiles across all the kubernetes hosts of a StarlingX kubernetes
|
||||||
|
cluster should be simple for end user.
|
||||||
|
|
||||||
|
There are concerns over the system performance degradation when enabling the
|
||||||
|
AppArmor. However, some end users want it enabled for security reasons, and are willing
|
||||||
|
to accept the system performance degradation.
|
||||||
|
|
||||||
|
|
||||||
|
Use Cases
|
||||||
|
---------
|
||||||
|
|
||||||
|
AppArmor can be optionally enabled on a per host basis:
|
||||||
|
|
||||||
|
* Default recommendation will be that AppArmor is enabled on all hosts
|
||||||
|
* However, if required (e.g. to avoid performance impact on some cards), user could choose which host to enable AppArmor profiles (e.g., Enable AppArmor only on controllers in a Standard System) NOTE In this case, a user would need to use labels and node-selector in pod spec to ensure the pod is only scheduled on nodes with AppArmor enabled.
|
||||||
|
|
||||||
|
If AppArmor is enabled on host(s) of a StarlingX system, a user will be able to run selected user's hosted application pods under a selected AppArmor profile by specifying the AppArmor profile in the pod's spec.
|
||||||
|
|
||||||
|
**NOTE** Support for AppArmor profiles on the StarlingX Platform entities (both host processes and containers) themselves, is left to a future release.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
This change provides a way to enable/disable on a particular host.
|
||||||
|
|
||||||
|
* Default recommendation will be that AppArmor is enabled on all hosts
|
||||||
|
* However, if required (e.g. to avoid performance impact on some hosts), user could choose
|
||||||
|
which host to enable AppArmor profiles (e.g., Enable AppArmor only on controllers in a Standard System)
|
||||||
|
**NOTE** In this case, a user would need to use labels and node-selector in pod spec to ensure the pod
|
||||||
|
is only scheduled on nodes with AppArmor enabled.
|
||||||
|
|
||||||
|
In order apply the profiles to a particular pod, the profiles need to be available to the host machine
|
||||||
|
where pod is launched. Security Profile Operator (SPO, https://github.com/kubernetes-sigs/security-profiles-operator)
|
||||||
|
provides AppArmor profile management (i.e. loading/unloading) across k8s nodes. SPO defines an AppArmor Profile CRD, such
|
||||||
|
that end users' can define AppArmor profiles for SPO to manage.
|
||||||
|
|
||||||
|
SPO will be integrated as a system application using FluxCD manifest which utilizes helm chart that
|
||||||
|
deploys a controller/operator for centrally managing the AppArmor Profile resources, a webhook for validation of AppArmor
|
||||||
|
Profile resources and a daemonset for loading/unloading the AppArmor Profiles. The DaemonSet configuration creates an SPO
|
||||||
|
pod/container on each host in the StarlingX cluster.
|
||||||
|
Creating a DaemonSet configuration for AppArmor container ensures that SPO will run on all kubernetes nodes,
|
||||||
|
such as controllers, all-in-one controllers and worker nodes.
|
||||||
|
|
||||||
|
Once an AppArmor profile is loaded to the K8S nodes, it can be applied to a particular pod using annotations.
|
||||||
|
| example: container.apparmor.security.beta.kubernetes.io/<pod_name>:localhost/<profile_ref>. Please refer
|
||||||
|
https://kubernetes.io/docs/tutorials/security/apparmor/#example
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
In order apply the profiles to a particular pod, the profiles need to be available to the host
|
||||||
|
machine where pod is launched. There are several solutions available and below alternatives were evaluated:
|
||||||
|
|
||||||
|
* security-profiles-operator
|
||||||
|
* apparmor-loader
|
||||||
|
* kube_apparmor_Manager
|
||||||
|
* Manually copying and loading profiles on each host (DEV Options)
|
||||||
|
|
||||||
|
Below is a comparision table
|
||||||
|
|
||||||
|
.. list-table:: Alternatives to load AppArmor profiles in K8s cluster
|
||||||
|
:widths: 10 20 30 20 20
|
||||||
|
:header-rows: 1
|
||||||
|
|
||||||
|
* -
|
||||||
|
- AppArmor Loader
|
||||||
|
- Security Profile Operator(SPO)
|
||||||
|
- AppArmor_Kube_Manager
|
||||||
|
- Manually Loading Profiles (DEV options)
|
||||||
|
* - Community Type
|
||||||
|
- K8S
|
||||||
|
- K8S
|
||||||
|
- Private, sysdiglab
|
||||||
|
- NA
|
||||||
|
* - Approach
|
||||||
|
- Daemon Set
|
||||||
|
- Daemon Set
|
||||||
|
- SSH based copy
|
||||||
|
- Manual copy (SSH based)
|
||||||
|
* - Developement Type
|
||||||
|
- POC
|
||||||
|
- Controlled as Alpha, beta release
|
||||||
|
- Not sure
|
||||||
|
- NA
|
||||||
|
* - Active
|
||||||
|
- Last updated 4 years back
|
||||||
|
- Very Much Active
|
||||||
|
- Last updated 2 years back
|
||||||
|
- NA
|
||||||
|
* - Release Frequency
|
||||||
|
- NA
|
||||||
|
- Quarterly basis
|
||||||
|
- NA
|
||||||
|
- NA
|
||||||
|
* - Latest Version
|
||||||
|
- No version control
|
||||||
|
- V0.4.3
|
||||||
|
- V0.0.5
|
||||||
|
- NA
|
||||||
|
* - Last Release Data
|
||||||
|
-
|
||||||
|
- Jun 07, 2022
|
||||||
|
- Apr 04, 2020
|
||||||
|
- NA
|
||||||
|
* - Roadmap
|
||||||
|
- No
|
||||||
|
- Yes, KEP and Roadmap in place
|
||||||
|
- No
|
||||||
|
- NA
|
||||||
|
* - Github Link
|
||||||
|
- https://github.com/kubernetes/kubernetes/tree/master/test/images/apparmor-loader
|
||||||
|
- https://github.com/kubernetes-sigs/security-profiles-operator
|
||||||
|
- https://github.com/sysdiglabs/kube-apparmor-manager
|
||||||
|
- NA
|
||||||
|
* - Functionality
|
||||||
|
- Only Load AppArmor profiles on all worker nodes in k8s cluster
|
||||||
|
- Out-of-tree Kubernetes enhancement | Supports SELinux, seccomp, AppArmor | AppArmor profiles load and unload | Feature enabling and disabling using feature gate | Supplementary features – Metrics , service monitoring, log enricher, | Debugging features- CPU and Memory profile endpoints
|
||||||
|
- Load AppArmor profiles on all worker nodes in k8s cluster | Sync feature to cater profile change synchronize across worker nodes
|
||||||
|
- Copy the profiles on each host using SSH | Load the profile on each host using aa-enforce, aa-complain, or apparmor_parser | Unload the profiles on each host using aa-disable or apparmer_parser
|
||||||
|
|
||||||
|
|
||||||
|
Security Profile Operator (SPO) is the best option available and will continue our effort in incorporating it in StarlingX based on below advantages.
|
||||||
|
|
||||||
|
* Better community support
|
||||||
|
* Feature rich
|
||||||
|
* Roadmap in place
|
||||||
|
|
||||||
|
There are concerns about the stability of AppArmor support in SPO and the basic functionality of deploying AppArmor
|
||||||
|
profiles to various nodes is not working at this point. We will continue our effort to
|
||||||
|
track SPO fixes to apparmor functionality.
|
||||||
|
|
||||||
|
* Retest updates to SPO and provide feedback to contributor quickly
|
||||||
|
* Offer any assistance we can to SPO contributor
|
||||||
|
* Identify any missing/desired functionality (e.g. complain vs enforce)
|
||||||
|
|
||||||
|
For now, we will proceed in parallel with
|
||||||
|
|
||||||
|
* using DEV option of linux AppArmor tools (aa commands, https://gitlab.com/apparmor/apparmor/-/wikis/ManPages) to load apparmor profiles on hosts AND
|
||||||
|
* work closely in testing and providing feedback (and possibly changes) to SPO project
|
||||||
|
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
A host level attribute "apparmor" would be added to the host table.
|
||||||
|
This can contain two possible values "enabled" or "disabled"
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
AppArmor can be enabled or disabled at runtime on a per-host basis.
|
||||||
|
|
||||||
|
* Below cli command(s) would be used to enable or disable AppArmor in kernel for a particular host
|
||||||
|
* system host-lock <hostname>
|
||||||
|
* system host-update <hostname> apparmor=enabled/disabled **Example:** system host-update controller-0 apparmor=enabled
|
||||||
|
* system host-unlock <hostname>
|
||||||
|
|
||||||
|
The above command(s) would enable the AppArmor in kernel with NO services (on host or containerized)
|
||||||
|
in the WRCP/StarlingX Solution running under an apparmor profile
|
||||||
|
**NOTE** will explore whether default profiles enabled in Debian Bullseye can be used as default
|
||||||
|
when AppArmor is enabled
|
||||||
|
|
||||||
|
* Changes in the below API to reflect AppArmor. |
|
||||||
|
GET /v1/ihosts/{host_id} |
|
||||||
|
Description: Shows detailed information about a specific host |
|
||||||
|
INPUT: |
|
||||||
|
host_id : csapi:UUID: part of user-info |
|
||||||
|
Output: |
|
||||||
|
"apparmor": "enabled" or "apparmor": "disabled" would be added to the output json |
|
||||||
|
|
||||||
|
* Addition of a new attribute 'apparmor' to the below API:
|
||||||
|
|
||||||
|
| PATCH /v1/ihosts/{host_id}
|
||||||
|
| Apparmor attribute can be modified. Support would be added for apparmor (xsd:string)
|
||||||
|
| INPUT:
|
||||||
|
| host_id : csapi:UUID: part of user-info
|
||||||
|
| Example: [
|
||||||
|
| {
|
||||||
|
| "path":"/apparmor",
|
||||||
|
| "value":"enabled",
|
||||||
|
| "op":"replace"
|
||||||
|
| }
|
||||||
|
| ]
|
||||||
|
| Output:
|
||||||
|
| "apparmor": "enabled" or "apparmor": "disabled" would be added to the output json
|
||||||
|
| **NOTE**: Changing the 'apparmor' attribute, will ONLY be allowed when host is locked.
|
||||||
|
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
The AppArmor helps in running a more secure kubernetes deployment by restricting what
|
||||||
|
containers/pods are allowed to do. User should be able to run selected user's hosted
|
||||||
|
application pods under a selected AppArmor profile. AppArmor enforces the policy defined
|
||||||
|
in the profile as well as reports policy violation attempts.
|
||||||
|
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
* If apparmor is NOT enabled (disabled by default), then there is NO impact on end user
|
||||||
|
* If apparmor is used,
|
||||||
|
* there is admin user impact to enable apparmor on hosts and load apparmor profiles on host and
|
||||||
|
* there is general user impact that they 'can' run hosted containers under apparmor profiles for enhanced security.
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
AppArmor is known to have a performance impact when running. There will be a performance impact characterization
|
||||||
|
(StarlingX Infrastructure Operations and Datapaths) of enabling AppArmor in the StarlingX which involved getting data with:
|
||||||
|
|
||||||
|
* apparmor disabled
|
||||||
|
* apparmor enabled but NO apparmor profile applied to datapath app pod
|
||||||
|
* apparmor enabled and apparmor profile applied to datapath app pod
|
||||||
|
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Upgrade impact
|
||||||
|
--------------
|
||||||
|
|
||||||
|
A new host level attribute "apparmor" would be added to the host table. There should not be any impact on backup and restore, upgrade, and rollback.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
* Jagatguru Prasad Mishra (jmishra)
|
||||||
|
|
||||||
|
Other contributors:
|
||||||
|
* Sirin Shaikh (sshaikh)
|
||||||
|
* Kirti Singh (ksingh)
|
||||||
|
* Rahul Roshan Kachchap (rkachchap)
|
||||||
|
|
||||||
|
|
||||||
|
Repos Impacted
|
||||||
|
--------------
|
||||||
|
|
||||||
|
* starlingx/config and starlingx/stx-puppet repo would be modified to add API/CLI support for AppArmor.
|
||||||
|
* A new repo would be created for security-profiles-operator.
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
* Integration of SPO as a system application
|
||||||
|
* CLI, RESTAPI, DB update and config update backend
|
||||||
|
* Performance impact testing
|
||||||
|
* End-to-end integration testing
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
* Security Profile Operator (SPO, https://github.com/kubernetes-sigs/security-profiles-operator)
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
Testing will involve:
|
||||||
|
|
||||||
|
* Unit test cases for the new code introduced
|
||||||
|
* Installing/enabling the SPO system app
|
||||||
|
* verifying the enabling/disabling of Apparmor at host level using API and CLI
|
||||||
|
* Configuring Apparmor profiles across hosts
|
||||||
|
* Configuring Kubernetes Pods with Apparmor profiles
|
||||||
|
* Verifying that a policy violation by a container is logged in file "/var/log/syslog".
|
||||||
|
* Backup and Restore testing
|
||||||
|
* Upgrades testing
|
||||||
|
* DC testing
|
||||||
|
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
Some information that is useful to be documented will be:
|
||||||
|
|
||||||
|
* enabling Apparmor per host
|
||||||
|
* Configuring Apparmor profiles across hosts
|
||||||
|
* Configuring Kubernetes Pods with Apparmor profiles with example to do simple test
|
||||||
|
* Guidelines on how to write/develop apparmor profiles for your containers using aa-genprof
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
* https://www.appvia.io/blog/how-to-use-the-security-profiles-operator
|
||||||
|
* https://medium.com/information-and-technology/so-what-is-apparmor-64d7ae211ed
|
||||||
|
* https://github.com/kubernetes-sigs/security-profiles-operator
|
||||||
|
* https://github.com/kubernetes/kubernetes/tree/master/test/images/apparmor-loader
|
||||||
|
* https://github.com/sysdiglabs/kube-apparmor-manager
|
||||||
|
* https://kubernetes.io/docs/tutorials/security/apparmor/#example
|
||||||
|
|
||||||
|
|
||||||
|
History
|
||||||
|
=======
|
||||||
|
|
||||||
|
.. list-table:: Revisions
|
||||||
|
:header-rows: 1
|
||||||
|
|
||||||
|
* - Release Name
|
||||||
|
- Description
|
||||||
|
* - STX-8.0
|
||||||
|
- Introduced
|
||||||
Loading…
Reference in New Issue
Block a user