Doc Update for Harbor Support
Story: 2010721 Task: 50133 Change-Id: Id5adb9966bf8c081629116b46352a155a0e4dfd5 Signed-off-by: Elisamara Aoki Goncalves <elisamaraaoki.goncalves@windriver.com>
This commit is contained in:
Elisamara Aoki Goncalves
parent
56c38b03bb
commit
029d1af2d8
Binary file not shown.
|
After Width: | Height: | Size: 34 KiB |
BIN
doc/source/admintasks/kubernetes/figures/library-harbor.png
Normal file
BIN
doc/source/admintasks/kubernetes/figures/library-harbor.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
@ -0,0 +1,590 @@
|
||||
.. _harbor-as-system-app-1d1e3ec59823:
|
||||
|
||||
============================
|
||||
Harbor as System Application
|
||||
============================
|
||||
|
||||
.. rubric:: |context|
|
||||
|
||||
Harbor is an open-source registry that secures artifacts with policies and
|
||||
role-based access control, ensures images are scanned and free from
|
||||
vulnerabilities, and signs images as trusted. Harbor has been evolved to a
|
||||
complete |OCI| compliant cloud-native artifact registry.
|
||||
|
||||
With Harbor V2.0, users can manage images, manifest lists, Helm charts,
|
||||
|CNABs|, |OPAs| among others which all adhere to the |OCI| image specification.
|
||||
It also allows for pulling, pushing, deleting, tagging, replicating, and
|
||||
scanning such kinds of artifacts. Signing images and manifest list are also
|
||||
possible now.
|
||||
|
||||
Harbor supports replication of images between registries, and offers advanced
|
||||
security features such as user management, access control and activity
|
||||
auditing.
|
||||
|
||||
See https://goharbor.io/docs/2.0.0/ for more details on Harbor.
|
||||
|
||||
-------------------
|
||||
Harbor Installation
|
||||
-------------------
|
||||
|
||||
.. rubric:: |prereq|
|
||||
|
||||
- CephFS backed |PVCs| are recommended for Harbor such that when configuring
|
||||
multiple replicas, for |AIO-DX| or Standard configurations, both Harbor
|
||||
replicas can read and write to the registry.
|
||||
|
||||
- Create a secret as described below:
|
||||
|
||||
- Generate certificates and create secret.
|
||||
|
||||
A |CA| cert and server cert creation procedure using cert-manager is
|
||||
specified below:
|
||||
|
||||
Create the certificate for Harbor using Cert-Manager and using the
|
||||
local |CA|, system-local-ca, as the issuer. Note that the certificate
|
||||
should be created in the ``harbor-tls SECRET`` in the Harbor
|
||||
``NAMESPACE``.
|
||||
|
||||
For example:
|
||||
|
||||
#. Create the following Harbor certificate yaml configuration file:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
~(keystone_admin)]$ cat <<EOF > harbor-certificate.yaml
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: harbor-certificate
|
||||
namespace: harbor
|
||||
spec:
|
||||
secretName: harbor-tls
|
||||
issuerRef:
|
||||
name: system-local-ca
|
||||
kind: ClusterIssuer
|
||||
duration: 2160h # 90 days
|
||||
renewBefore: 360h # 15 days
|
||||
commonName: < oam floating IP Address or FQDN >
|
||||
subject:
|
||||
organizations:
|
||||
- ABC-Company
|
||||
organizationalUnits:
|
||||
- StarlingX-harbor
|
||||
ipAddresses:
|
||||
- < oam floating IP address >
|
||||
dnsNames:
|
||||
- < harbor dns> # e.g. harbor.yourdomian.com
|
||||
- < notary dns > # optional, required only if exposed on ingress e.g. notary.yourdomian.com
|
||||
EOF
|
||||
|
||||
#. Apply the configuration:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
~(keystone_admin)]$ kubectl apply -f harbor-certificate.yaml
|
||||
|
||||
#. Verify the configuration:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
~(keystone_admin)]$ kubectl get certificate harbor-certificate -n harbor
|
||||
|
||||
After successful configuration, the certificate's Ready status
|
||||
will be True.
|
||||
|
||||
- nodePort
|
||||
|
||||
#. Create Harbor using NodePort to expose the service
|
||||
|
||||
.. note::
|
||||
|
||||
The instructions below assume that the NodePorts 30102, 30103
|
||||
and 30104 are available; i.e. not used by any other
|
||||
applications.
|
||||
|
||||
#. Locate the Harbor system application tarball in
|
||||
``/usr/local/share/applications/helm``.
|
||||
|
||||
For example:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
/usr/local/share/applications/helm/harbor-<version>.tgz
|
||||
|
||||
#. Upload the Harbor application.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
~(keystone_admin)]$ system application-upload /usr/local/share/applications/helm/harbor-<version>.tgz
|
||||
|
||||
#. Configure the Helm Overrides for Harbor.
|
||||
|
||||
Below values need to be configured for nodePort:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
|
||||
expose:
|
||||
|
||||
type: nodePort # Type should be nodeport
|
||||
tls:
|
||||
enabled: true
|
||||
certSource: secret
|
||||
secret: # Certificate Source is secret
|
||||
secretName: "harbor-tls" # A secret containing tls.crt and tls.key
|
||||
notarySecretName: "harbor-tls" # A secret containing tls.crt and tls.key
|
||||
|
||||
nodePort:
|
||||
# The name of NodePort service
|
||||
name: harbor
|
||||
ports:
|
||||
http:
|
||||
# The service port Harbor listens on when serving HTTP
|
||||
port: 80
|
||||
# The node port Harbor listens on when serving HTTP
|
||||
nodePort: 30002
|
||||
https:
|
||||
# The service port Harbor listens on when serving HTTPS
|
||||
port: 443
|
||||
# The node port Harbor listens on when serving HTTPS
|
||||
nodePort: 30003
|
||||
# Only needed when notary.enabled is set to true
|
||||
notary:
|
||||
# The service port Notary listens on
|
||||
port: 4443
|
||||
# The node port Notary listens on
|
||||
nodePort: 30004
|
||||
|
||||
|
||||
externalURL: https://harbor.yourdomian.com:30003 # URL of harbor listing on 30003 port
|
||||
|
||||
For |AIO-DX| and standard setup, add below ``storageClass`` and
|
||||
``accessModes`` override.
|
||||
|
||||
Underlying PVCs pre-requisistes: ``Harbor-Jobservice`` and
|
||||
``Harbor-Registry`` microservice.
|
||||
|
||||
For example:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
persistence:
|
||||
enabled: true
|
||||
resourcePolicy: "keep"
|
||||
persistentVolumeClaim:
|
||||
registry:
|
||||
existingClaim: ""
|
||||
storageClass: "cephfs"
|
||||
subPath: ""
|
||||
accessMode: ReadWriteMany
|
||||
size: 5Gi
|
||||
annotations: {}
|
||||
jobservice:
|
||||
jobLog:
|
||||
existingClaim: ""
|
||||
storageClass: "cephfs"
|
||||
subPath: ""
|
||||
accessMode: ReadWriteMany
|
||||
size: 1Gi
|
||||
annotations: {}
|
||||
|
||||
#. Execute Helm overrides.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
~(keystone_admin)]$ system helm-override-update harbor harbor harbor --values values.yaml
|
||||
|
||||
#. Apply/Create the Harbor system application.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
~(keystone_admin)]$ system application-apply harbor
|
||||
|
||||
- Ingress
|
||||
|
||||
Create Harbor using Ingress to expose the service.
|
||||
|
||||
.. note::
|
||||
|
||||
The instructions below assume that the URL
|
||||
``harbor.yourdomain.com`` has been configured in the |DNS| server
|
||||
owning ``yourdomain.com`` as the ``OAM FLOATING IP Address`` of
|
||||
|prod|.
|
||||
|
||||
#. Locate the Harbor system application tarball in
|
||||
``/usr/local/share/applications/helm``.
|
||||
|
||||
For example:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
/usr/local/share/applications/helm/harbor-<version>.tgz
|
||||
|
||||
#. Upload the Harbor application.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
~(keystone_admin)]$ system application-upload /usr/local/share/applications/helm/harbor-<version>.tgz
|
||||
|
||||
#. Configure the Helm overrides for Harbor configuration.
|
||||
|
||||
The values below need to be configured for ingress in the
|
||||
``values.yaml`` file.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
expose:
|
||||
type: ingress. # Type should be ingress
|
||||
tls:
|
||||
enabled: true
|
||||
certSource: secret
|
||||
secret: # Certificate Source is secret
|
||||
secretName: "harbor-tls" # Above created secret name
|
||||
notarySecretName: "harbor-tls" # Above created secret name
|
||||
ingress:
|
||||
hosts:
|
||||
core: harbor.yourdomian.com # Harbor Domain name
|
||||
notary: notary.yourdomian.com # Notary Domain name
|
||||
annotations:
|
||||
kubernetes.io/ingress.class: nginx. # Add ingressclass name. It would be # "nginx" if you are using default ingress # controller.
|
||||
nginx.org/client-max-body-size: "0". # Add this notation for nginx otherwise nginx # will reject the image pull & push
|
||||
externalURL: https://harbor.yourdomian.com # URL of harbor
|
||||
|
||||
|
||||
For |AIO-DX| and standard setup, add below ``storageClass`` and
|
||||
``accessModes`` override for |PVC| used for ``Harbor-Jobservice``
|
||||
and ``Harbor-Registry`` microservice.
|
||||
|
||||
For example:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
persistence:
|
||||
enabled: true
|
||||
resourcePolicy: "keep"
|
||||
persistentVolumeClaim:
|
||||
registry:
|
||||
existingClaim: ""
|
||||
storageClass: "cephfs"
|
||||
subPath: ""
|
||||
accessMode: ReadWriteMany
|
||||
size: 5Gi
|
||||
annotations: {}
|
||||
jobservice:
|
||||
jobLog:
|
||||
existingClaim: ""
|
||||
storageClass: "cephfs"
|
||||
subPath: ""
|
||||
accessMode: ReadWriteMany
|
||||
size: 1Gi
|
||||
annotations: {}
|
||||
|
||||
Update the Helm overrides.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
~(keystone_admin)]$ system helm-override-update harbor harbor harbor --values values.yaml
|
||||
|
||||
#. Apply/Create the Harbor system application.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
~(keystone_admin)]$ system application-apply harbor
|
||||
|
||||
|
||||
-------------------------------------------------
|
||||
Configure LDAP Authentication for Harbor Registry
|
||||
-------------------------------------------------
|
||||
|
||||
To configure Harbor to use |prod| Local |LDAP| for authentication, follow the
|
||||
instructions in `Configure LDAP/Active Directory Authentication
|
||||
<https://goharbor.io/docs/2.8.0/administration/configure-authentication/ldap-auth/>`__
|
||||
with the following values:
|
||||
|
||||
For |prod| local |LDAP|:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
LDP URL: ldap://controller
|
||||
|
||||
LDAP search DN: cn=ldapadmin,dc=cgcs,dc=local
|
||||
|
||||
LDAP Search Password: <Password of ldapadmin>
|
||||
|
||||
LDAP Base DN: dc=cgcs,dc=local
|
||||
|
||||
LDAP UID: cn
|
||||
|
||||
--------------------------------------
|
||||
Push an Image to a <project> in Harbor
|
||||
--------------------------------------
|
||||
|
||||
#. Run :command:`sudo su` before Docker login.
|
||||
|
||||
#. Docker Login.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
docker login <harbor_address> -u <username>
|
||||
|
||||
.. note::
|
||||
|
||||
Replace ``<harbor_address>`` with actual harborURL and replace
|
||||
``<username>`` with your actual username.
|
||||
|
||||
#. Tag the image.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
docker tag redis:latest <harbor_address>/<project>/redis:latest
|
||||
|
||||
#. Push the image.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
docker push <harbor_address>/<project>/redis:latest
|
||||
|
||||
-------------------------
|
||||
Pull an Image from Harbor
|
||||
-------------------------
|
||||
|
||||
Use command to pull an image:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
docker pull <harbor_address>/<project>/ redis:latest
|
||||
|
||||
Where ``<harbor-address>`` is either:
|
||||
|
||||
- for ``'Ingress' expose: harbor.yourdomian.com``
|
||||
|
||||
- for ``'NodePort' expose: https:// <oam-floating-ip>:30103``
|
||||
|
||||
----------------------------------
|
||||
Push a Helm Chart as an OCI Object
|
||||
----------------------------------
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
helm package <Chart_Path>
|
||||
|
||||
helm push <chart_package> oci://<harbor_address>/<project>/
|
||||
|
||||
-----------------
|
||||
Pull a Helm Chart
|
||||
-----------------
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
helm pull oci://<chart_package/<project>/harbor
|
||||
|
||||
|
||||
------------------
|
||||
Project Management
|
||||
------------------
|
||||
|
||||
A project in Harbor contains all the repositories (images) of a particular
|
||||
application; e.g. the |prod| project in Harbor would contain all the
|
||||
images from |prod|.
|
||||
|
||||
For more details on creating and configuring a project see: `Create Projects
|
||||
<https://goharbor.io/docs/2.8.0/working-with-projects/create-projects/>`__ and
|
||||
`Project Configuration
|
||||
<https://goharbor.io/docs/2.8.0/working-with-projects/project-configuration/>`__.
|
||||
|
||||
|
||||
---------------------------------
|
||||
Use Images from a Remote Registry
|
||||
---------------------------------
|
||||
|
||||
A project in a local Harbor registry can be used as a pull-through cache for a
|
||||
remote registry. For example, a Harbor registry project on a |prod| Subcloud
|
||||
can be used as a pull-through cache of a Harbor registry project on its |prod|
|
||||
system controller. This can be achieved using a proxy project which acts as a
|
||||
proxy to the remote registry, or using a replication of the remote registry.
|
||||
|
||||
.. rubric:: Add a proxy project
|
||||
|
||||
Proxy cache allows you to use Harbor to proxy and cache images from a target
|
||||
public or private registry. It does not allow to push the image by the user.
|
||||
Below are the steps to create a proxy project:
|
||||
|
||||
#. Create a registry endpoint. See `Creating Replication Endpoints
|
||||
<https://goharbor.io/docs/2.8.0/administration/configuring-replication/create-replication-endpoints/>`__.
|
||||
|
||||
#. Create a new project using the above created registry endpoint. See
|
||||
`Configure Proxy Cache
|
||||
<https://goharbor.io/docs/2.8.0/administration/configure-proxy-cache/>`__.
|
||||
|
||||
.. rubric:: Add Replication Rule
|
||||
|
||||
Setting up pull based replication replicates the images from the remote
|
||||
registry based on a trigger. Trigger can be Manual, scheduled, or event based
|
||||
as shown below.
|
||||
|
||||
.. image:: figures/new-replication-rule.png
|
||||
|
||||
#. Create a replication endpoint. See `Creating Replication Endpoints
|
||||
<https://goharbor.io/docs/2.8.0/administration/configuring-replication/create-replication-endpoints/>`__.
|
||||
|
||||
#. Create a new replication rule with replication mode as ``pull based`` as
|
||||
describe in `Creating a Replication Rule
|
||||
<https://goharbor.io/docs/2.8.0/administration/configuring-replication/create-replication-rules/>`__.
|
||||
|
||||
|
||||
------------------------
|
||||
Add Users to the Project
|
||||
------------------------
|
||||
|
||||
You can add individual users to a project, for more details see `Assign Users
|
||||
to a Project
|
||||
<https://goharbor.io/docs/2.8.0/working-with-projects/create-projects/add-users/>`__.
|
||||
|
||||
- |LDAP|/AD users can be added to the project.
|
||||
|
||||
- |LDAP|/AD groups to projects and assign a role to the group.
|
||||
|
||||
- You can see the various roles and user permission associated with the roles
|
||||
in `User Permissions By Role
|
||||
<https://goharbor.io/docs/2.8.0/administration/managing-users/user-permissions-by-role/>`__.
|
||||
|
||||
|
||||
-------------------------------
|
||||
Configure Signed Images Support
|
||||
-------------------------------
|
||||
|
||||
For more details see: `Implementing Content Trust
|
||||
<https://goharbor.io/docs/2.8.0/working-with-projects/project-configuration/implementing-content-trust/>`__.
|
||||
|
||||
#. Select cosign or notary for content trust. Harbor will then
|
||||
only allow verified images to be pulled from the project.
|
||||
|
||||
.. image:: figures/library-harbor.png
|
||||
|
||||
#. Enable content trust by setting the following environment variables on the
|
||||
machine on which you run the Docker client.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
export DOCKER_CONTENT_TRUST=1
|
||||
|
||||
export DOCKER_CONTENT_TRUST_SERVER=https://<harbor_address>:<notary port[30004]>
|
||||
|
||||
If you push the image for the first time, you will be asked to enter the
|
||||
root key passphrase.
|
||||
|
||||
------------------------------
|
||||
Configure Trivy Scanner Plugin
|
||||
------------------------------
|
||||
|
||||
Trivy is installed and configured as a default scanner.
|
||||
|
||||
.. image:: figures/interrogation-services.png
|
||||
:width: 800
|
||||
|
||||
-----------------------------
|
||||
Configure Size of Registry DB
|
||||
-----------------------------
|
||||
|
||||
Registry DB size can be configured by setting following in ``values.yaml``
|
||||
under:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
persistence:
|
||||
registry:
|
||||
size: 5Gi
|
||||
jobservice:
|
||||
jobLog:
|
||||
size: 1Gi
|
||||
|
||||
Use :command:`system helm-override` command to set the value (Default set to
|
||||
5Gi).
|
||||
|
||||
------------------------------------------------------
|
||||
Enforcement of Image Security Policies Using Portieris
|
||||
------------------------------------------------------
|
||||
|
||||
Portieris allows to configure trust policies for an individual namespace or
|
||||
cluster-wide and checks the image against a signed image list on a specified
|
||||
notary server to enforce the configured image policies. An Administrator can
|
||||
enforce |prod| image security policies using the Portieris admission
|
||||
controller.
|
||||
|
||||
It is required to pull from a registry using a ``docker-registry`` secret.
|
||||
Enforcing trust for anonymous image pulls is not supported.
|
||||
|
||||
To use portieris, an administrator needs to follow below steps:
|
||||
|
||||
#. Install portieris as specified in :ref:`install-portieris`.
|
||||
|
||||
#. Create a ``docker-registry`` secret.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
kubectl create secret docker-registry \
|
||||
-n harbor harbor-registry-secret \
|
||||
--docker-server=<harbor-dns>:port \
|
||||
--docker-username=admin \
|
||||
--docker-password=Test@123
|
||||
|
||||
.. note::
|
||||
|
||||
If the pod creation with the above secret fails, the user should try
|
||||
with new secret with ``--docker-server`` as ``<harbor-dns>``.
|
||||
|
||||
#. Configure image policy to allow images from Harbor registry + notary as
|
||||
specified
|
||||
:ref:`portieris-clusterimagepolicy-and-imagepolicy-configuration`. Below
|
||||
example shows the policy allowing image from Harbor registry.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
apiVersion: portieris.cloud.ibm.com/v1
|
||||
kind: ImagePolicy
|
||||
metadata:
|
||||
name: allow-custom
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
namespace: harbor
|
||||
spec:
|
||||
repositories:
|
||||
- name: "<harbor-dns>:30003/*"
|
||||
policy:
|
||||
trust:
|
||||
enabled: true
|
||||
trustServer: "https://<notary dns>:30004" # Optional, custom trust server for repository
|
||||
|
||||
#. Pull a signed image from Harbor registry in a pod using ``harbor-secret``
|
||||
created above. Please note that image policy and pod should be created in
|
||||
the same namespace.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: test-pod-public
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- sleep
|
||||
- '3600'
|
||||
image: <harbor-dns>:30003/public-demo/redis:latest
|
||||
imagePullPolicy: Always
|
||||
name: test-pod
|
||||
tolerations:
|
||||
- key: "node-role.kubernetes.io/master"
|
||||
operator: "Exists"
|
||||
effect: "NoSchedule"
|
||||
imagePullSecrets:
|
||||
- name: harbor-registry-secret
|
||||
|
||||
----------
|
||||
Limitation
|
||||
----------
|
||||
|
||||
Harbor application cannot be deployed during bootstrap due to the bootstrap
|
||||
deployment dependencies such as early availability of storage class.
|
||||
@ -88,6 +88,15 @@ O-RAN O2 Interface
|
||||
|
||||
oran-o2-application-b50a0c899e66
|
||||
|
||||
--------------------
|
||||
Harbor as System App
|
||||
--------------------
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
harbor-as-system-app-1d1e3ec59823
|
||||
|
||||
--------------------------------------
|
||||
Technology Preview - Istio Application
|
||||
--------------------------------------
|
||||
|
||||
@ -27,6 +27,8 @@
|
||||
.. |CDI| replace:: :abbr:`CDI (Containerized Data Importer)`
|
||||
.. |CLI| replace:: :abbr:`CLI (Command Line Interface)`
|
||||
.. |CLIs| replace:: :abbr:`CLIs (Command Line Interfaces)`
|
||||
.. |CNAB| replace:: :abbr:`CNAB (Cloud Native Application Bundle)`
|
||||
.. |CNABs| replace:: :abbr:`CNABs (Cloud Native Application Bundles)`
|
||||
.. |CNI| replace:: :abbr:`CNI (Container Networking Interface)`
|
||||
.. |CNIs| replace:: :abbr:`CNIs (Container Networking Interfaces)`
|
||||
.. |CoW| replace:: :abbr:`CoW (Copy on Write)`
|
||||
@ -121,9 +123,12 @@
|
||||
.. |OAM| replace:: :abbr:`OAM (Operations, administration and management)`
|
||||
.. |OEM| replace:: :abbr:`OEM (Original Equipment Manufacturer)`
|
||||
.. |OC| replace:: :abbr:`OC (Ordinary Clock)`
|
||||
.. |OCI| replace:: :abbr:`OCI (Open Container Initiative)`
|
||||
.. |OID| replace:: :abbr:`OID (Object Identifier)`
|
||||
.. |OIDC| replace:: :abbr:`OIDC (OpenID Connect)`
|
||||
.. |ONAP| replace:: :abbr:`ONAP (Open Network Automation Program)`
|
||||
.. |OPA| replace:: :abbr:`OPA (Open Policy Agent)`
|
||||
.. |OPAs| replace:: :abbr:`OPAs (Open Policy Agents)`
|
||||
.. |OVS| replace:: :abbr:`OVS (Open Virtual Switch)`
|
||||
.. |OSD| replace:: :abbr:`OSD (Object Storage Daemons)`
|
||||
.. |OSDs| replace:: :abbr:`OSDs (Object Storage Daemons)`
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user