zuul/zuul-operator
Code Issues Proposed changes
fef7c272f7
zuul-operator/doc/source/index.rst
Jan Gutter b723bc761b
Publish zuul-operator to quay.io 
* non-ci tested scripts need updating - these are intended for
  local development workflows.
* This now publishes the operator to quay.io/zuul-ci
* This is mostly a rework of
  https://review.opendev.org/c/zuul/zuul-operator/+/881245 and
  https://review.opendev.org/c/zuul/zuul-operator/+/924870
  with much appreciation and credits to original authors

Change-Id: I0ffd39fceeaf4947ed329dd4ddb7283cdc98df86
2024-08-18 12:32:57 +01:00

658 lines
20 KiB
ReStructuredText
Raw Blame History

Zuul Operator
=============
This is a Kubernetes Operator for the Zuul Project Gating System.
Zuul has a number of components and depencencies, and this operator is
designed to simplify creating and maintaining Zuul systems in
Kubernetes.
Somewhat unusually, this operator offers the ability to completely
manage Zuul's operational dependencies, to the point of even
installing other operators upon which it relies. Be sure to read
about deployment options if you want to perform some of these tasks
yourself.
Simple Example
--------------
The quickest way to get a running Zuul is to allow the operator to
manage all of the dependencies for you. In this case, the operator
will:
* Install cert-manager and set up a self-signed cluster issuer
* Install the Percona XtraDB operator and create a three-node PXC
database cluster
* Create a three-node ZooKeeper cluster
* And of course, create a Zuul system
.. note::
Installing other operators requires a high level of access, so when
used in this manner, zuul-operator runs with cluster admin
privileges. If you would like the operator to run with reduced
privileges, see Managing Operator Dependencies.
From the root of the zuul-operator repo, run:
.. code-block:: bash
kubectl apply -f deploy/crds/zuul-ci_v1alpha1_zuul_crd.yaml
kubectl apply -f deploy/rbac-admin.yaml
kubectl apply -f deploy/operator.yaml
You probably want a namespace, so go ahead and create one with:
.. code-block:: bash
kubectl create namespace zuul
You will need to prepare two config files for Zuul: the Nodepool
config file and the Zuul tenant config file. See the Zuul and
Nodepool manuals for how to prepare those. When they are ready, add
them to Kubernetes with commands like:
.. code-block:: bash
kubectl -n zuul create secret generic zuul-nodepool-config --from-file=nodepool.yaml
kubectl -n zuul create secret generic zuul-tenant-config --from-file=main.yaml
Then create a file called ``zuul.yaml`` which looks like:
.. code-block:: yaml
---
apiVersion: operator.zuul-ci.org/v1alpha2
kind: Zuul
metadata:
name: zuul
spec:
executor:
count: 1
sshkey:
secretName: gerrit-secrets
scheduler:
config:
secretName: zuul-tenant-config
launcher:
config:
secretName: zuul-nodepool-config
web:
count: 1
connections:
opendev:
driver: git
baseurl: https://opendev.org
This will create the most basic of Zuul installations, with one each
of the `zuul-executor`, `zuul-scheduler`, and `zuul-web` processes.
It will also create a Nodepool launcher for each of the providers
listed in your ``nodepool.yaml``. If your Zuul tenant config file
requires more connections, be sure to add them here.
Managing Operator Dependencies
------------------------------
You may not want zuul-operator to install other operators (for
example, if your cluster has other users and you don't want
cert-manager or pxc-operator to be tied to a Zuul installation, or if
you would prefer to avoid granting zuul-operator cluster admin
privileges). In that case, you may install the other operators
yourself and still allow zuul-operator to use those other operators.
It can still create a PXC cluster for you as long as the pxc-operator
is present.
To use this mode of operation, make sure the following dependencies
are installed before using zuul-operator:
* Cert-manager (at least version 1.2.0)
* Percona-xtradb-cluster-operator (at least version 1.7.0)
With these installed, you may install zuul-operator with reduced
privileges:
.. code-block:: bash
kubectl apply -f deploy/crds/zuul-ci_v1alpha1_zuul_crd.yaml
kubectl apply -f deploy/rbac.yaml
kubectl apply -f deploy/operator.yaml
After this point, usage is the same as other methods.
Externally Managed Zuul Dependencies
------------------------------------
If you want zuul-operator to do even less work, you can have it avoid
managing either ZooKeeper or the SQL database.
Externally Managed ZooKeeper
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you already have a ZooKeeper instance you would like Zuul to use,
add the following to the `Zuul` spec:
.. code-block:: yaml
---
apiVersion: operator.zuul-ci.org/v1alpha2
kind: Zuul
spec:
zookeeper:
connectionString: ...
secretName: ...
The ``connectionString`` field should be a standard ZooKeeper
connection string, and the ``secretName`` field should be a Kubernetes
TLS secret with the client cert for Zuul to use when connecting to
ZooKeeper. TLS is required.
Externally Managed Database
~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you would like to use an existing database, add the following to
the `Zuul` spec:
.. code-block:: yaml
---
apiVersion: operator.zuul-ci.org/v1alpha2
kind: Zuul
spec:
database:
dburi: ...
The ``dburi`` field should contain a Python db-api URI; it corresponds
to the ``dburi`` entry in ``zuul.conf``.
Secrets
-------
The operator uses Kubernetes secrets as input values for several
aspects of operation. There are some general rules about how secrets
are used:
* For configuration files, secret keys are expected to be the typical
filename (for example, ``nodepool.yaml`` for the Nodepool config
file).
* For Zuul connection entries, secret keys correspond to the
configuration file attributes for that section (e.g., ``app_key``
for the github driver).
See the reference documentation for the specific `secretName` entry
for details.
Zuul Preview
------------
The operator has optional support for deploying a zuul-preview
service. This is an experimental add-on for Zuul to serve Zuul
artifacts from the root of a domain (this can be useful for serving
static HTML/Javascript sites). If you enable this, the operator will
configure a ``zuul-preview`` service to which you may route an Ingress
or LoadBalancer.
Zuul Registry
-------------
The operator has optional support for deploying a zuul-registry
service. This is an experimental add-on for Zuul to act as an
intermediate registry for the container image jobs in `zuul-jobs`.
If you enable this, the operator will, by default, configure a
``zuul-registry`` service in a manner appropriate for access from
within the cluster only. If you need to access the registry from
outside the cluster, you will need to additionally add an Ingress or
LoadBalancer, as well as provide TLS certs with the appropriate
hostname. Currently, zuul-registry performs its own TLS termination.
If you usue this, you will also need to provide a ``registry.yaml``
config file in a secret. You only need to provide the ``users`` and,
if you are accessing the registry outside the cluster, the
``public-url`` setting (omit it if you are accessing it from within
the cluster only).
.. code-block:: yaml
registry:
users:
- name: testuser
pass: testpass
access: write
Specification Reference
-----------------------
This is a fully populated example (with the exception of connection
entries which can contain `zuul.conf` attributes passed through
verbatim):
.. code-block:: yaml
apiVersion: zuul-ci.org/v1alpha2
kind: Zuul
spec:
imagePrefix: quay.io/zuul-ci
imagePullSecrets:
- name: my-docker-secret
zuulImageVersion: latest
zuulPreviewImageVersion: latest
zuulRegistryImageVersion: latest
nodepoolImageVersion: latest
database:
secretName: mariadbSecret
zookeeper:
hosts: zk.example.com:2282
secretName: zookeeperTLS
merger:
count: 5
git_user_email: zuul@example.org
git_user_name: Example Zuul
executor:
count: 5
manage_ansible: false
web:
count: 1
status_url: https://zuul.example.org
fingergw:
count: 1
scheduler:
count: 1
connections:
gerrit:
driver: gerrit
server: gerrit.example.com
secretName: gerritSecrets
user: zuul
baseurl: http://gerrit.example.com:8080
github:
driver: github
secretName: githubSecrets
rate_limit_logging: false
app_id: 1234
jobVolumes:
- context: trusted
access: ro
path: /authdaemon/token
volume:
name: gcp-auth
hostPath:
path: /var/authdaemon/executor
type: DirectoryOrCreate
.. attr:: Zuul
The Zuul kind is currently the only resource directly handled by
the operator. It holds a complete description of a Zuul system
(though at least partly via secrets referenced by this resource).
.. attr:: spec
.. attr:: imagePrefix
:default: quay.io/zuul-ci
The prefix to use for images. The image names are fixed
(``zuul-executor``, etc). However, changing the prefix will
allow you to use custom images or private registries.
.. attr:: imagePullSecrets
:type: list
:default: []
If supplied, this value is passed through to Kubernetes. It
should be a list of secrets.
.. attr:: name
The name of the image pull secret.
.. attr:: zuulImageVersion
:default: latest
The image tag to append to the Zuul images.
.. attr:: zuulPreviewImageVersion
:default: latest
The image tag to append to the Zuul Preview images.
.. attr:: zuulRegistryImageVersion
:default: latest
The image tag to append to the Zuul Registry images.
.. attr:: nodepoolImageVersion
:default: latest
The image tag to append to the Nodepool images.
.. attr:: database
This is not required unless you want to manage the database
yourself. If you omit this section, zuul-operator will
create a Percona XtraDB cluster for you.
You may add any attribute corresponding to the `database`
section of zuul.conf here. The ``dburi`` attribute will come
from the secret below.
.. attr:: secretName
The name of a secret containing connection information for
the database.
The key name in the secret should be ``dburi``.
.. attr:: allowUnsafeConfig
:default: False
If you are running in a resource constrained environment
(such as minikube), the requested resource values for the
Percona XtraDB may be too large. Set this to True to
override the default values and construct the cluster
regardless. Only use this for testing.
.. attr:: zookeeper
This is not required unless you want to manage the ZooKeeper
cluster yourself. If you omit this section, zuul-operator
will create a ZooKeeper cluster for you
You may add any attribute corresponding to the `zookeeper`
section of zuul.conf here. The ``hosts`` and TLS attributes
will come from the secret below.
.. attr:: hosts
A standard ZooKeeper connection string.
.. attr:: secretName
The name of a secret containing a TLS client certificate
and key for ZooKeeper. This should be (or the format
should match) a standard Kubernetes TLS secret.
The key names in the secret should be:
* ``ca.crt``
* ``tls.crt``
* ``tls.key``
.. attr:: storageClassName
:default: ''
The name of the Kubernetes storage class to use when
running using the managed ZooKeeper instance.
.. attr:: env
A list of environment variables. This will be passed through
to the Pod specifications for the scheduler, launcher, and
web. This may be used to set http_proxy environment
variables.
.. attr:: scheduler
.. attr:: config
.. attr:: secretName
The name of a secret containing the Zuul tenant config
file.
The key name in the secret should be ``main.yaml``.
.. attr:: storageClassName
:default: ''
The name of the Kubernetes storage class to use when
running using the managed ZooKeeper instance.
.. attr:: launcher
.. attr:: config
.. attr:: secretName
The name of a secret containing the Nodepool config
file.
The key name in the secret should be ``nodepool.yaml``.
.. attr:: executor
.. attr:: count
:default: 1
How many executors to manage. This is a required
component and should be at least 1.
.. attr:: sshkey
.. attr:: secretName
The name of a secret containing the SSH private key
that executors should use when logging into Nodepool
nodes. You will need to arrange for the public half of
this key to be installed on those nodes via whatever
mechanism provided by your cloud.
The key name in the secret should be ``sshkey``.
.. attr:: terminationGracePeriodSeconds
:default: 21600
When performing a rolling restart of the executors, wait
this long for jobs to finish normally. If an executor
takes longer than this amount of time, it will be
hard-stopped (and jobs will abort and retry). The default
is 6 hours, but depending on the workload, a higher or
lower value may be appropriate.
.. attr:: merger
.. attr:: count
:default: 0
How many mergers to manage. Executors also act as mergers
so this is not required. They may be useful on a busy
system.
.. attr:: web
.. attr:: count
:default: 1
How many Zuul webservers to manage. This is a required
component and should be at least 1.
.. attr:: fingergw
.. attr:: count
:default: 1
How many Zuul finger gateway servers to manage.
.. attr:: connections
This is a mapping designed to match the `connections` entries
in the main Zuul config file (`zuul.conf`). Each key in the
mapping is the name of a connection (this is the name you
will use in the tenant config file), and the values are
key/value pairs that are directly added to that connectien
entry in `zuul.conf`. In the case of keys which are
typically files (for example, SSH keys), the values will be
written to disk for you (so you should include the full
values here and not the path).
You may provide any of these values directly in this resource
or using the secret described below. You may use both, and
the values will be combined (for example, you may include all
the values here except a private key which you include in a
secret).
Example:
.. code-block:: yaml
connections:
opendev:
driver: git
baseurl: https://opendev.org
gerrit:
driver: git
baseurl: https://gerrit.examplec.mo
secretName: gerrit-secrets
.. attr:: <name>
The name of the connection. You will use this is the
tenant config file. All of the attributes describing this
connection should be included underneath this key.
.. attr:: secretName
The name of a secret describing this connection. All
of the keys and values in this secret will be merged
with the keys and values described in this connection
entry. If you need to provide a file (for example, the
``sshkey`` attribute of a Gerrit connection), include
the contents as the value of the ``sshkey`` attribute
in the secret.
.. attr:: externalConfig
A mapping of secrets for specific Nodepool drivers. Some
Nodepool drivers use external files for configuration (e.g.,
`clouds.yaml` for OpenStack). To provide these to Nodepool,
add them to a secret and specify the name of that secret in
an entry in externalConfig.
For example, a secret for OpenStack might look like:
.. code-block:: yaml
apiVersion: v1
kind: Secret
metadata:
name: openstack-secret
stringData:
clouds.yaml: "..."
To use that with Nodepool, add the following to the Operator
resource definition:
.. code-block:: yaml
externalConfig:
openstack:
secretName: openstack-secret
This will cause a `clouds.yaml` file to be created at
`/etc/openstack/clouds.yaml`.
Some Nodepool drivers may need environment variables set in
order to use these secrets. See :attr:`Zuul.spec.env` to add
those.
The keys in this mapping will become directories under
`/etc/`, and the secrets referenced will be mounted in those
directories.
.. attr:: <name>
The directory to mount under `/etc`.
.. attr:: SecretName
The name of a secret that should be mounted at `/etc/<name>`.
.. attr:: jobVolumes
A list of Kubernetes volumes to be bind mounted into the
executor's execution context. These correspond to the
`trusted_ro_paths`, `untrusted_ro_paths`, `trusted_rw_paths`,
and `untrusted_ro_paths` entries in `zuul.conf`.
The first part of each entry describes how the volume should
appear in the executor, and the `volume` attribute describes
the Kubernetes volume.
.. attr:: context
One of the following values:
.. value:: trusted
To be mounted in the `trusted` execution context.
.. value:: untrusted
To be mounted in the `untrusted` execution context.
.. attr:: access
One of the following values:
.. value:: rw
To be mounted read/write.
.. value:: ro
To be mounted read-only.
.. attr:: path
The mount point within the execution context.
.. attr:: volume
A mapping corresponding to a Kubernetes volume.
.. attr:: preview
.. attr:: count
:default: 0
How many Zuul Preview servers to manage.
.. attr:: registry
.. attr:: count
:default: 0
How many Zuul Registry servers to manage.
.. attr:: volumeSize
:default: 80G
The requested size of the registry storage volume.
.. attr:: storageClassName
:default: ''
The name of the Kubernetes storage class to use when
running using the managed ZooKeeper instance.
.. attr:: tls
.. attr:: secretName
The name of a secret containing a TLS client certificate
and key for Zuul Registry. This should be (or the format
should match) a standard Kubernetes TLS secret.
If you omit this, the operator will create a secret for
you.
.. attr:: config
.. attr:: secretName
The name of a secret containing a registry
configuration file. The key in the secret should be
``registry.yaml``. Only provide the ``users`` and, if
exposing the registry outside the cluster, the
``public-url`` entries.
View Git Blame Copy Permalink