system-config/doc/source/zuul.rst

title

Zuul

Zuul

Zuul is a pipeline-oriented project gating system. It facilitates running tests and automated tasks in response to Gerrit events.

At a Glance

Hosts

• http://status.openstack.org/zuul
• http://zuul.openstack.org
• http://zuul-dev.openstack.org
• zm*.openstack.org
• zl*.openstack.org

Puppet

• https://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/
• modules/openstack_project/manifests/zuul_prod.pp
• modules/openstack_project/manifests/zuul_dev.pp

Configuration

• zuul/layout.yaml

Projects

• https://git.openstack.org/cgit/openstack-infra/zuul

Bugs

• https://storyboard.openstack.org/#!/project/679

Resources

• Zuul Reference Manual

Chat

• #zuul on freenode

Overview

The OpenStack project uses a number of pipelines in Zuul:

check

Newly uploaded patchsets enter this pipeline to receive an initial +/-1 Verified vote.

gate

Changes that have been approved by core developers are enqueued in order in this pipeline, and if they pass tests, will be merged.

post

This pipeline runs jobs that operate after each change is merged.

pre-release

This pipeline runs jobs on projects in response to pre-release tags.

release

When a commit is tagged as a release, this pipeline runs jobs that publish archives and documentation.

silent

This pipeline is used for silently testing new jobs.

experimental

This pipeline is used for on-demand testing of new jobs.

periodic

This pipeline has jobs triggered on a timer for e.g. testing for environmental changes daily.

Zuul watches events in Gerrit (using the Gerrit "stream-events" command) and matches those events to the pipelines above. If a match is found, it adds the change to the pipeline and starts running related jobs.

The gate pipeline uses speculative execution to improve throughput. Changes are tested in parallel under the assumption that changes ahead in the queue will merge. If they do not, Zuul will abort and restart tests without the affected changes. This means that many changes may be tested in parallel while continuing to assure that each commit is correctly tested.

Zuul's current status may be viewed at http://status.openstack.org/zuul/.

Zuul's configuration is stored in zuul/layout.yaml. Anyone may propose a change to the configuration by editing that file and submitting the change to Gerrit for review.

For the full syntax of Zuul's configuration file format, see the Zuul reference manual.

Sysadmin

Zuul and gear are lightweight - it should be possible to run both on a 1G instance for small deployments. OpenStack's deployment requires at least a 2G instance at the time of writing.

Zuul is stateless, so the server does not need backing up. However zuul talks through git and ssh so you will need to manually check ssh host keys as the zuul user. e.g.:

sudo su - zuul
ssh -p 29418 review.openstack.org

Restarts

Zuul restarts are disruptive, so non-emergency restarts should always be scheduled for quieter times of the day, week and cycle. To be as courteous to developers as possible, just prior to a restart the Zuul Status Page should be checked to see the status of the gate. If there is a series of changes nearly merged, wait until that has been completed.

Since Zuul is stateless, some work needs to be done to save and then re-enqueue patches when restarts are done. To accomplish this, start by running zuul-changes.py to save the check and gate queues:

python /opt/zuul/tools/zuul-changes.py http://zuul.openstack.org \
  check >check.sh
python /opt/zuul/tools/zuul-changes.py http://zuul.openstack.org \
  gate >gate.sh

These check.sh and gate.sh scripts will be used after the restart to re-enqueue the changes.

Now use service zuul stop to stop zuul and then run ps to make sure the process has actually stopped, it may take several seconds for it to finally go away.

With Zuul stopped, delete all the used nodes in nodepool. Wait for one of each variety to come up before using service zuul start to start zuul again.

Once Zuul is started, run netcat against localhost 4730 port to confirm that all the node types (particularly the uncommon ones) are registered with Gearman before re-enqueuing patches. For instance:

echo "status" | nc localhost 4730 | grep :centos7

When you are satisfied that all the node types have returned, first run the gate.sh script and then check.sh to re-enqueue the changes from before the restart:

./gate.sh
./check.sh

You may watch the Zuul Status Page to confirm that changes are returning to the queues.

Mergers

Servers with names matching the pattern zm*.openstack.org are Zuul Mergers. These are horizontally scalable components of Zuul which perform git operations for the benefit of jobs. They serve git repositories via Apache over http, and jobs fetch changes to test from them. They can be started and stopped at will, and new ones added as necessary to accommodate load. If you remove a merger, be sure to leave Apache running for several hours until the last job that may have been launched with instructions to fetch from that merger has completed.

Launchers

We use an Ansible based launcher in Zuul to actually run jobs. This component runs on a horizontally scalable set of servers named zl*.openstack.org. It reads job configuration from Jenkins Job Builder files in the project-config repository and translates that into Ansible playbooks which it runs on our workers. Our jobs are configured to upload as much information as possible along with their logs, but if there is an error which can not be diagnosed in that manner, Ansible logs are available in the launcher-debug log file on the launcher host. You may use the Zuul build UUID to track assignment of a given job from the Zuul scheduler to the Zuul launcher used by that job.