Add a zuul01.openstack.org

In preparation for replacing the zuulv3.openstack.org host with a
larger instance, set up the necessary support in
Puppet/Hiera/Ansible. While we're here, remove or replace old
references to the since-deleted zuul.openstack.org instance, and
where possible update documentation and configuration to refer to
the new zuul.openstack.org CNAME instead of the zuulv3.openstack.org
FQDN so as to smooth the future transition.

Change-Id: Ie51e133afb238dcfdbeff09747cbd2e53093ef84

doc/source/github.rst

@@ -17,7 +17,7 @@ At a Glance
 :Puppet:
   * https://git.openstack.org/cgit/openstack-infra/system-config/tree/
   * :file:`modules/openstack_project/manifests/gerrit.pp`
-  * :file:`hiera/fqdn/zuulv3.openstack.org.yaml`
+  * :file:`hiera/group/zuul-scheduler.yaml`
 :Projects:
   * https://git.openstack.org/cgit/openstack-infra/zuul
   * https://git.openstack.org/cgit/openstack-infra/jeepyb
@@ -68,22 +68,22 @@ OAuth Credentials which are all stored in hiera.
 
 The ID is a numerical identifier found on the App settings page labeled **ID**.
 The ID is placed into the ``app_id`` field in the ``github``
-entry in ``zuul_connection_secrets`` for the ``zuulv3.openstack.org`` FQDN.
+entry in ``zuul_connection_secrets`` for the ``zuul-scheduler`` group.
 
 The Private key can only be retrieved when it is generated, so in the case it
 is lost a new one must be generated and the resulting value put into hiera.
 The Private key content is stored as ``zuul_github_app_key`` in private hiera
 and is written to ``/etc/zuul/github.key``. That path is placed into
 ``app_key`` field in the ``github`` entry in ``zuul_connections`` for the
-``zuulv3.openstack.org`` FQDN.
+``zuul-scheduler`` group.
 
 GitHub sends JSON payloads via HTTP POST to the URL configured in the Webhook
 URL setting. The current value of this setting for Zuul v3 is:
-https://zuulv3.openstack.org/connection/github/payload. It includes the
+https://zuul.openstack.org/connection/github/payload. It includes the
 configured "Webhook Secret" so that Zuul can verify that the payload actually
 did come from GitHub. The "Webhook Secret" is placed into the ``webhook_token``
 field in the ``github`` entry in ``zuul_connection_secrets`` for the
-``zuulv3.openstack.org`` FQDN.
+``zuul-scheduler`` group.
 
 The OAuth credentials for the OpenStack Zuul App are currently unused.
 

doc/source/signing.rst

@@ -459,7 +459,7 @@ as a secret to Zuul for use by release jobs.
     > https://git.openstack.org/cgit/openstack-infra/zuul/plain/tools/encrypt_secret.py?\
     > h=feature/zuulv3
     root@puppetmaster:~# python encrypt_secret.py --infile temporary.gnupg/for-zuul \
-    > --outfile temporary.gnupg/zuul.yaml https://zuulv3.openstack.org gerrit \
+    > --outfile temporary.gnupg/zuul.yaml https://zuul.openstack.org gerrit \
     > openstack-infra/project-config
     writing RSA key
     Public key length: 4096 bits (512 bytes)

doc/source/zuulv3.rst

@@ -18,8 +18,8 @@ At a Glance
 ===========
 
 :Hosts:
-  * http://zuulv3.openstack.org
+  * http://zuul.openstack.org
-  * zuulv3.openstack.org
+  * zuul.openstack.org
   * ze*.openstack.org
 :Puppet:
   * https://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/
@@ -82,7 +82,7 @@ many changes may be tested in parallel while continuing to assure that
 each commit is correctly tested.
 
 Zuul's current status may be viewed at
-`<http://zuulv3.openstack.org/>`_.
+`<http://zuul.openstack.org/>`_.
 
 Zuul's configuration is stored in :config:`zuul/main.yaml`.  Anyone
 may propose a change to the configuration by editing that file and
@@ -111,7 +111,7 @@ Scheduler
 ---------
 
 The Zuul Scheduler and gear are all co-located on a single host,
-zuulv3.openstack.org.
+referred to by the ``zuul.openstack.org`` CNAME in DNS.
 
 Zuul is stateless, so the server does not need backing up. However
 zuul talks through git and ssh so you will need to manually check ssh
@@ -130,7 +130,7 @@ the executors using gear.
 OpenStack's Zuul installation is also configured to write job results into
 a MySQL database via the SQL Reporter plugin. The database for that is a
 Rackspace Cloud DB and is configured in the ``mysql`` entry of the
-``zuul_connection_secrets`` entry for the ``zuulv3.openstack.org`` FQDN.
+``zuul_connection_secrets`` entry for the ``zuul-scheduler`` group.
 
 Restarting the Scheduler
 ------------------------
@@ -147,9 +147,9 @@ running `zuul-changes.py
 <https://git.openstack.org/cgit/openstack-infra/zuul/tree/tools/zuul-changes.py>`_
 to save the check and gate queues::
 
-  python /opt/zuul/tools/zuul-changes.py http://zuulv3.openstack.org \
+  python /opt/zuul/tools/zuul-changes.py http://zuul.openstack.org \
     check >check.sh
-  python /opt/zuul/tools/zuul-changes.py http://zuulv3.openstack.org \
+  python /opt/zuul/tools/zuul-changes.py http://zuul.openstack.org \
     gate >gate.sh
 
 These check.sh and gate.sh scripts will be used after the restart to
@@ -191,7 +191,7 @@ Web
 ---
 
 Zuul Web is a horizontally scalable service. It is currently running colocated
-with the scheduler on zuulv3.openstack.org. Zuul Web provides live console
+with the scheduler on zuul.openstack.org. Zuul Web provides live console
 streaming and will be the home of various web dashboards such as the status
 page.
 
@@ -223,4 +223,4 @@ found on the :ref:`github` page at :ref:`openstack_zuul_app`.
 
 .. _OpenStack Zuul: https://github.com/apps/openstack-zuul
 .. _Zuul Reference Manual: https://docs.openstack.org/infra/zuul/feature/zuulv3
-.. _Zuul Status Page: http://zuulv3.openstack.org
+.. _Zuul Status Page: http://zuul.openstack.org

hiera/group/zuul-scheduler.yaml

@@ -0,0 +1,71 @@
+---
+zuul_connections:
+  - name: 'smtp'
+    driver: 'smtp'
+    server: 'localhost'
+    port: '25'
+    default_from: 'zuul@zuul.openstack.org'
+    default_to: 'zuul.reports@zuul.openstack.org'
+
+  - name: 'gerrit'
+    driver: 'gerrit'
+    server: 'review.openstack.org'
+    canonical_hostname: 'git.openstack.org'
+    user: 'zuul'
+    sshkey: '/var/lib/zuul/ssh/id_rsa'
+    gitweb_url_template: 'https://git.openstack.org/cgit/{project.name}/commit/?id={sha}'
+
+  - name: 'mysql'
+    driver: 'sql'
+
+  - name: 'github'
+    driver: 'github'
+    app_key: '/etc/zuul/github.key'
+
+gearman_server_ssl_cert: |
+  -----BEGIN CERTIFICATE-----
+  MIIEYTCCA0mgAwIBAgIJAKkAn3gh0LBPMA0GCSqGSIb3DQEBCwUAMIG5MQswCQYD
+  VQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxDzANBgNVBAcMBkF1c3RpbjEdMBsGA1UE
+  CgwUT3BlblN0YWNrIEZvdW5kYXRpb24xFzAVBgNVBAsMDkluZnJhc3RydWN0dXJl
+  MR0wGwYDVQQDDBR6dXVsdjMub3BlbnN0YWNrLm9yZzEyMDAGCSqGSIb3DQEJARYj
+  b3BlbnN0YWNrLWluZnJhQGxpc3RzLm9wZW5zdGFjay5vcmcwHhcNMTcwNjE2MjA1
+  NDAyWhcNMjcwNjE0MjA1NDAyWjCBszELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVRl
+  eGFzMQ8wDQYDVQQHDAZBdXN0aW4xHTAbBgNVBAoMFE9wZW5TdGFjayBGb3VuZGF0
+  aW9uMRcwFQYDVQQLDA5JbmZyYXN0cnVjdHVyZTEXMBUGA1UEAwwOZ2Vhcm1hbi5z
+  ZXJ2ZXIxMjAwBgkqhkiG9w0BCQEWI29wZW5zdGFjay1pbmZyYUBsaXN0cy5vcGVu
+  c3RhY2sub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3aMR61f/
+  LZkP/acuqiCEiSFF4GI1ViNkOSPEq0CP4HfNckeW0///x6vI/uaR4MlF8g8qNFGB
+  j2FCYRW1gEzS7TLoP3xYs4SMnvXvZRbdxcozOop506quLmlfPDF1o2GzLSQYDNXe
+  WbpYiNM+EdgBjqLz4G5DdaXMMw2zYP21kbtSxJIvrpqeW/TKBGWDI2bBH81PFb9B
+  gq1P4XxI/Aw7Ez6hApLV2D6DP7JidQUGOzvGw7LUEZjLEscQU7HH8j1qDvrM2gV4
+  FRSRrtw8Yr/erBsaNr84guEZQREqiOjr1HvMZK5o1vGb69ArWSk9b8PW+A2uxvfS
+  ukv7hvNsuCouHQIDAQABo3AwbjAJBgNVHRMEAjAAMCEGCWCGSAGG+EIBDQQUFhJj
+  bGllbnQgY2VydGlmaWNhdGUwHQYDVR0OBBYEFImAuHnbfxpEEZwiiro9KEa8YA+1
+  MB8GA1UdIwQYMBaAFFP8JfdXPn8mhZLaXMa8NQIJlmneMA0GCSqGSIb3DQEBCwUA
+  A4IBAQBTNIVB758W+wBtCMlIRFUPBiR+w+7RRsY8HXME5unvO65PcsfLKQXOr3i/
+  K2SliyyBliwKY+wtbvQZVltpBiloDqslSMD6veb5YsZDzTZ+x8xP1GEhcB3c6CsN
+  0RDJ/xUGv2IXgQW8kw+MINILr9iQA6fn9dBN0OqimlchPHtvA9gO7Rv+IV3zZP+Q
+  yNWoBiZ6H5ANIt6vfcK0BHGDB6GXN9f1gpgsJd3l3vs3t/FgP1qYJiDd5VvcOXxt
+  uJziOvdg7jte0u609MWj3DOdey4HsxlEU27w13kzGI6RpPquvl/YB8Y6WMAIL8in
+  1GRv9pIfENRRHOiC57p0RSQZZ/2V
+  -----END CERTIFICATE-----
+
+zuul_ssl_cert_file_contents: |
+  -----BEGIN CERTIFICATE-----
+  MIICzjCCAbagAwIBAgIJAMV1mxY+iSJpMA0GCSqGSIb3DQEBCwUAMB8xHTAbBgNV
+  BAMMFHp1dWx2My5vcGVuc3RhY2sub3JnMB4XDTE3MDYwMjE5MzUwMloXDTI3MDUz
+  MTE5MzUwMlowHzEdMBsGA1UEAwwUenV1bHYzLm9wZW5zdGFjay5vcmcwggEiMA0G
+  CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDvgAf85YVjjBTHYJnIx8VA1VvSAidD
+  LHp2Yn+7DgUfHXjNdpftTgvWxnzXMFaglNzrNrixGNlkg1sdGDJ+DB/mvptKJUEH
+  WMfOVI98Eo0dx5w+lcP8XGTg6/SY59+PiqNpCmi+T49axQO2XKNlt+ZJsSVaEhEj
+  E2OrkZY+A8RFj07TUjSMv/pmo3AxgVjFoWszDT8pj30CTT3lg3eXXJwlqrH/P9IQ
+  FnwRSt3sR60ahFFJnvHdL1FJl/I0W5nWD6LNEpX7ryaIUIqMhQpQjGDpvG77ntfW
+  A5zhBVWPC7p2k6OaUD6AjlPMJLZh5YbyGaRN4l2Z4oizBGjoq1Qv9QehAgMBAAGj
+  DTALMAkGA1UdEwQCMAAwDQYJKoZIhvcNAQELBQADggEBAOFIxTTiw10jWRKQuRKU
+  KskncSNj3ZxSjwPTOQs++hLjYYYlKA4LbWwokp7u5rTpJP/NHYLHXIda6l/Ne3JG
+  +Mby/vu0TKMX2z+0IQx3MZG7b+4NkH4jg40Q+Y879n0jvOfBplHtJB1UmQYk51fs
+  Hbrb6vvxeLRJ74JZX6t756gZnagzAoLj7DtmTfruUVjD/kRJK8gUCyKMNvN6PH3u
+  5Ls4WwOME+bFdFcxBJjj1LSKGlZoE22mSVlRqHvVXVfM9XTolvw5PequFhiPXYyj
+  ESN9QfRuVeKltTl8NdDgwlYjBBUYR5omuX5LLWUSXuvQK/dYM4ahERf3ivbXMjhF
+  M+Q=
+  -----END CERTIFICATE-----

manifests/site.pp

@@ -1443,33 +1443,102 @@ node 'zuulv3.openstack.org' {
 
 }
 
-# Node-OS: trusty
+# Node-OS: xenial
-node 'zuul.openstack.org' {
+node /^zuul\d+\.openstack\.org$/ {
+  $gerrit_server        = 'review.openstack.org'
+  $gerrit_user          = 'zuul'
+  $gerrit_ssh_host_key  = hiera('gerrit_zuul_user_ssh_key_contents')
+  $zuul_ssh_private_key = hiera('zuul_ssh_private_key_contents')
+  $zuul_url             = "http://zuul.openstack.org/p"
+  $git_email            = 'zuul@openstack.org'
+  $git_name             = 'OpenStack Zuul'
+  $revision             = 'feature/zuulv3'
+
   $gearman_workers = [
-    'nodepool.openstack.org',
+    'ze01.openstack.org',
+    'ze02.openstack.org',
+    'ze03.openstack.org',
+    'ze04.openstack.org',
+    'ze05.openstack.org',
+    'ze06.openstack.org',
+    'ze07.openstack.org',
+    'ze08.openstack.org',
+    'ze09.openstack.org',
+    'ze10.openstack.org',
+    'zm01.openstack.org',
+    'zm02.openstack.org',
+    'zm03.openstack.org',
+    'zm04.openstack.org',
+    'zm05.openstack.org',
+    'zm06.openstack.org',
+    'zm07.openstack.org',
+    'zm08.openstack.org',
   ]
   $iptables_rules = regsubst ($gearman_workers, '^(.*)$', '-m state --state NEW -m tcp -p tcp --dport 4730 -s \1 -j ACCEPT')
 
   class { 'openstack_project::server':
-    iptables_public_tcp_ports => [80, 443],
+    iptables_public_tcp_ports => [79, 80, 443],
     iptables_rules6           => $iptables_rules,
     iptables_rules4           => $iptables_rules,
     sysadmins                 => hiera('sysadmins', []),
   }
 
-  class { 'openstack_project::zuul_prod':
+  class { '::project_config':
-    project_config_repo            => 'https://git.openstack.org/openstack-infra/project-config',
+    url => 'https://git.openstack.org/openstack-infra/project-config',
-    gerrit_server                  => 'review.openstack.org',
+  }
-    gerrit_user                    => 'jenkins',
+
-    gerrit_ssh_host_key            => hiera('gerrit_ssh_rsa_pubkey_contents'),
+  # NOTE(pabelanger): We call ::zuul directly, so we can override all in one
-    zuul_ssh_private_key           => hiera('zuul_ssh_private_key_contents'),
+  # settings.
-    url_pattern                    => 'http://logs.openstack.org/{build.parameters[LOG_PATH]}',
+  class { '::zuul':
+    gerrit_server                => $gerrit_server,
+    gerrit_user                  => $gerrit_user,
+    zuul_ssh_private_key         => $zuul_ssh_private_key,
+    git_email                    => $git_email,
+    git_name                     => $git_name,
+    revision                     => $revision,
+    python_version               => 3,
+    zookeeper_hosts              => 'nodepool.openstack.org:2181',
+    zookeeper_session_timeout    => 40,
+    zuulv3                       => true,
+    connections                  => hiera('zuul_connections', []),
+    connection_secrets           => hiera('zuul_connection_secrets', []),
+    zuul_status_url              => 'http://127.0.0.1:8001/openstack',
+    zuul_web_url                 => 'http://127.0.0.1:9000/openstack',
+    gearman_client_ssl_cert      => hiera('gearman_client_ssl_cert'),
+    gearman_client_ssl_key       => hiera('gearman_client_ssl_key'),
+    gearman_server_ssl_cert      => hiera('gearman_server_ssl_cert'),
+    gearman_server_ssl_key       => hiera('gearman_server_ssl_key'),
+    gearman_ssl_ca               => hiera('gearman_ssl_ca'),
     proxy_ssl_cert_file_contents => hiera('zuul_ssl_cert_file_contents'),
     proxy_ssl_key_file_contents  => hiera('zuul_ssl_key_file_contents'),
-    proxy_ssl_chain_file_contents  => hiera('zuul_ssl_chain_file_contents'),
-    zuul_url                       => 'http://zuul.openstack.org/p',
     statsd_host                  => 'graphite.openstack.org',
   }
+
+  file { "/etc/zuul/github.key":
+    ensure  => present,
+    owner   => 'zuul',
+    group   => 'zuul',
+    mode    => '0600',
+    content => hiera('zuul_github_app_key'),
+    require => File['/etc/zuul'],
+  }
+
+  class { '::zuul::scheduler':
+    layout_dir     => $::project_config::zuul_layout_dir,
+    require        => $::project_config::config_dir,
+    python_version => 3,
+    use_mysql      => true,
+  }
+
+  class { '::zuul::web': }
+  class { '::zuul::fingergw': }
+
+  include bup
+  bup::site { 'rax.ord':
+    backup_user   => 'bup-zuulv3',
+    backup_server => 'backup01.ord.rax.ci.openstack.org',
+  }
+
 }
 
 # Node-OS: xenial

modules/openstack_project/templates/status.vhost.erb

@@ -76,7 +76,7 @@ NameVirtualHost <%= @vhost_name %>:<%= @port %>
       </IfVersion>
   </Directory>
 
-  RedirectMatch temp ^/zuul(.*) http://zuulv3.openstack.org/
+  RedirectMatch temp ^/zuul(.*) http://zuul.openstack.org/
 
   ErrorLog /var/log/apache2/<%= @name %>_error.log
   LogLevel warn

playbooks/remote_puppet_else.yaml

@@ -1,4 +1,4 @@
-- hosts: '!review.openstack.org:!git0*:!zuulv3*:!afs*:!baremetal*:!controller*:!compute*:!puppetmaster*:!disabled'
+- hosts: '!review.openstack.org:!git0*:!zuul[0-9]+.*:!zuulv3*:!afs*:!baremetal*:!controller*:!compute*:!puppetmaster*:!disabled'
   strategy: free
   gather_facts: true
   roles:

playbooks/remote_puppet_git.yaml

@@ -29,6 +29,15 @@
         project_config_ref: "{{ hostvars.localhost.gitinfo.after }}"
       vars:
         puppet_timeout: 60m
+- hosts: "zuul[0-9]+.openstack.org:!disabled"
+  strategy: free
+  gather_facts: true
+  roles:
+    - role: puppet
+      facts:
+        project_config_ref: "{{ hostvars.localhost.gitinfo.after }}"
+      vars:
+        puppet_timeout: 60m
 - hosts: "zuulv3.openstack.org:!disabled"
   strategy: free
   gather_facts: true