Split config from projects list

The projects list is a common list for devs to interact with. The config
in the list is not, but the config in the list means the file needs to
be in an erb template.

Split the two concerns, similar to zuul. Put the config in a config file
and the project data in a yaml file.

Change-Id: I708b8655b4b1ce377f3b7369e987418c1d72d977

doc/source/gerrit.rst

@@ -24,7 +24,8 @@ At a Glance
   * :file:`modules/openstack_project/manifests/review.pp`
   * :file:`modules/openstack_project/manifests/review_dev.pp`
 :Configuration:
-  * :file:`modules/openstack_project/templates/review.projects.yaml.erb`
+  * :file:`modules/openstack_project/templates/review.projects.ini.erb`
+  * :file:`modules/openstack_project/files/review.projects.yaml`
 :Projects:
   * http://code.google.com/p/gerrit/
 :Bugs:
@@ -136,7 +137,7 @@ account name and add ssh keys - you'll need those.
 Once you've created your groups you should create the
 ``openstack-project-creator`` account by hand (the account name is
 referenced from
-:file:`modules/openstack_project/templates/review.projects.yaml.erb`)
+:file:`modules/openstack_project/templates/review.projects.ini.erb`)
 using::
 
   cat $pubkey | ssh -p 29418 $USER@$HOST gerrit create-account \

doc/source/jeepyb.rst

@@ -20,7 +20,8 @@ At a Glance
   * :file:`modules/openstack_project/manifests/review.pp`
   * :file:`modules/openstack_project/manifests/review_dev.pp`
 :Configuration:
-  * :file:`modules/openstack_project/templates/review.projects.yaml.erb`
+  * :file:`modules/openstack_project/templates/review.projects.ini.erb`
+  * :file:`modules/openstack_project/files/review.projects.yaml`
   * :file:`modules/openstack_project/files/pypi-mirror.yaml`
 :Projects:
   * https://git.openstack.org/cgit/openstack-infra/jeepyb
@@ -36,25 +37,28 @@ project in Gerrit, create the new project on Github, create a local
 git replica on the Gerrit host, configure the project Access Controls,
 and create new groups in Gerrit.
 
+The global configuration data needed for ``manage-projects`` to know how to
+connect to things or how to operate is in
+:file:`modules/openstack_project/templates/review.projects.ini.erb`.
+
+#. Config values::
+
+     [projects]
+     homepage=http://example.org
+     local-git-dir=/var/lib/git
+     gerrit-host=review.example.org
+     gerrit-user=example-project-creator
+     gerrit-key=/home/gerrit2/.ssh/example_project_id_rsa
+     github-config=/etc/github/github-projects.secure.config
+     has-wiki=False
+     has-issues=False
+     has-pull-requests=False
+     has-downloads=False
+
 OpenStack Gerrit projects are configured in the
-:file:`modules/openstack_project/templates/review.projects.yaml.erb`.
+:file:`modules/openstack_project/files/review.projects.yaml`.
 file.  When this file is updated, ``manage-projects`` is run
-automatically.  This file contains two sections, the first is a set of
+automatically.
-default config values that each project can override, and the second
-is a list of projects (each may contain their own overrides).
-
-#. Config default values::
-
-     - homepage: http://example.org
-       local-git-dir: /var/lib/git
-       gerrit-host: review.example.org
-       gerrit-user: example-project-creator
-       gerrit-key: /home/gerrit2/.ssh/example_project_id_rsa
-       github-config: /etc/github/github-projects.secure.config
-       has-wiki: False
-       has-issues: False
-       has-pull-requests: False
-       has-downloads: False
 
 #. Project definition::
 
@@ -69,7 +73,7 @@ is a list of projects (each may contain their own overrides).
 The above config gives puppet and its related scripts enough information
 to create new projects, but not enough to add access controls to each
 project. To add access control you need to have have an ``acl-config``
-option for the project in ``review.projects.yaml.erb`` file. That option
+option for the project in ``review.projects.yaml`` file. That option
 should have a value that is a path to the ``project.config`` for that
 project.
 
@@ -99,7 +103,7 @@ a single project you will want to do the following:
              mergeContent = true
 
 #. Add a project entry for the project in
-   ``modules/openstack_project/templates/review.projects.yaml.erb``.::
+   ``modules/openstack_project/files/review.projects.yaml``.::
 
      - project: openstack/project-name
        acl-config: /home/gerrit2/acls/project-name.config

doc/source/running-your-own.rst

@@ -179,8 +179,9 @@ Migrate the manifests:
     . file to let git add it).  ``http://ci.openstack.org/stackforge.html``
     covers how it gets populated when your infrastructure is working.
 
-  * Ditto projects.yaml, which is passed in from your review.pp - something like
+  * Ditto projects.yaml and projects.ini, which is passed in from your
-    $PROJECT/templates/review.projects.yaml.erb
+    review.pp - something like $PROJECT/files/review.projects.yaml
+    and $PROJECT/templates/review.projects.ini.erb
 
   * set_agreements is a database migration tool for gerrit CLAs; not needed
     unless you have CLAs.

doc/source/stackforge.rst

@@ -49,10 +49,9 @@ with simple changes to the openstack-infra/config repository. To start make
 sure you have cloned the openstack-infra/config repository
 ``git clone https://git.openstack.org/openstack-infra/config``.
 
-First you need to add your StackForge project to the master project
+First you need to add your StackForge project to the master project list.
-list. Edit
+Edit ``modules/openstack_project/files/review.projects.yaml`` and add a
-``modules/openstack_project/templates/review.projects.yaml.erb``
+new section for your project in alphabetical order within the file.
-and add a new section for your project in alphabetical order within the file.
 It should look something like::
 
   - project: stackforge/project-name

modules/openstack_project/files/nodepool/scripts/cache_git_repos.py

@@ -22,7 +22,7 @@ import urllib2
 from common import run_local
 
 URL = ('http://git.openstack.org/cgit/openstack-infra/config/plain/'
-       'modules/openstack_project/templates/review.projects.yaml.erb')
+       'modules/openstack_project/files/review.projects.yaml')
 PROJECT_RE = re.compile('^-?\s+project:\s+(.*)$')
 
 

modules/openstack_project/files/review-dev.projects.yaml

@@ -0,0 +1,8 @@
+- project: gtest-org/gtest
+- project: completely-bogus-org/test
+- project: gtest-org/test
+- project: gtest-org/test2
+- project: gtest-org/test3
+- project: gtest-org/test-manage-project
+  acl-config: /home/gerrit2/acls/test-manage-project.config
+- project: gtest-org/test.hyphen

modules/openstack_project/files/review.projects.yaml

@@ -1,16 +1,3 @@
-- homepage: http://openstack.org
-  acl-dir: /home/gerrit2/acls
-  local-git-dir: <%= local_git_dir %>
-  gerrit-host: <%= fqdn %>
-  gerrit-user: openstack-project-creator
-  gerrit-committer: OpenStack Project Creator <openstack-infra@lists.openstack.org>
-  gerrit-key: <%= ssh_project_key %>
-  github-config: /etc/github/github-projects.secure.config
-  has-wiki: False
-  has-issues: False
-  has-pull-requests: False
-  has-downloads: False
----
 - project: openstack-dev/cookiecutter
   description: Cookiecutter Template for new OpenStack projects
   upstream: git://github.com/emonty/cookiecutter-openstack.git

modules/openstack_project/manifests/gerrit.pp

@@ -42,6 +42,7 @@ class openstack_project::gerrit (
   $script_key_file = '/home/gerrit2/.ssh/id_rsa',
   $script_logging_conf = '/home/gerrit2/.sync_logging.conf',
   $projects_file = 'UNDEF',
+  $projects_config = 'UNDEF',
   $github_username = '',
   $github_oauth_token = '',
   $github_project_username = '',
@@ -313,7 +314,17 @@ class openstack_project::gerrit (
       owner   => 'gerrit2',
       group   => 'gerrit2',
       mode    => '0444',
-      content => template($projects_file),
+      source  => $projects_file,
+      replace => true,
+      require => Class['::gerrit'],
+    }
+
+    file { '/home/gerrit2/projects.ini':
+      ensure  => present,
+      owner   => 'gerrit2',
+      group   => 'gerrit2',
+      mode    => '0444',
+      content => template($projects_config),
       replace => true,
       require => Class['::gerrit'],
     }

modules/openstack_project/manifests/git_backend.pp

@@ -78,7 +78,7 @@ class openstack_project::git_backend (
     owner   => 'cgit',
     group   => 'cgit',
     mode    => '0444',
-    content => template('openstack_project/review.projects.yaml.erb'),
+    source  => 'puppet:///modules/openstack_project/review.projects.yaml',
     replace => true,
   }
 

modules/openstack_project/manifests/review.pp

@@ -112,7 +112,9 @@ class openstack_project::review (
     script_key_file                 => '/home/gerrit2/.ssh/launchpadsync_rsa',
     script_logging_conf             => '/home/gerrit2/.sync_logging.conf',
     projects_file                   =>
-      'openstack_project/review.projects.yaml.erb',
+      'puppet:///modules/openstack_project/review.projects.yaml',
+    projects_config                 =>
+      'openstack_project/review.projects.ini',
     github_username                 => 'openstack-gerrit',
     github_oauth_token              => $github_oauth_token,
     github_project_username         => $github_project_username,

modules/openstack_project/manifests/review_dev.pp

@@ -59,7 +59,9 @@ class openstack_project::review_dev (
     script_key_file                 => '/home/gerrit2/.ssh/launchpadsync_rsa',
     script_logging_conf             => '/home/gerrit2/.sync_logging.conf',
     projects_file                   =>
-      'openstack_project/review-dev.projects.yaml.erb',
+      'puppet:///modules/openstack_project/review-dev.projects.yaml',
+    projects_config                 =>
+      'openstack_project/review-dev.projects.ini.erb',
     github_username                 => 'openstack-gerrit-dev',
     github_oauth_token              => $github_oauth_token,
     github_project_username         => $github_project_username,

modules/openstack_project/templates/review-dev.projects.ini.erb

@@ -0,0 +1,11 @@
+[projects]
+homepage=http://openstack.org
+acl-dir=/home/gerrit2/acls
+local-git-dir=<%= local_git_dir %>
+gerrit-host=<%= fqdn %>
+gerrit-user=openstack-dev-project-creator
+gerrit-key=<%= ssh_project_key %>
+github-config=/etc/github/github-projects.secure.config
+has-wiki=False
+has-issues=False
+has-downloads=False

modules/openstack_project/templates/review-dev.projects.yaml.erb

@@ -1,19 +0,0 @@
-- homepage: http://openstack.org
-  acl-dir: /home/gerrit2/acls
-  local-git-dir: <%= local_git_dir %>
-  gerrit-host: <%= fqdn %>
-  gerrit-user: openstack-dev-project-creator
-  gerrit-key: <%= ssh_project_key %>
-  github-config: /etc/github/github-projects.secure.config
-  has-wiki: False
-  has-issues: False
-  has-downloads: False
----
-- project: gtest-org/gtest
-- project: completely-bogus-org/test
-- project: gtest-org/test
-- project: gtest-org/test2
-- project: gtest-org/test3
-- project: gtest-org/test-manage-project
-  acl-config: /home/gerrit2/acls/test-manage-project.config
-- project: gtest-org/test.hyphen

modules/openstack_project/templates/review.projects.ini.erb

@@ -0,0 +1,13 @@
+[projects]
+homepage=http://openstack.org
+acl-dir=/home/gerrit2/acls
+local-git-dir=<%= local_git_dir %>
+gerrit-host=<%= fqdn %>
+gerrit-user=openstack-project-creator
+gerrit-committer=OpenStack Project Creator <openstack-infra@lists.openstack.org>
+gerrit-key=<%= ssh_project_key %>
+github-config=/etc/github/github-projects.secure.config
+has-wiki=False
+has-issues=False
+has-pull-requests=False
+has-downloads=False

tools/check_projects_yaml_alphabetized.sh

@@ -7,13 +7,7 @@ export TMPDIR=`/bin/mktemp -d`
 trap "rm -rf $TMPDIR" EXIT
 
 pushd $TMPDIR
-
+PROJECTS_LIST=$OLDPWD/modules/openstack_project/files/review.projects.yaml
-if [ -f $OLDPWD/modules/openstack_project/templates/review.projects.yaml.erb ]
-then
-    PROJECTS_LIST=$OLDPWD/modules/openstack_project/templates/review.projects.yaml.erb
-else
-    PROJECTS_LIST=$OLDPWD/modules/openstack_project/files/review.projects.yaml
-fi
 
 sed -e '/^- project: /!d' -e 's/^- project: //' $PROJECTS_LIST > projects_list