diff --git a/doc/source/developer-docs/scripts.rst b/doc/source/developer-docs/scripts.rst index 872024e76b..0d12112480 100644 --- a/doc/source/developer-docs/scripts.rst +++ b/doc/source/developer-docs/scripts.rst @@ -14,30 +14,39 @@ these scripts from the root of the repository clone. For example: Bootstrapping ^^^^^^^^^^^^^ -bootstrap-aio.sh ----------------- - -The ``bootstrap-aio.sh`` script prepares a host for an *all-in-one* (AIO) -deployment for the purposes of development and gating. Create the necessary -partitions, directories, and configurations. Configurable via environment -variables to work with Jenkins. - bootstrap-ansible.sh -------------------- -The ``bootstrap-ansible.sh`` script installs Ansible including core and extras -module repositories and Galaxy roles. +The ``bootstrap-ansible.sh`` script installs Ansible including `core`_ and +`extras`_ module repositories and Galaxy roles. -Configurable environment variables: +While there are several configurable environment variables which this script +uses, the following are commonly used: -* ``ANSIBLE_GIT_RELEASE`` - Version of Ansible to install. -* ``ANSIBLE_ROLE_FILE`` - Galaxy roles to install. Defaults to - contents of ``ansible-role-requirements.yml`` file. +* ``ANSIBLE_GIT_RELEASE`` - The version of Ansible to install. + +* ``ANSIBLE_ROLE_FILE`` - The location of a yaml file which ansible-galaxy can + consume which specifies which roles to download and install. The default + value for this is ``ansible-role-requirements.yml``. The script also creates the ``openstack-ansible`` wrapper tool that provides the variable files to match ``/etc/openstack_deploy/user_*.yml`` as arguments to ``ansible-playbook`` as a convenience. +.. core: https://github.com/ansible/ansible-modules-core +.. extras: https://github.com/ansible/ansible-modules-extras + +bootstrap-aio.sh +---------------- + +The ``bootstrap-aio.sh`` script prepares a host for an `All-In-One`_ (AIO) +deployment for the purposes of development and gating. The script creates the +necessary partitions, directories, and configurations. The script can be +configured using environment variables - more details are provided on the +`All-In-One`_ page. + +.. All-In-One: quickstart-aio + Development and Testing ^^^^^^^^^^^^^^^^^^^^^^^ @@ -83,8 +92,8 @@ Configurable environment variables: ``openstack_tempest_gate.sh`` script, defined in the ``os_tempest`` role. Defaults to ``scenario heat_api cinder_backup``. -PEP8 ----- +Lint Tests +---------- Python coding conventions are tested using `PEP8`_, with the following convention exceptions: @@ -98,11 +107,6 @@ Testing may be done locally by executing: tox -e pep8 -.. PEP8: https://www.python.org/dev/peps/pep-0008/ - -Bashate -------- - Bash coding conventions are tested using `Bashate`_, with the following convention exceptions: @@ -114,10 +118,34 @@ Testing may be done locally by executing: tox -e bashate -.. Bashate: https://github.com/openstack-dev/bashate +Ansible is lint tested using `ansible-lint`_. -Documentation -------------- +Testing may be done locally by executing: + +.. code-block:: bash + + tox -e ansible-lint + +Ansible playbook syntax is tested using ansible-playbook. + +Testing may be done locally by executing: + +.. code-block:: bash + + tox -e ansible-syntax + +A consolidated set of all lint tests may be done locally by executing: + +.. code-block:: bash + + tox -e linters + +.. PEP8: https://www.python.org/dev/peps/pep-0008/ +.. Bashate: https://github.com/openstack-dev/bashate +.. ansible-lint: https://github.com/willthames/ansible-lint + +Documentation Build +------------------- Documentation is developed in reStructuredText_ (RST) and compiled into HTML using Sphinx. @@ -133,43 +161,25 @@ Documentation may be built locally by executing: Gating ^^^^^^ -gate-check-commit.sh --------------------- +Every commit to OpenStack-Ansible is verified by OpenStack-CI through the +following jobs: -The ``gate-check-commit.sh`` script executes a suite of tests necessary for -each commit to the repository. By default, the script runs the bootstrap -scripts, builds an *all-in-one* deployment of OSA, and runs various Tempest -tests on it. +* ``gate-openstack-ansible-docs``: This job executes the + `Documentation Build`_. -Configurable environment variables: +* ``gate-openstack-ansible-linters``: This job executes the `Lint Tests`_. -* ``BOOTSTRAP_AIO`` - Boolean (yes/no) to run AIO bootstrap script. Defaults - to ``yes``. -* ``BOOTSTRAP_ANSIBLE`` - Boolean (yes/no) to run Ansible bootstrip script. - Defaults to ``yes``. -* ``RUN_TEMPEST`` - Boolean (yes/no) to run Tempest tests. Defaults to - ``yes``. +* ``gate-openstack-ansible-dsvm-commit``: This job executes the + ``gate-check-commit.sh`` script which executes a convergence test and then a + functional test. -gate-check-docs.sh ------------------- + The convergence test is the execution of an AIO build + which aims to test the primary code path for a functional environment. The + functional test then executes OpenStack's Tempest testing suite to verify + that the environment that has deployed successfully actually works. -The ``gate-check-docs.sh`` script invokes Sphinx to build the HTML -documentation from RST source. - -gate-check-lint.sh ------------------- - -The ``gate-check-lint.sh`` script executes a suite of tests necessary for each -commit to the repository to verify correct YAML and Python syntax. - -All files that begin with a Python shebang pass through *flake8* which ignores -the following rules due to Ansible conventions: - - * F403 - 'from ansible.module_utils.basic import \*' - * H303 - No wildcard imports - -Ansible playbooks pass through ``ansible-playbook --syntax-check`` -and ``ansible-lint``. + While this script is primarily developed and maintained for use in + OpenStack-CI, it can be used in other environments. -------------- diff --git a/scripts/gate-check-docs.sh b/scripts/gate-check-docs.sh deleted file mode 100755 index 05e305bc6a..0000000000 --- a/scripts/gate-check-docs.sh +++ /dev/null @@ -1,38 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2014, Rackspace US, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -## Shell Opts ---------------------------------------------------------------- -set -e -u -x - - -## Library Check ------------------------------------------------------------- -info_block "Checking for required libraries." 2> /dev/null || source $(dirname ${0})/scripts-library.sh - - -## Main ---------------------------------------------------------------------- -info_block "Running Basic Ansible Lint Check" - - -# Install the development requirements. -if [ -f "test-requirements.txt" ]; then - pip2 install -r test-requirements.txt || pip install -r test-requirements.txt -fi - -pushd doc - info_block "Building HTML docs" - - # Treat warnings as errors. - make html SPHINXOPTS="-W" -popd diff --git a/scripts/gate-check-lint.sh b/scripts/gate-check-lint.sh deleted file mode 100755 index f98b70455f..0000000000 --- a/scripts/gate-check-lint.sh +++ /dev/null @@ -1,52 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2014, Rackspace US, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -## Shell Opts ---------------------------------------------------------------- -set -e -u -x - - -## Library Check ------------------------------------------------------------- -info_block "Checking for required libraries." 2> /dev/null || source $(dirname ${0})/scripts-library.sh - - -## Main ---------------------------------------------------------------------- -info_block "Running Basic Ansible Lint Check" - -# next, bootstrap Ansible -source $(dirname ${0})/bootstrap-ansible.sh - -# Install the development requirements. -if [ -f "test-requirements.txt" ]; then - pip2 install -r test-requirements.txt || pip install -r test-requirements.txt -else - pip2 install ansible-lint || pip install ansible-lint -fi - -# Create keys if they don't already exist. -ssh_key_create - -# Perform our simple sanity checks. -pushd playbooks - echo -e '[all]\nlocalhost ansible_connection=local' | tee local_only_inventory - - # Do a basic syntax check on all playbooks and roles. - info_block "Running Syntax Check" - ansible-playbook -i local_only_inventory --syntax-check *.yml --list-tasks - - # Perform a lint check on all playbooks and roles. - info_block "Running Lint Check" - ansible-lint --version - ansible-lint *.yml -popd diff --git a/tox.ini b/tox.ini index 04190d89b0..1ec7fa59fe 100644 --- a/tox.ini +++ b/tox.ini @@ -1,7 +1,7 @@ [tox] minversion = 1.6 skipsdist = True -envlist = docs,pep8,bashate,releasenotes +envlist = docs,linters,releasenotes [testenv] usedevelop = True @@ -42,23 +42,75 @@ commands = [flake8] # Ignores the following rules due to how ansible modules work in general -# F403 'from ansible.module_utils.basic import *' used; unable to detect undefined names +# F403 'from ansible.module_utils.basic import *' used; +# unable to detect undefined names # H303 No wildcard (*) import. ignore=F403,H303 -# Run bashate check for all bash scripts -# Ignores the following rules: -# E003: Indent not multiple of 4 (we prefer to use multiples of 2) [testenv:bashate] -deps = bashate +deps = + bashate whitelist_externals = bash commands = - bash -c "grep -Irl \ - -e '!/usr/bin/env bash' \ - -e '!/bin/bash' \ - -e '!/bin/sh' \ - --exclude-dir '.*' \ - --exclude-dir '*.egg' \ - --exclude-dir '*.egg-info' \ - --exclude 'tox.ini' \ - {toxinidir} | xargs bashate --verbose --ignore=E003" + # Run bashate check for all bash scripts + # Ignores the following rules: + # E003: Indent not multiple of 4 (we prefer to use multiples of 2) + # E005: File does not begin with #! or have .sh prefix + # E006: Line longer than 79 columns + bash -c "grep -Irl \ + -e '!/usr/bin/env bash' \ + -e '!/bin/bash' \ + -e '!/bin/sh' \ + --exclude-dir '.*' \ + --exclude-dir '*.egg' \ + --exclude-dir '*.egg-info' \ + --exclude 'tox.ini' \ + {toxinidir} | xargs bashate --verbose --ignore=E003,E005,E006" + +[testenv:ansible-lint] +deps = + ansible>1.9,<2.0 + ansible-lint +whitelist_externals = bash +commands = + # Perform an Ansible lint check + bash -c "ansible-lint {toxinidir}/playbooks/*.yml" + +[testenv:ansible-syntax] +deps = + ansible>1.9,<2.0 +whitelist_externals = bash +commands = + # Perform an Ansible syntax check + bash -c "mkdir -p {envtmpdir}/ansible; \ + export ANSIBLE_CONFIG={envtmpdir}/ansible/ansible.cfg; \ + export ANSIBLE_ROLES_PATH={envtmpdir}/ansible/roles; \ + sed 's|/etc/ansible|{envtmpdir}/ansible|' \ + {toxinidir}/tests/ansible.cfg | \ + tee {envtmpdir}/ansible/ansible.cfg; \ + sed 's|path: /etc/ansible|path: {envtmpdir}/ansible|' \ + {toxinidir}/ansible-role-requirements.yml | \ + tee {envtmpdir}/ansible/ansible-role-requirements.yml; \ + ansible-galaxy install \ + --role-file={envtmpdir}/ansible/ansible-role-requirements.yml \ + --roles-path={envtmpdir}/ansible/roles \ + --ignore-errors \ + --force; \ + ansible-playbook -i 'localhost ansible-connection=local,' \ + --syntax-check \ + --list-tasks \ + {toxinidir}/playbooks/*.yml" + +[testenv:linters] +deps = + {[testenv:pep8]deps} + {[testenv:bashate]deps} + {[testenv:ansible-lint]deps} + # note that the ansible-syntax deps are omitted on purpose as the + # ansible-lint env contains duplicate items in its dep list +whitelist_externals = bash +commands = + {[testenv:pep8]commands} + {[testenv:bashate]commands} + {[testenv:ansible-lint]commands} + {[testenv:ansible-syntax]commands}