Always install Ansible with pip

Instaling and using Ansible from source for bifrost has several
drawbacks, mainly due to how Ansible's 'ansible/hacking/env-setup'
script mangles with PATH and PYTHONPATH, which complicates running it as
part of other scripts. Besides, cloning the whole repo and it's
submodules is somewhat longer.

The main reason why we were doing that at all was a necessity to install
some additional Ansible modules from newer Ansible versions, which we
dropped right into the source of Ansible code - but this does not have to
be so.

Luckily for us, all Ansible versions we target to support can load
modules from 'library' directory next to playbooks/roles,
and we already use that for 'os_ironic_facts' module.
The need to install a particular module can be assessed by running
ad-hoc 'ansible' command against localhost with the module in question
and without any arguments ('ansible localhost -m <module>'):
- if the module is available in Ansible, the stderr will contain
  "changed" substring (as part of the standard module output)
- if the module is absent form Ansible, "changed" string will be absent
  from stderr too, in which case we can download the module from github
  directly into 'playbooks/library' directory.

This patch removes possibility of installing Ansible from source, and
always installs a released Ansible version via pip.
If not installed into venv, Ansible will be installed in user's ~/.local
directory via 'pip install --user'.
The missing but needed modules are downloaded as described above.

Some level of backward compatibility is provided:
- when the ANSIBLE_GIT_BRANCH has form of 'stable-X.Y', the
  env-setup.sh script will do the next best thing and install latest
  available Ansible version of X.Y.w.z

Also, ANSIBLE_PIP_VERSION can now accept a full pip version specifier:
- if ANSIBLE_PIP_VERSION starts with a digit, this exact version will be
  installed (as 'ansible==X.Y.W.Z')
- otherwize this whole variable is assigned as Ansible version specifier
  for pip, e.g

    env ANSIBLE_PIP_VERSION="<2.2" env-setup.sh

  will result in pip being called as

    pip install -U "ansible<2.2"

Closes-Bug: #1663562
Change-Id: I2c9f47abbbb6740d03978f684ad2c876749655b7

.gitignore

@@ -61,3 +61,9 @@ releasenotes/build
 
 # Ansible retry files
 *.retry
+
+# Ansible modules that may be downloaded during env-setup.sh
+playbooks/library/os_ironic.py
+playbooks/library/os_ironic_node.py
+playbooks/library/os_ironic_inspect.py
+playbooks/library/os_keystone_service.py

README.rst

@@ -132,21 +132,34 @@ If you are running the installation behind a proxy, export the
 environment variables ``http_proxy`` and ``https_proxy`` so that
 Ansible will use these proxy settings.
 
-The below script ``env-setup.sh`` will install ansible and all of bifrost's
-dependencies. You can configure the ansible installation location by setting
-``ANSIBLE_INSTALL_ROOT`` environment variable. The default value will be
-``/opt/stack``.
+The recommended path for use is with a local Ansible installation, and to
+install the library requirements. Alternatively the ``env-setup.sh`` script
+will install ansible and all of bifrost's dependencies.
 
-Note:
+If you use ``env-setup.sh``, ansible will be installed along
+with its missing Python dependencies into user's ``~/.local`` directory.
 
-  Only ansible installation location will be moved as part of the
-  environment variable.  The other components will continue to be cloned under
-  ``/opt/stack``
+Warning::
 
-Then run::
+  Use of the ``env-setup.sh`` script can squash an existing
+  Ansible installation, and is intended primarily for development
+  and testing.
+
+Note::
+
+  The next setup steps require elevated privilges, and might need to
+  be executed with the ``sudo`` command, depending on the access rights
+  of the user executing the command.
+
+If using the environment setup script::
 
   bash ./scripts/env-setup.sh
-  source ${ANSIBLE_INSTALL_ROOT}/ansible/hacking/env-setup
+  export PATH=${HOME}/.local/bin:${PATH}
+  cd playbooks
+
+Otherwise::
+
+  pip install -r requirements.txt
   cd playbooks
 
 The second part is an Ansible playbook that installs and configures ironic

playbooks/roles/bifrost-prep-for-install/tasks/main.yml

@@ -14,6 +14,7 @@
 # limitations under the License.
 ---
 - name: ensure installation root folder exists
+  become: yes
   file:
     state: directory
     dest: "{{ git_root }}"

releasenotes/notes/ansible-pip-install-2b66bd82ce9ed4f6.yaml

@@ -0,0 +1,56 @@
+---
+features:
+  - |
+    Bifrost now prefers to use a system with Ansible already installed.
+    When this is the case, execution of the ``env-setup.sh`` script is not
+    required as it is geared for development and testing use of bifrost.
+
+    In order to use the playbooks on a system with Ansible already installed,
+    the library requirements must be installed prior to playbook
+    execution:
+
+        ``pip install -r requirements.txt``
+
+    Administrative privileges may be required if the packages
+    must be installed system wide.
+
+  - |
+    The environment setup script will now attempt to install bifrost from PyPI
+    instead of using a stable branch. This is to address stability issues
+    with Ansible stable branches.
+
+    If not requested to be installed into virtualenv, Ansible will be installed
+    into user's ``~/.local`` directory to not clobber possibly existing
+    system installation.
+    To use such installed Ansible, modifications of ``$PATH``
+    environment variable might be required to include ``~/.local/bin`` path.
+
+    Some backwards compatibility is provided via the use of the
+    ``ANSIBLE_GIT_BRANCH`` variable, where a user can define ``stable-X.Y``
+    and the latest available version in that series will be installed.
+    To install the Ansible 2.1 series as part of the env-setup script,
+    execute ``env ANSIBLE_GIT_BRANCH="stable-2.1" scripts/env-setup``.
+
+    Similarly, ``ANSIBLE_PIP_VERSION`` can be utilized to specify
+    the exact version, or range of version desired. Example:
+
+        ``ANSIBLE_PIP_VERSION=2.1.0.1`` or ``ANSIBLE_PIP_VERSION=<2.2``
+
+issues:
+  - |
+    If installing bifrost in a virtualenv (venv) and running playbooks
+    against localhost, you must install the basic python requirements
+    on a system-wide level due to the operating behavior of Ansible.
+
+fixes:
+  - |
+    Due to breaking change in the stable branch tags utilized with Ansible,
+    bifrost now utilizes installation of Ansible from PyPI.
+
+deprecations:
+  - |
+    The ``ANSIBLE_INSTALL_ROOT`` variable has been deprecated and is used
+    only to raise a warning for third party scripts.
+  - |
+    The ``ANSIBLE_FROM_PYPI`` variable no longer has any effect, as Ansible
+    is always installed from PyPI now.

releasenotes/notes/ocata-summary-040e557460bab2bc.yaml

@@ -5,7 +5,8 @@ prelude: >
     Coupled with a number of fixes, and improvements, users upgrading should
     take the time to read the entire release notes. A few highlights are below:
 
-        * Bifrost now installs and utilizes Ansible 2.1 by default.
+        * Bifrost now installs and utilizes Ansible 2.1 by default from
+          PyPI.
         * Ironic's default of modifying a pre-existing ironic.conf upon the
           installation being re-executed, has been changed to a utilize a
           template file.

scripts/ansible-pip-str.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python
+
+# Copyright (c) 2017 Mirantis Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helper script to choose which ansible version to install for bifrost"""
+
+from __future__ import print_function
+import sys
+
+in_str = sys.argv[1]
+HELP_MSG = ("Unsupported version or format %s - "
+            "Supporting format [stable-]MAJ.MIN where MAJ.MIN is 1.9 or 2.x"
+            % in_str)
+
+if in_str.startswith('stable-'):
+    in_version = in_str.split('stable-')[1]
+else:
+    if in_str[0].isdecimal():
+        print("ansible==%s" % in_str)
+    else:
+        print("ansible%s" % in_str)
+    sys.exit(0)
+
+if len(in_version) != 3 and in_version[1] != '.':
+    print(HELP_MSG)
+    sys.exit(1)
+else:
+    maj_version = in_version[0]
+    try:
+        min_version = int(in_version[2])
+    except ValueError:
+        print(HELP_MSG)
+        sys.exit(1)
+
+if maj_version == '1' and min_version == 9:
+    upper_bound = '2.0'
+elif maj_version == '2':
+    upper_bound = '2.%i' % (min_version + 1)
+else:
+    print(HELP_MSG)
+    sys.exit(1)
+
+print("ansible<%s" % upper_bound)

scripts/env-setup.sh

@@ -1,12 +1,73 @@
 #!/bin/bash
 set -eu
 
-ANSIBLE_FROM_PYPI=${ANSIBLE_FROM_PYPI:-"False"}
-
 source $(dirname $0)/install-deps.sh
+# NOTE(pas-ha) the above exports some useful variables like
+# $PYTHON , $PIP and $VENV depending on venv install or not
 
-if [[ "$ANSIBLE_FROM_PYPI" == "True" ]]; then
-    source $(dirname $0)/install-ansible-pip.sh
+ANSIBLE_PIP_VERSION=${ANSIBLE_PIP_VERSION:-${ANSIBLE_GIT_BRANCH:-stable-2.1}}
+
+ANSIBLE_PIP_STRING=$(${PYTHON} $(dirname $0)/ansible-pip-str.py ${ANSIBLE_PIP_VERSION})
+
+if [ -n "${VENV-}" ]; then
+    ${PIP} install --upgrade "${ANSIBLE_PIP_STRING}"
+    ANSIBLE=${VENV}/bin/ansible
 else
-    source $(dirname $0)/install-ansible-source.sh
+    ${PIP} install --user --upgrade "${ANSIBLE_PIP_STRING}"
+    ANSIBLE=${HOME}/.local/bin/ansible
 fi
+
+PLAYBOOKS_LIBRARY_PATH=$(dirname $0)/../playbooks/library
+
+function check_get_module () {
+    local module=${1}
+    local module_url_base=${2}
+    ${ANSIBLE} localhost -m ${module} | grep "changed" || \
+        wget "${module_url_base}/${module}.py" -O "${PLAYBOOKS_LIBRARY_PATH}/${module}.py"
+}
+
+# Note(TheJulia): These files should be in the ansible library folder
+# and this functionality exists for a level of ansible 1.9.x
+# backwards compatability although the modules were developed
+# for Ansible 2.0.
+check_get_module os_ironic \
+    https://raw.githubusercontent.com/ansible/ansible-modules-core/stable-2.0/cloud/openstack
+
+check_get_module os_ironic_node \
+    https://raw.githubusercontent.com/ansible/ansible-modules-core/stable-2.0/cloud/openstack
+
+# os_ironic_inspect has appeared in Ansible 2.1
+check_get_module os_ironic_inspect \
+    https://raw.githubusercontent.com/ansible/ansible-modules-extras/stable-2.1/cloud/openstack
+
+# os_keystone_service has appeared in Ansible 2.2
+check_get_module os_keystone_service \
+    https://raw.githubusercontent.com/ansible/ansible-modules-extras/stable-2.2/cloud/openstack
+
+# NOTE(pas-ha) the following is a temporary workaround for third-party CI
+# scripts that try to source Ansible's hacking/env-setup
+# after running this very script
+# TODO(pas-ha) remove after deprecation (in Pike?) and when third-party CIs
+# (in particular OPNFV) are fixed
+ANSIBLE_INSTALL_ROOT=${ANSIBLE_INSTALL_ROOT:-/opt/stack}
+u=$(whoami)
+g=$(groups | awk '{print $1}')
+if [ ! -d ${ANSIBLE_INSTALL_ROOT} ]; then
+    mkdir -p ${ANSIBLE_INSTALL_ROOT} || (sudo mkdir -p ${ANSIBLE_INSTALL_ROOT})
+fi
+sudo -H chown -R $u:$g ${ANSIBLE_INSTALL_ROOT}
+mkdir -p ${ANSIBLE_INSTALL_ROOT}/ansible/hacking
+echo "echo Sourcing this file is no longer needed! Ansible is always installed from PyPI" > ${ANSIBLE_INSTALL_ROOT}/ansible/hacking/env-setup
+
+echo
+echo "To use bifrost, do"
+
+if [ -n "${VENV-}" ]; then
+    echo "source ${VENV}/bin/activate"
+else
+    echo "Prepend ~/.local/bin to your PATH if it is not that way already.."
+    echo ".. or use full path to local Ansible at ~/.local/bin/ansible-playbook"
+fi
+echo "source env-vars"
+echo "Then run playbooks as normal."
+echo

scripts/install-ansible-pip.sh

@@ -1,15 +0,0 @@
-#!/bin/bash
-set -eu
-
-ANSIBLE_PIP_VERSION=${ANSIBLE_PIP_VERSION:-"2.2"}
-sudo -H -E ${PIP} install "ansible==$ANSIBLE_PIP_VERSION"
-
-echo
-echo "To use bifrost, do"
-
-if [ -n "${VENV-}" ]; then
-    echo "source ${VENV}/bin/activate"
-fi
-echo "source env-vars"
-echo "Then run playbooks as normal."
-echo

scripts/install-ansible-source.sh

@@ -1,74 +0,0 @@
-#!/bin/bash
-set -eu
-
-ANSIBLE_GIT_URL=${ANSIBLE_GIT_URL:-https://github.com/ansible/ansible.git}
-ANSIBLE_GIT_BRANCH=${ANSIBLE_GIT_BRANCH:-stable-2.1}
-ANSIBLE_INSTALL_ROOT=${ANSIBLE_INSTALL_ROOT:-/opt/stack}
-
-function check_get_module () {
-    local file=${1}
-    local url=${2}
-    if [ ! -e ${file} ]; then
-        wget -O ${file} ${url}
-    fi
-}
-
-u=$(whoami)
-g=$(groups | awk '{print $1}')
-
-if [ ! -d ${ANSIBLE_INSTALL_ROOT} ]; then
-    mkdir -p ${ANSIBLE_INSTALL_ROOT} || (sudo mkdir -p ${ANSIBLE_INSTALL_ROOT})
-fi
-sudo -H chown -R $u:$g ${ANSIBLE_INSTALL_ROOT}
-cd ${ANSIBLE_INSTALL_ROOT}
-
-if [ ! -d ansible ]; then
-    git clone $ANSIBLE_GIT_URL --recursive -b $ANSIBLE_GIT_BRANCH
-    cd ansible
-else
-    cd ansible
-    git remote update origin --prune
-    git fetch --tags
-    git checkout $ANSIBLE_GIT_BRANCH
-    git pull --rebase origin $ANSIBLE_GIT_BRANCH
-    git submodule update --init --recursive
-    git fetch
-fi
-# Note(TheJulia): These files should be in the ansible folder
-# and this functionality exists for a level of ansible 1.9.x
-# backwards compatability although the modules were developed
-# for Ansible 2.0.
-
-check_get_module `pwd`/lib/ansible/modules/core/cloud/openstack/os_ironic.py \
-    https://raw.githubusercontent.com/ansible/ansible-modules-core/stable-2.0/cloud/openstack/os_ironic.py
-check_get_module `pwd`/lib/ansible/modules/core/cloud/openstack/os_ironic_node.py \
-    https://raw.githubusercontent.com/ansible/ansible-modules-core/stable-2.0/cloud/openstack/os_ironic_node.py
-
-# os_ironic_inspect has appeared in Ansible 2.1
-check_get_module `pwd`/lib/ansible/modules/extras/cloud/openstack/os_ironic_inspect.py \
-    https://raw.githubusercontent.com/ansible/ansible-modules-extras/stable-2.1/cloud/os_ironic_inspect.py
-
-# os_keystone_service has appeared in Ansible 2.2
-check_get_module `pwd`/lib/ansible/modules/extras/cloud/openstack/os_keystone_service.py \
-    https://raw.githubusercontent.com/ansible/ansible-modules-extras/stable-2.2/cloud/openstack/os_keystone_service.py
-
-sudo -H -E ${PIP} install --upgrade ${ANSIBLE_INSTALL_ROOT}/ansible
-
-if [ -n "${VENV-}" ]; then
-    echo
-    echo "To use bifrost, do"
-
-    echo "source ${VENV}/bin/activate"
-    echo "source env-vars"
-    echo "Then run playbooks as normal."
-    echo
-else
-    echo
-    echo "If you're using this script directly, execute the"
-    echo "following commands to update your shell."
-    echo
-    echo "source env-vars"
-    echo "source ${ANSIBLE_INSTALL_ROOT}/ansible/hacking/env-setup"
-    echo
-fi
-

scripts/test-bifrost.sh

@@ -55,24 +55,21 @@ elif [ $SOURCE = "test-bifrost-keystone-auth.sh" ]; then
      ENABLE_KEYSTONE="true"
 fi
 
-# Source Ansible
-# NOTE(TheJulia): Ansible stable-1.9 source method tosses an error deep
-# under the hood which -x will detect, so for this step, we need to suspend
-# and then re-enable the feature.
-set +x +o nounset
 if [ ${USE_VENV} = "true" ]; then
     export VENV=/opt/stack/bifrost
     export PATH=${VENV}/bin:${PATH}
     $SCRIPT_HOME/env-setup.sh
+    # Note(cinerama): activate is not compatible with "set -u";
+    # disable it just for this line.
+    set +u
     source /opt/stack/bifrost/bin/activate
+    set -u
     ANSIBLE=${VENV}/bin/ansible-playbook
     ENABLE_VENV="true"
 else
     $SCRIPT_HOME/env-setup.sh
-    source ${ANSIBLE_INSTALL_ROOT}/ansible/hacking/env-setup
-    ANSIBLE=$(which ansible-playbook)
+    ANSIBLE=${HOME}/.local/bin/ansible-playbook
 fi
-set -x -o nounset
 
 # Adjust options for DHCP, VM, or Keystone tests
 if [ ${USE_DHCP} = "true" ]; then