diff --git a/.gitignore b/.gitignore index abb5e7712..81adab414 100644 --- a/.gitignore +++ b/.gitignore @@ -61,3 +61,9 @@ releasenotes/build # Ansible retry files *.retry + +# Ansible modules that may be downloaded during env-setup.sh +playbooks/library/os_ironic.py +playbooks/library/os_ironic_node.py +playbooks/library/os_ironic_inspect.py +playbooks/library/os_keystone_service.py diff --git a/README.rst b/README.rst index 49dc547ec..c8555f751 100644 --- a/README.rst +++ b/README.rst @@ -132,21 +132,34 @@ If you are running the installation behind a proxy, export the environment variables ``http_proxy`` and ``https_proxy`` so that Ansible will use these proxy settings. -The below script ``env-setup.sh`` will install ansible and all of bifrost's -dependencies. You can configure the ansible installation location by setting -``ANSIBLE_INSTALL_ROOT`` environment variable. The default value will be -``/opt/stack``. +The recommended path for use is with a local Ansible installation, and to +install the library requirements. Alternatively the ``env-setup.sh`` script +will install ansible and all of bifrost's dependencies. -Note: +If you use ``env-setup.sh``, ansible will be installed along +with its missing Python dependencies into user's ``~/.local`` directory. - Only ansible installation location will be moved as part of the - environment variable. The other components will continue to be cloned under - ``/opt/stack`` +Warning:: -Then run:: + Use of the ``env-setup.sh`` script can squash an existing + Ansible installation, and is intended primarily for development + and testing. + +Note:: + + The next setup steps require elevated privilges, and might need to + be executed with the ``sudo`` command, depending on the access rights + of the user executing the command. + +If using the environment setup script:: bash ./scripts/env-setup.sh - source ${ANSIBLE_INSTALL_ROOT}/ansible/hacking/env-setup + export PATH=${HOME}/.local/bin:${PATH} + cd playbooks + +Otherwise:: + + pip install -r requirements.txt cd playbooks The second part is an Ansible playbook that installs and configures ironic diff --git a/playbooks/roles/bifrost-prep-for-install/tasks/main.yml b/playbooks/roles/bifrost-prep-for-install/tasks/main.yml index 9429860ee..cf2cc9b1b 100644 --- a/playbooks/roles/bifrost-prep-for-install/tasks/main.yml +++ b/playbooks/roles/bifrost-prep-for-install/tasks/main.yml @@ -14,6 +14,7 @@ # limitations under the License. --- - name: ensure installation root folder exists + become: yes file: state: directory dest: "{{ git_root }}" diff --git a/releasenotes/notes/ansible-pip-install-2b66bd82ce9ed4f6.yaml b/releasenotes/notes/ansible-pip-install-2b66bd82ce9ed4f6.yaml new file mode 100644 index 000000000..ac70b509c --- /dev/null +++ b/releasenotes/notes/ansible-pip-install-2b66bd82ce9ed4f6.yaml @@ -0,0 +1,56 @@ +--- +features: + - | + Bifrost now prefers to use a system with Ansible already installed. + When this is the case, execution of the ``env-setup.sh`` script is not + required as it is geared for development and testing use of bifrost. + + In order to use the playbooks on a system with Ansible already installed, + the library requirements must be installed prior to playbook + execution: + + ``pip install -r requirements.txt`` + + Administrative privileges may be required if the packages + must be installed system wide. + + - | + The environment setup script will now attempt to install bifrost from PyPI + instead of using a stable branch. This is to address stability issues + with Ansible stable branches. + + If not requested to be installed into virtualenv, Ansible will be installed + into user's ``~/.local`` directory to not clobber possibly existing + system installation. + To use such installed Ansible, modifications of ``$PATH`` + environment variable might be required to include ``~/.local/bin`` path. + + Some backwards compatibility is provided via the use of the + ``ANSIBLE_GIT_BRANCH`` variable, where a user can define ``stable-X.Y`` + and the latest available version in that series will be installed. + To install the Ansible 2.1 series as part of the env-setup script, + execute ``env ANSIBLE_GIT_BRANCH="stable-2.1" scripts/env-setup``. + + Similarly, ``ANSIBLE_PIP_VERSION`` can be utilized to specify + the exact version, or range of version desired. Example: + + ``ANSIBLE_PIP_VERSION=2.1.0.1`` or ``ANSIBLE_PIP_VERSION=<2.2`` + +issues: + - | + If installing bifrost in a virtualenv (venv) and running playbooks + against localhost, you must install the basic python requirements + on a system-wide level due to the operating behavior of Ansible. + +fixes: + - | + Due to breaking change in the stable branch tags utilized with Ansible, + bifrost now utilizes installation of Ansible from PyPI. + +deprecations: + - | + The ``ANSIBLE_INSTALL_ROOT`` variable has been deprecated and is used + only to raise a warning for third party scripts. + - | + The ``ANSIBLE_FROM_PYPI`` variable no longer has any effect, as Ansible + is always installed from PyPI now. diff --git a/releasenotes/notes/ocata-summary-040e557460bab2bc.yaml b/releasenotes/notes/ocata-summary-040e557460bab2bc.yaml index 02507fd5f..15dae055d 100644 --- a/releasenotes/notes/ocata-summary-040e557460bab2bc.yaml +++ b/releasenotes/notes/ocata-summary-040e557460bab2bc.yaml @@ -5,7 +5,8 @@ prelude: > Coupled with a number of fixes, and improvements, users upgrading should take the time to read the entire release notes. A few highlights are below: - * Bifrost now installs and utilizes Ansible 2.1 by default. + * Bifrost now installs and utilizes Ansible 2.1 by default from + PyPI. * Ironic's default of modifying a pre-existing ironic.conf upon the installation being re-executed, has been changed to a utilize a template file. diff --git a/scripts/ansible-pip-str.py b/scripts/ansible-pip-str.py new file mode 100644 index 000000000..8ce6635cb --- /dev/null +++ b/scripts/ansible-pip-str.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python + +# Copyright (c) 2017 Mirantis Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Helper script to choose which ansible version to install for bifrost""" + +from __future__ import print_function +import sys + +in_str = sys.argv[1] +HELP_MSG = ("Unsupported version or format %s - " + "Supporting format [stable-]MAJ.MIN where MAJ.MIN is 1.9 or 2.x" + % in_str) + +if in_str.startswith('stable-'): + in_version = in_str.split('stable-')[1] +else: + if in_str[0].isdecimal(): + print("ansible==%s" % in_str) + else: + print("ansible%s" % in_str) + sys.exit(0) + +if len(in_version) != 3 and in_version[1] != '.': + print(HELP_MSG) + sys.exit(1) +else: + maj_version = in_version[0] + try: + min_version = int(in_version[2]) + except ValueError: + print(HELP_MSG) + sys.exit(1) + +if maj_version == '1' and min_version == 9: + upper_bound = '2.0' +elif maj_version == '2': + upper_bound = '2.%i' % (min_version + 1) +else: + print(HELP_MSG) + sys.exit(1) + +print("ansible<%s" % upper_bound) diff --git a/scripts/env-setup.sh b/scripts/env-setup.sh index adf641648..c2d1b3d34 100755 --- a/scripts/env-setup.sh +++ b/scripts/env-setup.sh @@ -1,12 +1,73 @@ #!/bin/bash set -eu -ANSIBLE_FROM_PYPI=${ANSIBLE_FROM_PYPI:-"False"} - source $(dirname $0)/install-deps.sh +# NOTE(pas-ha) the above exports some useful variables like +# $PYTHON , $PIP and $VENV depending on venv install or not -if [[ "$ANSIBLE_FROM_PYPI" == "True" ]]; then - source $(dirname $0)/install-ansible-pip.sh +ANSIBLE_PIP_VERSION=${ANSIBLE_PIP_VERSION:-${ANSIBLE_GIT_BRANCH:-stable-2.1}} + +ANSIBLE_PIP_STRING=$(${PYTHON} $(dirname $0)/ansible-pip-str.py ${ANSIBLE_PIP_VERSION}) + +if [ -n "${VENV-}" ]; then + ${PIP} install --upgrade "${ANSIBLE_PIP_STRING}" + ANSIBLE=${VENV}/bin/ansible else - source $(dirname $0)/install-ansible-source.sh + ${PIP} install --user --upgrade "${ANSIBLE_PIP_STRING}" + ANSIBLE=${HOME}/.local/bin/ansible fi + +PLAYBOOKS_LIBRARY_PATH=$(dirname $0)/../playbooks/library + +function check_get_module () { + local module=${1} + local module_url_base=${2} + ${ANSIBLE} localhost -m ${module} | grep "changed" || \ + wget "${module_url_base}/${module}.py" -O "${PLAYBOOKS_LIBRARY_PATH}/${module}.py" +} + +# Note(TheJulia): These files should be in the ansible library folder +# and this functionality exists for a level of ansible 1.9.x +# backwards compatability although the modules were developed +# for Ansible 2.0. +check_get_module os_ironic \ + https://raw.githubusercontent.com/ansible/ansible-modules-core/stable-2.0/cloud/openstack + +check_get_module os_ironic_node \ + https://raw.githubusercontent.com/ansible/ansible-modules-core/stable-2.0/cloud/openstack + +# os_ironic_inspect has appeared in Ansible 2.1 +check_get_module os_ironic_inspect \ + https://raw.githubusercontent.com/ansible/ansible-modules-extras/stable-2.1/cloud/openstack + +# os_keystone_service has appeared in Ansible 2.2 +check_get_module os_keystone_service \ + https://raw.githubusercontent.com/ansible/ansible-modules-extras/stable-2.2/cloud/openstack + +# NOTE(pas-ha) the following is a temporary workaround for third-party CI +# scripts that try to source Ansible's hacking/env-setup +# after running this very script +# TODO(pas-ha) remove after deprecation (in Pike?) and when third-party CIs +# (in particular OPNFV) are fixed +ANSIBLE_INSTALL_ROOT=${ANSIBLE_INSTALL_ROOT:-/opt/stack} +u=$(whoami) +g=$(groups | awk '{print $1}') +if [ ! -d ${ANSIBLE_INSTALL_ROOT} ]; then + mkdir -p ${ANSIBLE_INSTALL_ROOT} || (sudo mkdir -p ${ANSIBLE_INSTALL_ROOT}) +fi +sudo -H chown -R $u:$g ${ANSIBLE_INSTALL_ROOT} +mkdir -p ${ANSIBLE_INSTALL_ROOT}/ansible/hacking +echo "echo Sourcing this file is no longer needed! Ansible is always installed from PyPI" > ${ANSIBLE_INSTALL_ROOT}/ansible/hacking/env-setup + +echo +echo "To use bifrost, do" + +if [ -n "${VENV-}" ]; then + echo "source ${VENV}/bin/activate" +else + echo "Prepend ~/.local/bin to your PATH if it is not that way already.." + echo ".. or use full path to local Ansible at ~/.local/bin/ansible-playbook" +fi +echo "source env-vars" +echo "Then run playbooks as normal." +echo diff --git a/scripts/install-ansible-pip.sh b/scripts/install-ansible-pip.sh deleted file mode 100644 index 7248055fc..000000000 --- a/scripts/install-ansible-pip.sh +++ /dev/null @@ -1,15 +0,0 @@ -#!/bin/bash -set -eu - -ANSIBLE_PIP_VERSION=${ANSIBLE_PIP_VERSION:-"2.2"} -sudo -H -E ${PIP} install "ansible==$ANSIBLE_PIP_VERSION" - -echo -echo "To use bifrost, do" - -if [ -n "${VENV-}" ]; then - echo "source ${VENV}/bin/activate" -fi -echo "source env-vars" -echo "Then run playbooks as normal." -echo diff --git a/scripts/install-ansible-source.sh b/scripts/install-ansible-source.sh deleted file mode 100644 index 11b0857e8..000000000 --- a/scripts/install-ansible-source.sh +++ /dev/null @@ -1,74 +0,0 @@ -#!/bin/bash -set -eu - -ANSIBLE_GIT_URL=${ANSIBLE_GIT_URL:-https://github.com/ansible/ansible.git} -ANSIBLE_GIT_BRANCH=${ANSIBLE_GIT_BRANCH:-stable-2.1} -ANSIBLE_INSTALL_ROOT=${ANSIBLE_INSTALL_ROOT:-/opt/stack} - -function check_get_module () { - local file=${1} - local url=${2} - if [ ! -e ${file} ]; then - wget -O ${file} ${url} - fi -} - -u=$(whoami) -g=$(groups | awk '{print $1}') - -if [ ! -d ${ANSIBLE_INSTALL_ROOT} ]; then - mkdir -p ${ANSIBLE_INSTALL_ROOT} || (sudo mkdir -p ${ANSIBLE_INSTALL_ROOT}) -fi -sudo -H chown -R $u:$g ${ANSIBLE_INSTALL_ROOT} -cd ${ANSIBLE_INSTALL_ROOT} - -if [ ! -d ansible ]; then - git clone $ANSIBLE_GIT_URL --recursive -b $ANSIBLE_GIT_BRANCH - cd ansible -else - cd ansible - git remote update origin --prune - git fetch --tags - git checkout $ANSIBLE_GIT_BRANCH - git pull --rebase origin $ANSIBLE_GIT_BRANCH - git submodule update --init --recursive - git fetch -fi -# Note(TheJulia): These files should be in the ansible folder -# and this functionality exists for a level of ansible 1.9.x -# backwards compatability although the modules were developed -# for Ansible 2.0. - -check_get_module `pwd`/lib/ansible/modules/core/cloud/openstack/os_ironic.py \ - https://raw.githubusercontent.com/ansible/ansible-modules-core/stable-2.0/cloud/openstack/os_ironic.py -check_get_module `pwd`/lib/ansible/modules/core/cloud/openstack/os_ironic_node.py \ - https://raw.githubusercontent.com/ansible/ansible-modules-core/stable-2.0/cloud/openstack/os_ironic_node.py - -# os_ironic_inspect has appeared in Ansible 2.1 -check_get_module `pwd`/lib/ansible/modules/extras/cloud/openstack/os_ironic_inspect.py \ - https://raw.githubusercontent.com/ansible/ansible-modules-extras/stable-2.1/cloud/os_ironic_inspect.py - -# os_keystone_service has appeared in Ansible 2.2 -check_get_module `pwd`/lib/ansible/modules/extras/cloud/openstack/os_keystone_service.py \ - https://raw.githubusercontent.com/ansible/ansible-modules-extras/stable-2.2/cloud/openstack/os_keystone_service.py - -sudo -H -E ${PIP} install --upgrade ${ANSIBLE_INSTALL_ROOT}/ansible - -if [ -n "${VENV-}" ]; then - echo - echo "To use bifrost, do" - - echo "source ${VENV}/bin/activate" - echo "source env-vars" - echo "Then run playbooks as normal." - echo -else - echo - echo "If you're using this script directly, execute the" - echo "following commands to update your shell." - echo - echo "source env-vars" - echo "source ${ANSIBLE_INSTALL_ROOT}/ansible/hacking/env-setup" - echo -fi - diff --git a/scripts/test-bifrost.sh b/scripts/test-bifrost.sh index 79add4086..408d86fe9 100755 --- a/scripts/test-bifrost.sh +++ b/scripts/test-bifrost.sh @@ -55,24 +55,21 @@ elif [ $SOURCE = "test-bifrost-keystone-auth.sh" ]; then ENABLE_KEYSTONE="true" fi -# Source Ansible -# NOTE(TheJulia): Ansible stable-1.9 source method tosses an error deep -# under the hood which -x will detect, so for this step, we need to suspend -# and then re-enable the feature. -set +x +o nounset if [ ${USE_VENV} = "true" ]; then export VENV=/opt/stack/bifrost export PATH=${VENV}/bin:${PATH} $SCRIPT_HOME/env-setup.sh + # Note(cinerama): activate is not compatible with "set -u"; + # disable it just for this line. + set +u source /opt/stack/bifrost/bin/activate + set -u ANSIBLE=${VENV}/bin/ansible-playbook ENABLE_VENV="true" else $SCRIPT_HOME/env-setup.sh - source ${ANSIBLE_INSTALL_ROOT}/ansible/hacking/env-setup - ANSIBLE=$(which ansible-playbook) + ANSIBLE=${HOME}/.local/bin/ansible-playbook fi -set -x -o nounset # Adjust options for DHCP, VM, or Keystone tests if [ ${USE_DHCP} = "true" ]; then