c63ecadb08
When doing a multi-node devstack deployment starting in Ocata the child compute nodes must be discovered and mapped to the single nova cell (cell1). In the upstream CI we do this discovery in devstack-gate after the subnodes are stacked, but for anyone doing this manually we need to provide some notes on what needs to happen after child compute nodes are stacked for a multinode environment. Change-Id: I68418bcf28d86c60fe42537186d89458fa778bda Closes-Bug: #1688397
435 lines
14 KiB
ReStructuredText
435 lines
14 KiB
ReStructuredText
==============
|
|
Multi-Node Lab
|
|
==============
|
|
|
|
Here is OpenStack in a realistic test configuration with multiple
|
|
physical servers.
|
|
|
|
Prerequisites Linux & Network
|
|
=============================
|
|
|
|
Minimal Install
|
|
---------------
|
|
|
|
You need to have a system with a fresh install of Linux. You can
|
|
download the `Minimal
|
|
CD <https://help.ubuntu.com/community/Installation/MinimalCD>`__ for
|
|
Ubuntu releases since DevStack will download & install all the
|
|
additional dependencies. The netinstall ISO is available for
|
|
`Fedora <http://mirrors.kernel.org/fedora/releases/>`__
|
|
and
|
|
`CentOS/RHEL <http://mirrors.kernel.org/centos/>`__.
|
|
|
|
Install a couple of packages to bootstrap configuration:
|
|
|
|
::
|
|
|
|
apt-get install -y git sudo || yum install -y git sudo
|
|
|
|
Network Configuration
|
|
---------------------
|
|
|
|
The first iteration of the lab uses OpenStack's FlatDHCP network
|
|
controller so only a single network will be required. It should be on
|
|
its own subnet without DHCP; the host IPs and floating IP pool(s) will
|
|
come out of this block. This example uses the following:
|
|
|
|
- Gateway: 192.168.42.1
|
|
- Physical nodes: 192.168.42.11-192.168.42.99
|
|
- Floating IPs: 192.168.42.128-192.168.42.254
|
|
|
|
Configure each node with a static IP. For Ubuntu edit
|
|
``/etc/network/interfaces``:
|
|
|
|
::
|
|
|
|
auto eth0
|
|
iface eth0 inet static
|
|
address 192.168.42.11
|
|
netmask 255.255.255.0
|
|
gateway 192.168.42.1
|
|
|
|
For Fedora and CentOS/RHEL edit
|
|
``/etc/sysconfig/network-scripts/ifcfg-eth0``:
|
|
|
|
::
|
|
|
|
BOOTPROTO=static
|
|
IPADDR=192.168.42.11
|
|
NETMASK=255.255.255.0
|
|
GATEWAY=192.168.42.1
|
|
|
|
Installation shake and bake
|
|
===========================
|
|
|
|
Add the DevStack User
|
|
---------------------
|
|
|
|
OpenStack runs as a non-root user that has sudo access to root. There is
|
|
nothing special about the name, we'll use ``stack`` here. Every node
|
|
must use the same name and preferably uid. If you created a user during
|
|
the OS install you can use it and give it sudo privileges below.
|
|
Otherwise create the stack user:
|
|
|
|
::
|
|
|
|
useradd -s /bin/bash -d /opt/stack -m stack
|
|
|
|
This user will be making many changes to your system during installation
|
|
and operation so it needs to have sudo privileges to root without a
|
|
password:
|
|
|
|
::
|
|
|
|
echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
|
|
|
|
From here on use the ``stack`` user. **Logout** and **login** as the
|
|
``stack`` user.
|
|
|
|
Set Up Ssh
|
|
----------
|
|
|
|
Set up the stack user on each node with an ssh key for access:
|
|
|
|
::
|
|
|
|
mkdir ~/.ssh; chmod 700 ~/.ssh
|
|
echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyYjfgyPazTvGpd8OaAvtU2utL8W6gWC4JdRS1J95GhNNfQd657yO6s1AH5KYQWktcE6FO/xNUC2reEXSGC7ezy+sGO1kj9Limv5vrvNHvF1+wts0Cmyx61D2nQw35/Qz8BvpdJANL7VwP/cFI/p3yhvx2lsnjFE3hN8xRB2LtLUopUSVdBwACOVUmH2G+2BWMJDjVINd2DPqRIA4Zhy09KJ3O1Joabr0XpQL0yt/I9x8BVHdAx6l9U0tMg9dj5+tAjZvMAFfye3PJcYwwsfJoFxC8w/SLtqlFX7Ehw++8RtvomvuipLdmWCy+T9hIkl+gHYE4cS3OIqXH7f49jdJf jesse@spacey.local" > ~/.ssh/authorized_keys
|
|
|
|
Download DevStack
|
|
-----------------
|
|
|
|
Grab the latest version of DevStack:
|
|
|
|
::
|
|
|
|
git clone https://git.openstack.org/openstack-dev/devstack
|
|
cd devstack
|
|
|
|
Up to this point all of the steps apply to each node in the cluster.
|
|
From here on there are some differences between the cluster controller
|
|
(aka 'head node') and the compute nodes.
|
|
|
|
Configure Cluster Controller
|
|
----------------------------
|
|
|
|
The cluster controller runs all OpenStack services. Configure the
|
|
cluster controller's DevStack in ``local.conf``:
|
|
|
|
::
|
|
|
|
[[local|localrc]]
|
|
HOST_IP=192.168.42.11
|
|
FLAT_INTERFACE=eth0
|
|
FIXED_RANGE=10.4.128.0/20
|
|
FIXED_NETWORK_SIZE=4096
|
|
FLOATING_RANGE=192.168.42.128/25
|
|
MULTI_HOST=1
|
|
LOGFILE=/opt/stack/logs/stack.sh.log
|
|
ADMIN_PASSWORD=labstack
|
|
DATABASE_PASSWORD=supersecret
|
|
RABBIT_PASSWORD=supersecret
|
|
SERVICE_PASSWORD=supersecret
|
|
|
|
In the multi-node configuration the first 10 or so IPs in the private
|
|
subnet are usually reserved. Add this to ``local.sh`` to have it run
|
|
after every ``stack.sh`` run:
|
|
|
|
::
|
|
|
|
for i in `seq 2 10`; do /opt/stack/nova/bin/nova-manage fixed reserve 10.4.128.$i; done
|
|
|
|
Fire up OpenStack:
|
|
|
|
::
|
|
|
|
./stack.sh
|
|
|
|
A stream of activity ensues. When complete you will see a summary of
|
|
``stack.sh``'s work, including the relevant URLs, accounts and passwords
|
|
to poke at your shiny new OpenStack. The most recent log file is
|
|
available in ``stack.sh.log``.
|
|
|
|
Configure Compute Nodes
|
|
-----------------------
|
|
|
|
The compute nodes only run the OpenStack worker services. For additional
|
|
machines, create a ``local.conf`` with:
|
|
|
|
::
|
|
|
|
[[local|localrc]]
|
|
HOST_IP=192.168.42.12 # change this per compute node
|
|
FLAT_INTERFACE=eth0
|
|
FIXED_RANGE=10.4.128.0/20
|
|
FIXED_NETWORK_SIZE=4096
|
|
FLOATING_RANGE=192.168.42.128/25
|
|
MULTI_HOST=1
|
|
LOGFILE=/opt/stack/logs/stack.sh.log
|
|
ADMIN_PASSWORD=labstack
|
|
DATABASE_PASSWORD=supersecret
|
|
RABBIT_PASSWORD=supersecret
|
|
SERVICE_PASSWORD=supersecret
|
|
DATABASE_TYPE=mysql
|
|
SERVICE_HOST=192.168.42.11
|
|
MYSQL_HOST=$SERVICE_HOST
|
|
RABBIT_HOST=$SERVICE_HOST
|
|
GLANCE_HOSTPORT=$SERVICE_HOST:9292
|
|
ENABLED_SERVICES=n-cpu,q-agt,n-api-meta,c-vol,placement-client
|
|
NOVA_VNC_ENABLED=True
|
|
NOVNCPROXY_URL="http://$SERVICE_HOST:6080/vnc_auto.html"
|
|
VNCSERVER_LISTEN=$HOST_IP
|
|
VNCSERVER_PROXYCLIENT_ADDRESS=$VNCSERVER_LISTEN
|
|
|
|
**Note:** the ``n-api-meta`` service is a version of the api server
|
|
that only serves the metadata service. It's needed because the
|
|
computes created won't have a routing path to the metadata service on
|
|
the controller.
|
|
|
|
Fire up OpenStack:
|
|
|
|
::
|
|
|
|
./stack.sh
|
|
|
|
A stream of activity ensues. When complete you will see a summary of
|
|
``stack.sh``'s work, including the relevant URLs, accounts and passwords
|
|
to poke at your shiny new OpenStack. The most recent log file is
|
|
available in ``stack.sh.log``.
|
|
|
|
Starting in the Ocata release, Nova requires a `Cells v2`_ deployment. Compute
|
|
node services must be mapped to a cell before they can be used.
|
|
|
|
After each compute node is stacked, verify it shows up in the
|
|
``nova service-list --binary nova-compute`` output. The compute service is
|
|
registered in the cell database asynchronously so this may require polling.
|
|
|
|
Once the compute node services shows up, run the ``./tools/discover_hosts.sh``
|
|
script from the control node to map compute hosts to the single cell.
|
|
|
|
The compute service running on the primary control node will be
|
|
discovered automatically when the control node is stacked so this really
|
|
only needs to be performed for subnodes.
|
|
|
|
.. _Cells v2: https://docs.openstack.org/nova/latest/user/cells.html
|
|
|
|
Cleaning Up After DevStack
|
|
--------------------------
|
|
|
|
Shutting down OpenStack is now as simple as running the included
|
|
``unstack.sh`` script:
|
|
|
|
::
|
|
|
|
./unstack.sh
|
|
|
|
A more aggressive cleanup can be performed using ``clean.sh``. It
|
|
removes certain troublesome packages and attempts to leave the system in
|
|
a state where changing the database or queue manager can be reliably
|
|
performed.
|
|
|
|
::
|
|
|
|
./clean.sh
|
|
|
|
Sometimes running instances are not cleaned up. DevStack attempts to do
|
|
this when it runs but there are times it needs to still be done by hand:
|
|
|
|
::
|
|
|
|
sudo rm -rf /etc/libvirt/qemu/inst*
|
|
sudo virsh list | grep inst | awk '{print $1}' | xargs -n1 virsh destroy
|
|
|
|
Options pimp your stack
|
|
=======================
|
|
|
|
Additional Users
|
|
----------------
|
|
|
|
DevStack creates two OpenStack users (``admin`` and ``demo``) and two
|
|
projects (also ``admin`` and ``demo``). ``admin`` is exactly what it
|
|
sounds like, a privileged administrative account that is a member of
|
|
both the ``admin`` and ``demo`` projects. ``demo`` is a normal user
|
|
account that is only a member of the ``demo`` project. Creating
|
|
additional OpenStack users can be done through the dashboard, sometimes
|
|
it is easier to do them in bulk from a script, especially since they get
|
|
blown away every time ``stack.sh`` runs. The following steps are ripe
|
|
for scripting:
|
|
|
|
::
|
|
|
|
# Get admin creds
|
|
. openrc admin admin
|
|
|
|
# List existing projects
|
|
openstack project list
|
|
|
|
# List existing users
|
|
openstack user list
|
|
|
|
# Add a user and project
|
|
NAME=bob
|
|
PASSWORD=BigSecret
|
|
PROJECT=$NAME
|
|
openstack project create $PROJECT
|
|
openstack user create $NAME --password=$PASSWORD --project $PROJECT
|
|
openstack role add Member --user $NAME --project $PROJECT
|
|
# The Member role is created by stack.sh
|
|
# openstack role assignment list
|
|
|
|
Swift
|
|
-----
|
|
|
|
Swift, OpenStack Object Storage, requires a significant amount of resources
|
|
and is disabled by default in DevStack. The support in DevStack is geared
|
|
toward a minimal installation but can be used for testing. To implement a
|
|
true multi-node test of swift, additional steps will be required. Enabling it is as
|
|
simple as enabling the ``swift`` service in ``local.conf``:
|
|
|
|
::
|
|
|
|
enable_service s-proxy s-object s-container s-account
|
|
|
|
Swift, OpenStack Object Storage, will put its data files in ``SWIFT_DATA_DIR`` (default
|
|
``/opt/stack/data/swift``). The size of the data 'partition' created
|
|
(really a loop-mounted file) is set by ``SWIFT_LOOPBACK_DISK_SIZE``. The
|
|
Swift config files are located in ``SWIFT_CONF_DIR`` (default
|
|
``/etc/swift``). All of these settings can be overridden in (wait for
|
|
it...) ``local.conf``.
|
|
|
|
Volumes
|
|
-------
|
|
|
|
DevStack will automatically use an existing LVM volume group named
|
|
``stack-volumes`` to store cloud-created volumes. If ``stack-volumes``
|
|
doesn't exist, DevStack will set up a 10Gb loop-mounted file to contain
|
|
it. This obviously limits the number and size of volumes that can be
|
|
created inside OpenStack. The size can be overridden by setting
|
|
``VOLUME_BACKING_FILE_SIZE`` in ``local.conf``.
|
|
|
|
``stack-volumes`` can be pre-created on any physical volume supported by
|
|
Linux's LVM. The name of the volume group can be changed by setting
|
|
``VOLUME_GROUP_NAME`` in ``localrc``. ``stack.sh`` deletes all logical
|
|
volumes in ``VOLUME_GROUP_NAME`` that begin with ``VOLUME_NAME_PREFIX`` as
|
|
part of cleaning up from previous runs. It is recommended to not use the
|
|
root volume group as ``VOLUME_GROUP_NAME``.
|
|
|
|
The details of creating the volume group depends on the server hardware
|
|
involved but looks something like this:
|
|
|
|
::
|
|
|
|
pvcreate /dev/sdc
|
|
vgcreate stack-volumes /dev/sdc
|
|
|
|
Syslog
|
|
------
|
|
|
|
DevStack is capable of using ``rsyslog`` to aggregate logging across the
|
|
cluster. It is off by default; to turn it on set ``SYSLOG=True`` in
|
|
``local.conf``. ``SYSLOG_HOST`` defaults to ``HOST_IP``; on the compute
|
|
nodes it must be set to the IP of the cluster controller to send syslog
|
|
output there. In the example above, add this to the compute node
|
|
``local.conf``:
|
|
|
|
::
|
|
|
|
SYSLOG_HOST=192.168.42.11
|
|
|
|
Using Alternate Repositories/Branches
|
|
-------------------------------------
|
|
|
|
The git repositories for all of the OpenStack services are defined in
|
|
``stackrc``. Since this file is a part of the DevStack package changes
|
|
to it will probably be overwritten as updates are applied. Every setting
|
|
in ``stackrc`` can be redefined in ``local.conf``.
|
|
|
|
To change the repository or branch that a particular OpenStack service
|
|
is created from, simply change the value of ``*_REPO`` or ``*_BRANCH``
|
|
corresponding to that service.
|
|
|
|
After making changes to the repository or branch, if ``RECLONE`` is not
|
|
set in ``localrc`` it may be necessary to remove the corresponding
|
|
directory from ``/opt/stack`` to force git to re-clone the repository.
|
|
|
|
For example, to pull nova, OpenStack Compute, from a proposed release candidate
|
|
in the primary nova repository:
|
|
|
|
::
|
|
|
|
NOVA_BRANCH=rc-proposed
|
|
|
|
To pull glance, OpenStack Image service, from an experimental fork:
|
|
|
|
::
|
|
|
|
GLANCE_BRANCH=try-something-big
|
|
GLANCE_REPO=https://github.com/mcuser/glance.git
|
|
|
|
Notes stuff you might need to know
|
|
==================================
|
|
|
|
Reset the Bridge
|
|
----------------
|
|
|
|
How to reset the bridge configuration:
|
|
|
|
::
|
|
|
|
sudo brctl delif br100 eth0.926
|
|
sudo ip link set dev br100 down
|
|
sudo brctl delbr br100
|
|
|
|
Set MySQL Password
|
|
------------------
|
|
|
|
If you forgot to set the root password you can do this:
|
|
|
|
::
|
|
|
|
mysqladmin -u root -pnova password 'supersecret'
|
|
|
|
Live Migration
|
|
--------------
|
|
|
|
In order for live migration to work with the default live migration URI::
|
|
|
|
[libvirt]
|
|
live_migration_uri = qemu+ssh://stack@%s/system
|
|
|
|
SSH keys need to be exchanged between each compute node:
|
|
|
|
1. The SOURCE root user's public RSA key (likely in /root/.ssh/id_rsa.pub)
|
|
needs to be in the DESTINATION stack user's authorized_keys file
|
|
(~stack/.ssh/authorized_keys). This can be accomplished by manually
|
|
copying the contents from the file on the SOURCE to the DESTINATION. If
|
|
you have a password configured for the stack user, then you can use the
|
|
following command to accomplish the same thing::
|
|
|
|
ssh-copy-id -i /root/.ssh/id_rsa.pub stack@DESTINATION
|
|
|
|
2. The DESTINATION host's public ECDSA key (/etc/ssh/ssh_host_ecdsa_key.pub)
|
|
needs to be in the SOURCE root user's known_hosts file
|
|
(/root/.ssh/known_hosts). This can be accomplished by running the
|
|
following on the SOURCE machine (hostname must be used)::
|
|
|
|
ssh-keyscan -H DEST_HOSTNAME | sudo tee -a /root/.ssh/known_hosts
|
|
|
|
3. Verify that login via ssh works without a password::
|
|
|
|
ssh -i /root/.ssh/id_rsa.pub stack@DESTINATION
|
|
|
|
In essence, this means that every compute node's root user's public RSA key
|
|
must exist in every other compute node's stack user's authorized_keys file and
|
|
every compute node's public ECDSA key needs to be in every other compute
|
|
node's root user's known_hosts file. Please note that if the root or stack
|
|
user does not have a SSH key, one can be generated using::
|
|
|
|
ssh-keygen -t rsa
|
|
|
|
The above steps are necessary because libvirtd runs as root when the
|
|
live_migration_uri uses the "qemu:///system" family of URIs. For more
|
|
information, see the `libvirt documentation`_.
|
|
|
|
.. _libvirt documentation: https://libvirt.org/drvqemu.html#securitydriver
|