diff --git a/bin/swift-init b/bin/swift-init index d1fa004f29..cd31ea8a93 100755 --- a/bin/swift-init +++ b/bin/swift-init @@ -180,7 +180,7 @@ if command == 'restart': for server in servers: do_start(server) -if command == 'reload': +if command == 'reload' or command == 'force-reload': for server in servers: do_stop(server, graceful=True) do_start(server) diff --git a/doc/source/admin_guide.rst b/doc/source/admin_guide.rst index aba899f5fa..a39d164534 100644 --- a/doc/source/admin_guide.rst +++ b/doc/source/admin_guide.rst @@ -108,8 +108,100 @@ different distro or OS, some care should be taken before using in production. Cluster Health -------------- -TODO: Greg, add docs here about how to use swift-stats-populate, and -swift-stats-report +There is a swift-stats-report tool for measuring overall cluster health. This +is accomplished by checking if a set of deliberately distributed containers and +objects are currently in their proper places within the cluster. + +For instance, a common deployment has three replicas of each object. The health +of that object can be measured by checking if each replica is in its proper +place. If only 2 of the 3 is in place the object's heath can be said to be at +66.66%, where 100% would be perfect. + +A single object's health, especially an older object, usually reflects the +health of that entire partition the object is in. If we make enough objects on +a distinct percentage of the partitions in the cluster, we can get a pretty +valid estimate of the overall cluster health. In practice, about 1% partition +coverage seems to balance well between accuracy and the amount of time it takes +to gather results. + +The first thing that needs to be done to provide this health value is create a +new account solely for this usage. Next, we need to place the containers and +objects throughout the system so that they are on distinct partitions. The +swift-stats-populate tool does this by making up random container and object +names until they fall on distinct partitions. Last, and repeatedly for the life +of the cluster, we need to run the swift-stats-report tool to check the health +of each of these containers and objects. + +These tools need direct access to the entire cluster and to the ring files +(installing them on an auth server or a proxy server will probably do). Both +swift-stats-populate and swift-stats-report use the same configuration file, +/etc/swift/stats.conf. Example conf file:: + + [stats] + auth_url = http://saio:11000/v1.0 + auth_user = test:tester + auth_key = testing + +There are also options for the conf file for specifying the dispersion coverage +(defaults to 1%), retries, concurrency, CSV output file, etc. though usually +the defaults are fine. + +Once the configuration is in place, run `swift-stats-populate -d` to populate +the containers and objects throughout the cluster. + +Now that those containers and objects are in place, you can run +`swift-stats-report -d` to get a dispersion report, or the overall health of +the cluster. Here is an example of a cluster in perfect health:: + + $ swift-stats-report -d + Queried 2621 containers for dispersion reporting, 19s, 0 retries + 100.00% of container copies found (7863 of 7863) + Sample represents 1.00% of the container partition space + + Queried 2619 objects for dispersion reporting, 7s, 0 retries + 100.00% of object copies found (7857 of 7857) + Sample represents 1.00% of the object partition space + +Now I'll deliberately double the weight of a device in the object ring (with +replication turned off) and rerun the dispersion report to show what impact +that has:: + + $ swift-ring-builder object.builder set_weight d0 200 + $ swift-ring-builder object.builder rebalance + ... + $ swift-stats-report -d + Queried 2621 containers for dispersion reporting, 8s, 0 retries + 100.00% of container copies found (7863 of 7863) + Sample represents 1.00% of the container partition space + + Queried 2619 objects for dispersion reporting, 7s, 0 retries + There were 1763 partitions missing one copy. + 77.56% of object copies found (6094 of 7857) + Sample represents 1.00% of the object partition space + +You can see the health of the objects in the cluster has gone down +significantly. Of course, I only have four devices in this test environment, in +a production environment with many many devices the impact of one device change +is much less. Next, I'll run the replicators to get everything put back into +place and then rerun the dispersion report:: + + ... start object replicators and monitor logs until they're caught up ... + $ swift-stats-report -d + Queried 2621 containers for dispersion reporting, 17s, 0 retries + 100.00% of container copies found (7863 of 7863) + Sample represents 1.00% of the container partition space + + Queried 2619 objects for dispersion reporting, 7s, 0 retries + 100.00% of object copies found (7857 of 7857) + Sample represents 1.00% of the object partition space + +So that's a summation of how to use swift-stats-report to monitor the health of +a cluster. There are a few other things it can do, such as performance +monitoring, but those are currently in their infancy and little used. For +instance, you can run `swift-stats-populate -p` and `swift-stats-report -p` to +get performance timings (warning: the initial populate takes a while). These +timings are dumped into a CSV file (/etc/swift/stats.csv by default) and can +then be graphed to see how cluster performance is trending. ------------------------ Debugging Tips and Tools diff --git a/doc/source/debian_package_guide.rst b/doc/source/debian_package_guide.rst new file mode 100644 index 0000000000..a474a432b0 --- /dev/null +++ b/doc/source/debian_package_guide.rst @@ -0,0 +1,143 @@ +======================================================= +Building and Deploying Custom Debian packages for Swift +======================================================= + +This documents setting up the prerequisites, downloading the code and building +Debian packages necessary to deploy the various components of the swift +project code. These steps were done on a server running +*Ubuntu 10.04 LTS (Lucid Lynx)*, but should give a good idea what to do on +other distros. + +------------------------------------------ +Instructions for Deploying Swift-Core PPAs +------------------------------------------ + +These packages are built from the current development branch "trunk" + +* Add swift-core/ppa repository. As root::: + + apt-get install python-software-properties + add-apt-repository ppa:swift-core/trunk + apt-get update + +* Install the swift base packages:: + + apt-get install python-swift + apt-get install swift + +* Install the swift packages depending on your implementations:: + + apt-get install swift-auth + apt-get install swift-proxy + apt-get install swift-account + apt-get install swift-container + apt-get install swift-object + +* Copy sample configuration files to `/etc/swift` directory + and rename them to `*.conf files`:: + + cp -a /usr/share/doc/swift/*.conf-sample /etc/swift/ + cd /etc/swift ; rename 's/\-sample$//' *.conf-sample + +* For servers running the swift-account, swift-container or + swift-object the rsync.conf file should be moved to + the `/etc` directory:: + + cd /etc/swift + mv rsyncd.conf /etc + +* Modify configuration files to meet your implementation requirements + the defaults have been not been geared to a multi-server implementation. + +--------------------------------------------------- +Instructions for Building Debian Packages for Swift +--------------------------------------------------- + +* Add swift-core/ppa repository and install prerequisites. As root:: + + apt-get install python-software-properties + add-apt-repository ppa:swift-core/ppa + apt-get update + apt-get install curl gcc bzr python-configobj python-coverage python-dev + python-nose python-setuptools python-simplejson python-xattr + python-webob python-eventlet python-greenlet debhelper python-sphinx + python-all python-openssl + +* As you + + #. Tell bzr who you are:: + + bzr whoami ' ' + bzr lp-login + + #. Create a local bazaar repository for dev/testing:: + + bzr init-repo swift + + #. Pull down the swift/debian files:: + + cd swift + bzr branch lp:~swift-core/swift/debian + + #. If you want to merge in a branch:: + + cd debian + bzr merge lp: + + #. Create the debian packages:: + + cd debian + bzr bd --builder='debuild -uc -us' + + #. Upload packages to your target servers:: + + cd .. + scp *.deb root@:~/. + + +---------------------------------------------------- +Instructions for Deploying Debian Packages for Swift +---------------------------------------------------- + +* On a Target Server, As root: + + #. Setup the swift ppa:: + + add-apt-repository ppa:swift-core/ppa + apt-get update + + #. Install dependencies:: + + apt-get install rsync python-openssl python-setuptools python-webob + python-simplejson python-xattr python-greenlet python-eventlet + + #. Install base packages:: + + dpkg -i python-swift__all.deb + dpkg -i swift__all.deb + + #. Install packages depending on your implementation:: + + dpkg -i swift-auth__all.deb + dpkg -i swift-proxy__all.deb + dpkg -i swift-account__all.deb + dpkg -i swift-container__all.deb + dpkg -i swift-object__all.deb + dpkg -i swift-doc__all.deb + + #. Copy sample configuration files to `/etc/swift` directory + and rename them to `*.conf files`:: + + cp -a /usr/share/doc/swift/*.conf-sample /etc/swift/ + cd /etc/swift + rename 's/\-sample$//' *.conf-sample + + #. For servers running the swift-account, swift-container or + swift-object the rsync.conf file should be moved to + the `/etc` directory:: + + cd /etc/swift/ + mv rsyncd.conf /etc + + #. Modify configuration files to meet your implementation requirements + the defaults have been not been geared to a multi-server implementation. diff --git a/doc/source/development_saio.rst b/doc/source/development_saio.rst index 88fd423c8f..e62f2492e9 100644 --- a/doc/source/development_saio.rst +++ b/doc/source/development_saio.rst @@ -11,8 +11,12 @@ virtual machine will emulate running a four node Swift cluster. It assumes you're using *VMware Fusion 3* on *Mac OS X Snow Leopard*, but should give a good idea what to do on other environments. -* Get the *Ubuntu 10.04 LTS (Lucid Lynx)* server image from: - http://cdimage.ubuntu.com/releases/10.04/release/ubuntu-10.04-dvd-amd64.iso +* Get the *Ubuntu 10.04 LTS (Lucid Lynx)* server image: + + - Ubuntu Server ISO: http://releases.ubuntu.com/10.04/ubuntu-10.04.1-server-amd64.iso (682 MB) + - Ubuntu Live/Install: http://cdimage.ubuntu.com/releases/10.04/release/ubuntu-10.04-dvd-amd64.iso (4.1 GB) + - Ubuntu Mirrors: https://launchpad.net/ubuntu/+cdmirrors + * Create guest virtual machine: #. `Continue without disc` diff --git a/doc/source/index.rst b/doc/source/index.rst index 3461a9083c..7e4681ca28 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -40,6 +40,7 @@ Deployment: deployment_guide admin_guide + debian_package_guide End User Guides: