Clay Gerrard de3acec4bf Set default wsgi workers to cpu_count

Change the default value of wsgi workers from 1 to auto.  The new default
value for workers in the proxy, container, account & object wsgi servers will
spawn as many workers per process as you have cpu cores.

This will not be ideal for some configurations, but it's much more likely to
produce a successful out of the box deployment.

Inspect the number of cpu_cores using python's multiprocessing when available.
Multiprocessing was added in python 2.6, but I know I've compiled python
without it before on accident.  The cpu_count method seems to be pretty system
agnostic, but it says it can raise NotImplementedError or sometimes return 0.

Add a new utility method 'config_auto_int_value' to pull an integer out of the
config which has a dynamic default.

 * drive by s/container/proxy/ in proxy-server.conf.5
 * fix misplaced max_clients in *-server.conf-sample
 * update doc/development_saio to force workers = 1

DocImpact

Change-Id: Ifa563d22952c902ab8cbe1d339ba385413c54e95

2013-07-18 22:57:18 -07:00

24 KiB

Raw Blame History

SAIO - Swift All In One

Instructions for setting up a development VM

This section documents setting up a virtual machine for doing Swift development. The virtual machine will emulate running a four node Swift cluster.

Get an Ubuntu 12.04 LTS (Precise Pangolin) server image or try something Fedora/CentOS.
Create guest virtual machine from the image.

Additional information about setting up a Swift development snapshot on other distributions is available on the wiki at http://wiki.openstack.org/SAIOInstructions.

What's in a <your-user-name>

Much of the configuration described in this guide requires escalated root privileges; however, we assume that administrator logs in an unprivileged user. Swift processes also run under a separate user and group, set by configuration option, and refered as <your-user-name>:<your-group-name>. The default user is swift, which may not exist on your system.

Perform the following commands as root.

Installing dependencies

On apt based systems,
1. apt-get update
2. apt-get install curl gcc memcached rsync sqlite3 xfsprogs git-core python-setuptools
3. apt-get install python-coverage python-dev python-nose python-simplejson python-xattr python-eventlet python-greenlet python-pastedeploy python-netifaces python-pip python-dnspython python-mock
On yum based systems,
1. yum install curl gcc memcached rsync sqlite xfsprogs git-core xinetd python-setuptools
2. yum install python-coverage python-devel python-nose python-simplejson python-xattr python-eventlet python-greenlet python-pastedeploy python-netifaces python-pip python-dnspython python-mock
This installs necessary system dependencies; and most of the python dependencies. Later in the process setuptools/distribute or pip will install and/or upgrade some other stuff - it's getting harder to avoid. You can also install anything else you want, like screen, ssh, vim, etc.

Next, choose either partition-section or loopback-section.

Using a partition for storage

If you are going to use a separate partition for Swift data, be sure to add another device when creating the VM, and follow these instructions.

fdisk /dev/sdb (set up a single partition)

mkfs.xfs /dev/sdb1

Edit /etc/fstab and add

/dev/sdb1 /mnt/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0

mkdir /mnt/sdb1

mount /mnt/sdb1

mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4

chown <your-user-name>:<your-group-name> /mnt/sdb1/*

mkdir /srv

for x in {1..4}; do ln -s /mnt/sdb1/$x /srv/$x; done

mkdir -p /etc/swift/object-server /etc/swift/container-server /etc/swift/account-server /srv/1/node/sdb1 /srv/2/node/sdb2 /srv/3/node/sdb3 /srv/4/node/sdb4 /var/run/swift

chown -R <your-user-name>:<your-group-name> /etc/swift /srv/[1-4]/ /var/run/swift -- Make sure to include the trailing slash after /srv/[1-4]/
Add to /etc/rc.local (before the exit 0):
mkdir -p /var/cache/swift /var/cache/swift2 /var/cache/swift3 /var/cache/swift4
chown <your-user-name>:<your-group-name> /var/cache/swift*
mkdir -p /var/run/swift
chown <your-user-name>:<your-group-name> /var/run/swift
Next, skip to rsync-section.

Using a loopback device for storage

If you want to use a loopback device instead of another partition, follow these instructions.

mkdir /srv

truncate -s 1GB /srv/swift-disk

(modify size to make a larger or smaller partition)

mkfs.xfs /srv/swift-disk

Edit /etc/fstab and add

/srv/swift-disk /mnt/sdb1 xfs loop,noatime,nodiratime,nobarrier,logbufs=8 0 0

mkdir /mnt/sdb1

mount /mnt/sdb1

mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4

chown <your-user-name>:<your-group-name> /mnt/sdb1/*

for x in {1..4}; do ln -s /mnt/sdb1/$x /srv/$x; done

mkdir -p /etc/swift/object-server /etc/swift/container-server /etc/swift/account-server /srv/1/node/sdb1 /srv/2/node/sdb2 /srv/3/node/sdb3 /srv/4/node/sdb4 /var/run/swift

chown -R <your-user-name>:<your-group-name> /etc/swift /srv/[1-4]/ /var/run/swift -- Make sure to include the trailing slash after /srv/[1-4]/
Add to /etc/rc.local (before the exit 0):
mkdir -p /var/cache/swift /var/cache/swift2 /var/cache/swift3 /var/cache/swift4
chown <your-user-name>:<your-group-name> /var/cache/swift*
mkdir -p /var/run/swift
chown <your-user-name>:<your-group-name> /var/run/swift

Setting up rsync

Create /etc/rsyncd.conf:

uid = <your-user-name>
gid = <your-group-name>
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 127.0.0.1

[account6012]
max connections = 25
path = /srv/1/node/
read only = false
lock file = /var/lock/account6012.lock

[account6022]
max connections = 25
path = /srv/2/node/
read only = false
lock file = /var/lock/account6022.lock

[account6032]
max connections = 25
path = /srv/3/node/
read only = false
lock file = /var/lock/account6032.lock

[account6042]
max connections = 25
path = /srv/4/node/
read only = false
lock file = /var/lock/account6042.lock

[container6011]
max connections = 25
path = /srv/1/node/
read only = false
lock file = /var/lock/container6011.lock

[container6021]
max connections = 25
path = /srv/2/node/
read only = false
lock file = /var/lock/container6021.lock

[container6031]
max connections = 25
path = /srv/3/node/
read only = false
lock file = /var/lock/container6031.lock

[container6041]
max connections = 25
path = /srv/4/node/
read only = false
lock file = /var/lock/container6041.lock

[object6010]
max connections = 25
path = /srv/1/node/
read only = false
lock file = /var/lock/object6010.lock

[object6020]
max connections = 25
path = /srv/2/node/
read only = false
lock file = /var/lock/object6020.lock

[object6030]
max connections = 25
path = /srv/3/node/
read only = false
lock file = /var/lock/object6030.lock

[object6040]
max connections = 25
path = /srv/4/node/
read only = false
lock file = /var/lock/object6040.lock

On Ubuntu, edit the following line in /etc/default/rsync:
```
RSYNC_ENABLE=true
```
On Fedora, edit the following line in /etc/xinetd.d/rsync:
```
disable = no
```
On Ubuntu, run service rsync restart, on xinetd based systems run service xinetd restart.
Verify rsync is accepting connections for all servers:
```
rsync rsync://pub@localhost/
```

Starting memcached

On non-Ubuntu distros you need to ensure memcached is running:

service memcached start

chkconfig memcached on

or:

systemctl enable memcached.service

systemctl start memcached.service

The tempauth middleware stores tokens in memcached. If memcached is not running, tokens cannot be validated, and accessing Swift becomes impossible.

Optional: Setting up rsyslog for individual logging

Create /etc/rsyslog.d/10-swift.conf:

# Uncomment the following to have a log containing all logs together
#local1,local2,local3,local4,local5.*   /var/log/swift/all.log

# Uncomment the following to have hourly proxy logs for stats processing
#$template HourlyProxyLog,"/var/log/swift/hourly/%$YEAR%%$MONTH%%$DAY%%$HOUR%"
#local1.*;local1.!notice ?HourlyProxyLog

local1.*;local1.!notice /var/log/swift/proxy.log
local1.notice           /var/log/swift/proxy.error
local1.*                ~

local2.*;local2.!notice /var/log/swift/storage1.log
local2.notice           /var/log/swift/storage1.error
local2.*                ~

local3.*;local3.!notice /var/log/swift/storage2.log
local3.notice           /var/log/swift/storage2.error
local3.*                ~

local4.*;local4.!notice /var/log/swift/storage3.log
local4.notice           /var/log/swift/storage3.error
local4.*                ~

local5.*;local5.!notice /var/log/swift/storage4.log
local5.notice           /var/log/swift/storage4.error
local5.*                ~

Edit /etc/rsyslog.conf and make the following change:
```
$PrivDropToGroup adm
```
mkdir -p /var/log/swift/hourly
chown -R syslog.adm /var/log/swift
chmod -R g+w /var/log/swift
service rsyslog restart

Getting the code

You can do the following commands as administrator user.

Check out the python-swiftclient repo

git clone https://github.com/openstack/python-swiftclient.git

Build a development installation of python-swiftclient

cd ~/python-swiftclient; sudo python setup.py develop; cd -

Check out the swift repo

git clone https://github.com/openstack/swift.git

Build a development installation of swift

cd ~/swift; sudo python setup.py develop; cd -

Install swift's test dependencies

sudo pip install -r swift/test-requirements.txt

Do the following commands as root, but verify that Swift has access to resulting configuration files.

Configuring each node

Sample configuration files are provided with all defaults in line-by-line comments.

Create `/etc/swift/proxy-server.conf`:

[DEFAULT]
bind_port = 8080
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL1
eventlet_debug = true

[pipeline:main]
pipeline = healthcheck cache tempauth proxy-logging proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:tempauth]
use = egg:swift#tempauth
user_admin_admin = admin .admin .reseller_admin
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:cache]
use = egg:swift#memcache

[filter:proxy-logging]
use = egg:swift#proxy_logging

Create `/etc/swift/swift.conf`:

[swift-hash]
# random unique strings that can never change (DO NOT LOSE)
swift_hash_path_prefix = changeme
swift_hash_path_suffix = changeme

Create `/etc/swift/account-server/1.conf`:

[DEFAULT]
devices = /srv/1/node
mount_check = false
disable_fallocate = true
bind_port = 6012
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL2
recon_cache_path = /var/cache/swift
eventlet_debug = true

[pipeline:main]
pipeline = recon account-server

[app:account-server]
use = egg:swift#account

[filter:recon]
use = egg:swift#recon

[account-replicator]
vm_test_mode = yes

[account-auditor]

[account-reaper]

Create `/etc/swift/account-server/2.conf`:

[DEFAULT]
devices = /srv/2/node
mount_check = false
disable_fallocate = true
bind_port = 6022
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL3
recon_cache_path = /var/cache/swift2
eventlet_debug = true

[pipeline:main]
pipeline = recon account-server

[app:account-server]
use = egg:swift#account

[filter:recon]
use = egg:swift#recon

[account-replicator]
vm_test_mode = yes

[account-auditor]

[account-reaper]

Create `/etc/swift/account-server/3.conf`:

[DEFAULT]
devices = /srv/3/node
mount_check = false
disable_fallocate = true
bind_port = 6032
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL4
recon_cache_path = /var/cache/swift3
eventlet_debug = true

[pipeline:main]
pipeline = recon account-server

[app:account-server]
use = egg:swift#account

[filter:recon]
use = egg:swift#recon

[account-replicator]
vm_test_mode = yes

[account-auditor]

[account-reaper]

Create `/etc/swift/account-server/4.conf`:

[DEFAULT]
devices = /srv/4/node
mount_check = false
disable_fallocate = true
bind_port = 6042
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL5
recon_cache_path = /var/cache/swift4
eventlet_debug = true

[pipeline:main]
pipeline = recon account-server

[app:account-server]
use = egg:swift#account

[filter:recon]
use = egg:swift#recon

[account-replicator]
vm_test_mode = yes

[account-auditor]

[account-reaper]

Create `/etc/swift/container-server/1.conf`:

[DEFAULT]
devices = /srv/1/node
mount_check = false
disable_fallocate = true
bind_port = 6011
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL2
recon_cache_path = /var/cache/swift
eventlet_debug = true

[pipeline:main]
pipeline = recon container-server

[app:container-server]
use = egg:swift#container

[filter:recon]
use = egg:swift#recon

[container-replicator]
vm_test_mode = yes

[container-updater]

[container-auditor]

[container-sync]

Create `/etc/swift/container-server/2.conf`:

[DEFAULT]
devices = /srv/2/node
mount_check = false
disable_fallocate = true
bind_port = 6021
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL3
recon_cache_path = /var/cache/swift2
eventlet_debug = true

[pipeline:main]
pipeline = recon container-server

[app:container-server]
use = egg:swift#container

[filter:recon]
use = egg:swift#recon

[container-replicator]
vm_test_mode = yes

[container-updater]

[container-auditor]

[container-sync]

Create `/etc/swift/container-server/3.conf`:

[DEFAULT]
devices = /srv/3/node
mount_check = false
disable_fallocate = true
bind_port = 6031
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL4
recon_cache_path = /var/cache/swift3
eventlet_debug = true

[pipeline:main]
pipeline = recon container-server

[app:container-server]
use = egg:swift#container

[filter:recon]
use = egg:swift#recon

[container-replicator]
vm_test_mode = yes

[container-updater]

[container-auditor]

[container-sync]

Create `/etc/swift/container-server/4.conf`:

[DEFAULT]
devices = /srv/4/node
mount_check = false
disable_fallocate = true
bind_port = 6041
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL5
recon_cache_path = /var/cache/swift4
eventlet_debug = true

[pipeline:main]
pipeline = recon container-server

[app:container-server]
use = egg:swift#container

[filter:recon]
use = egg:swift#recon

[container-replicator]
vm_test_mode = yes

[container-updater]

[container-auditor]

[container-sync]

Create `/etc/swift/object-server/1.conf`:

[DEFAULT]
devices = /srv/1/node
mount_check = false
disable_fallocate = true
bind_port = 6010
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL2
recon_cache_path = /var/cache/swift
eventlet_debug = true

[pipeline:main]
pipeline = recon object-server

[app:object-server]
use = egg:swift#object

[filter:recon]
use = egg:swift#recon

[object-replicator]
vm_test_mode = yes

[object-updater]

[object-auditor]

Create `/etc/swift/object-server/2.conf`:

[DEFAULT]
devices = /srv/2/node
mount_check = false
disable_fallocate = true
bind_port = 6020
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL3
recon_cache_path = /var/cache/swift2
eventlet_debug = true

[pipeline:main]
pipeline = recon object-server

[app:object-server]
use = egg:swift#object

[filter:recon]
use = egg:swift#recon

[object-replicator]
vm_test_mode = yes

[object-updater]

[object-auditor]

Create `/etc/swift/object-server/3.conf`:

[DEFAULT]
devices = /srv/3/node
mount_check = false
disable_fallocate = true
bind_port = 6030
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL4
recon_cache_path = /var/cache/swift3
eventlet_debug = true

[pipeline:main]
pipeline = recon object-server

[app:object-server]
use = egg:swift#object

[filter:recon]
use = egg:swift#recon

[object-replicator]
vm_test_mode = yes

[object-updater]

[object-auditor]

Create `/etc/swift/object-server/4.conf`:

[DEFAULT]
devices = /srv/4/node
mount_check = false
disable_fallocate = true
bind_port = 6040
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL5
recon_cache_path = /var/cache/swift4
eventlet_debug = true

[pipeline:main]
pipeline = recon object-server

[app:object-server]
use = egg:swift#object

[filter:recon]
use = egg:swift#recon

[object-replicator]
vm_test_mode = yes

[object-updater]

[object-auditor]

Update <your-user-name>:

find /etc/swift/ -name \*.conf | xargs sed -i "s/<your-user-name>/${USER}/"

Setting up scripts for running Swift

mkdir ~/bin
Create ~/bin/resetswift.

If you are using a loopback device substitute /dev/sdb1 with /srv/swift-disk.

If you did not set up rsyslog for individual logging, remove the find /var/log/swift... line:
#!/bin/bash

swift-init all stop
find /var/log/swift -type f -exec rm -f {} \;
sudo umount /mnt/sdb1
sudo mkfs.xfs -f /dev/sdb1
sudo mount /mnt/sdb1
sudo mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4
sudo chown <your-user-name>:<your-group-name> /mnt/sdb1/*
mkdir -p /srv/1/node/sdb1 /srv/2/node/sdb2 /srv/3/node/sdb3 /srv/4/node/sdb4
sudo rm -f /var/log/debug /var/log/messages /var/log/rsyncd.log /var/log/syslog
find /var/cache/swift* -type f -name *.recon -exec rm -f {} \;
sudo service rsyslog restart
sudo service memcached restart
Create `~/bin/remakerings`:
#!/bin/bash

cd /etc/swift

rm -f *.builder *.ring.gz backups/*.builder backups/*.ring.gz

swift-ring-builder object.builder create 10 3 1
swift-ring-builder object.builder add r1z1-127.0.0.1:6010/sdb1 1
swift-ring-builder object.builder add r1z2-127.0.0.1:6020/sdb2 1
swift-ring-builder object.builder add r1z3-127.0.0.1:6030/sdb3 1
swift-ring-builder object.builder add r1z4-127.0.0.1:6040/sdb4 1
swift-ring-builder object.builder rebalance
swift-ring-builder container.builder create 10 3 1
swift-ring-builder container.builder add r1z1-127.0.0.1:6011/sdb1 1
swift-ring-builder container.builder add r1z2-127.0.0.1:6021/sdb2 1
swift-ring-builder container.builder add r1z3-127.0.0.1:6031/sdb3 1
swift-ring-builder container.builder add r1z4-127.0.0.1:6041/sdb4 1
swift-ring-builder container.builder rebalance
swift-ring-builder account.builder create 10 3 1
swift-ring-builder account.builder add r1z1-127.0.0.1:6012/sdb1 1
swift-ring-builder account.builder add r1z2-127.0.0.1:6022/sdb2 1
swift-ring-builder account.builder add r1z3-127.0.0.1:6032/sdb3 1
swift-ring-builder account.builder add r1z4-127.0.0.1:6042/sdb4 1
swift-ring-builder account.builder rebalance
Create `~/bin/startmain`:
#!/bin/bash

swift-init main start
Create `~/bin/startrest`:
#!/bin/bash

swift-init rest start
chmod +x ~/bin/*
Edit ~/.bashrc and add to the end:
export SWIFT_TEST_CONFIG_FILE=/etc/swift/test.conf
export PATH=${PATH}:~/bin
. ~/.bashrc

remakerings

cp ~/swift/test/sample.conf /etc/swift/test.conf

~/swift/.unittests

startmain (The Unable to increase file descriptor limit. Running as non-root? warnings are expected and ok.)

Get an X-Storage-Url and `X-Auth-Token`: curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://127.0.0.1:8080/auth/v1.0

Check that you can GET account: curl -v -H 'X-Auth-Token: <token-from-x-auth-token-above>' <url-from-x-storage-url-above>

Check that swift works: swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat

~/swift/.functests (Note: functional tests will first delete everything in the configured accounts.)

~/swift/.probetests (Note: probe tests will reset your environment as they call resetswift for each test.)

Debugging Issues

If all doesn't go as planned, and tests fail, or you can't auth, or something doesn't work, here are some good starting places to look for issues:

Everything is logged using system facilities -- usually in /var/log/syslog, but possibly in /var/log/messages on e.g. Fedora -- so that is a good first place to look for errors (most likely python tracebacks).
Make sure all of the server processes are running. For the base functionality, the Proxy, Account, Container, and Object servers should be running.
If one of the servers are not running, and no errors are logged to syslog, it may be useful to try to start the server manually, for example: swift-object-server /etc/swift/object-server/1.conf will start the object server. If there are problems not showing up in syslog, then you will likely see the traceback on startup.
If you need to, you can turn off syslog for unit tests. This can be useful for environments where /dev/log is unavailable, or which cannot rate limit (unit tests generate a lot of logs very quickly). Open the file SWIFT_TEST_CONFIG_FILE points to, and change the value of fake_syslog to True.

24 KiB Raw Blame History