openstack/swift
Code Issues Proposed changes

docs: clean up SAIO formatting

Browse Source 
Drive-by: use six.moves in s3api; fix "unexpected indent" warning when building
docs on py3

Change-Id: I2a354e2624c763a68fcea7a6404e9c2fde30d631
This commit is contained in:
Tim Burke 2019-03-04 17:37:09 -08:00
parent 96013436a1
commit d185b607bb
3 changed files with 332 additions and 291 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

165
doc/source/development_saio.rst
View File

@ -76,7 +76,8 @@ Installing dependencies
python2-netifaces python2-pip python2-dnspython \
python2-mock
Note: This installs necessary system dependencies and *most* of the python
.. note::
This installs necessary system dependencies and *most* of the python
dependencies. Later in the process setuptools/distribute or pip will install
and/or upgrade packages.
@ -90,16 +91,16 @@ Using a partition for storage
If you are going to use a separate partition for Swift data, be sure to add
another device when creating the VM, and follow these instructions:
#. Set up a single partition::
#. Set up a single partition::
sudo fdisk /dev/sdb
sudo mkfs.xfs /dev/sdb1
#. Edit ``/etc/fstab`` and add::
#. Edit ``/etc/fstab`` and add::
/dev/sdb1 /mnt/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0
#. Create the mount point and the individualized links::
#. Create the mount point and the individualized links::
sudo mkdir /mnt/sdb1
sudo mount /mnt/sdb1
@ -116,14 +117,16 @@ another device when creating the VM, and follow these instructions:
# **Make sure to include the trailing slash after /srv/$x/**
for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done
Note: For OpenSuse users, a user's primary group is `users`, so you have 2 options:
.. note::
For OpenSuse users, a user's primary group is ``users``, so you have 2 options:
* Change `${USER}:${USER}` to `${USER}:users` in all references of this guide; or
* Change ``${USER}:${USER}`` to ``${USER}:users`` in all references of this guide; or
* Create a group for your username and add yourself to it::
sudo groupadd ${USER} && sudo gpasswd -a ${USER} ${USER}
Note: We create the mount points and mount the storage disk under
.. note::
We create the mount points and mount the storage disk under
/mnt/sdb1. This disk will contain one directory per simulated swift node,
each owned by the current swift user.
@ -133,7 +136,7 @@ another device when creating the VM, and follow these instructions:
exist. This prevents disk sync operations from writing to the root
partition in the event a drive is unmounted.
#. Next, skip to :ref:`common-dev-section`.
#. Next, skip to :ref:`common-dev-section`.
.. _loopback-section:
@ -144,7 +147,7 @@ Using a loopback device for storage
If you want to use a loopback device instead of another partition, follow
these instructions:
#. Create the file for the loopback device::
#. Create the file for the loopback device::
sudo mkdir /srv
sudo truncate -s 1GB /srv/swift-disk
@ -153,11 +156,11 @@ these instructions:
Modify size specified in the ``truncate`` command to make a larger or
smaller partition as needed.
#. Edit `/etc/fstab` and add::
#. Edit `/etc/fstab` and add::
/srv/swift-disk /mnt/sdb1 xfs loop,noatime,nodiratime,nobarrier,logbufs=8 0 0
#. Create the mount point and the individualized links::
#. Create the mount point and the individualized links::
sudo mkdir /mnt/sdb1
sudo mount /mnt/sdb1
@ -173,14 +176,16 @@ these instructions:
# **Make sure to include the trailing slash after /srv/$x/**
for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done
Note: For OpenSuse users, a user's primary group is `users`, so you have 2 options:
.. note::
For OpenSuse users, a user's primary group is ``users``, so you have 2 options:
* Change `${USER}:${USER}` to `${USER}:users` in all references of this guide; or
* Change ``${USER}:${USER}`` to ``${USER}:users`` in all references of this guide; or
* Create a group for your username and add yourself to it::
sudo groupadd ${USER} && sudo gpasswd -a ${USER} ${USER}
Note: We create the mount points and mount the loopback file under
.. note::
We create the mount points and mount the loopback file under
/mnt/sdb1. This file will contain one directory per simulated swift node,
each owned by the current swift user.
@ -229,11 +234,11 @@ To persist this, edit and add the following to ``/etc/fstab``::
Getting the code
----------------
#. Check out the python-swiftclient repo::
#. Check out the python-swiftclient repo::
cd $HOME; git clone https://github.com/openstack/python-swiftclient.git
#. Build a development installation of python-swiftclient::
#. Build a development installation of python-swiftclient::
cd $HOME/python-swiftclient; sudo python setup.py develop; cd -
@ -242,23 +247,25 @@ Getting the code
cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; sudo python setup.py develop; cd -
#. Check out the swift repo::
#. Check out the swift repo::
git clone https://github.com/openstack/swift.git
#. Build a development installation of swift::
#. Build a development installation of swift::
cd $HOME/swift; sudo pip install --no-binary cryptography -r requirements.txt; sudo python setup.py develop; cd -
Note: Due to a difference in libssl.so naming in OpenSuse to other Linux distros the wheel/binary wont work so the
cryptography must be built, thus the ``--no-binary cryptography``.
.. note::
Due to a difference in how ``libssl.so`` is named in OpenSuse vs. other Linux distros the
wheel/binary won't work; thus we use ``--no-binary cryptography`` to build ``cryptography``
locally.
Fedora 19 or later users might have to perform the following if development
installation of swift fails::
sudo pip install -U xattr
#. Install swift's test dependencies::
#. Install swift's test dependencies::
cd $HOME/swift; sudo pip install -r test-requirements.txt
@ -266,7 +273,7 @@ Getting the code
Setting up rsync
----------------
#. Create ``/etc/rsyncd.conf``::
#. Create ``/etc/rsyncd.conf``::
sudo cp $HOME/swift/doc/saio/rsyncd.conf /etc/
sudo sed -i "s/<your-user-name>/${USER}/" /etc/rsyncd.conf
@ -275,8 +282,9 @@ Setting up rsync
that is copied and fixed up above:
.. literalinclude:: /../saio/rsyncd.conf
:language: ini
#. On Ubuntu, edit the following line in ``/etc/default/rsync``::
#. On Ubuntu, edit the following line in ``/etc/default/rsync``::
RSYNC_ENABLE=true
@ -288,7 +296,7 @@ Setting up rsync
On OpenSuse, nothing needs to happen here.
#. On platforms with SELinux in ``Enforcing`` mode, either set to ``Permissive``::
#. On platforms with SELinux in ``Enforcing`` mode, either set to ``Permissive``::
sudo setenforce Permissive
@ -296,7 +304,7 @@ Setting up rsync
sudo setsebool -P rsync_full_access 1
#. Start the rsync daemon
#. Start the rsync daemon
* On Ubuntu 14.04, run::
@ -322,7 +330,7 @@ Setting up rsync
sudo service xinetd restart
#. Verify rsync is accepting connections for all servers::
#. Verify rsync is accepting connections for all servers::
rsync rsync://pub@localhost/
@ -362,7 +370,7 @@ running, tokens cannot be validated, and accessing Swift becomes impossible.
Optional: Setting up rsyslog for individual logging
---------------------------------------------------
#. Install the swift rsyslogd configuration::
#. Install the swift rsyslogd configuration::
sudo cp $HOME/swift/doc/saio/rsyslog.d/10-swift.conf /etc/rsyslog.d/
@ -379,13 +387,14 @@ Optional: Setting up rsyslog for individual logging
below:
.. literalinclude:: /../saio/rsyslog.d/10-swift.conf
:language: ini
#. Edit ``/etc/rsyslog.conf`` and make the following change (usually in the
#. Edit ``/etc/rsyslog.conf`` and make the following change (usually in the
"GLOBAL DIRECTIVES" section)::
$PrivDropToGroup adm
#. If using hourly logs (see above) perform::
#. If using hourly logs (see above) perform::
sudo mkdir -p /var/log/swift/hourly
@ -393,7 +402,7 @@ Optional: Setting up rsyslog for individual logging
sudo mkdir -p /var/log/swift
#. Setup the logging directory and start syslog:
#. Setup the logging directory and start syslog:
* On Ubuntu::
@ -415,89 +424,106 @@ After performing the following steps, be sure to verify that Swift has access
to resulting configuration files (sample configuration files are provided with
all defaults in line-by-line comments).
#. Optionally remove an existing swift directory::
#. Optionally remove an existing swift directory::
sudo rm -rf /etc/swift
#. Populate the ``/etc/swift`` directory itself::
#. Populate the ``/etc/swift`` directory itself::
cd $HOME/swift/doc; sudo cp -r saio/swift /etc/swift; cd -
sudo chown -R ${USER}:${USER} /etc/swift
#. Update ``<your-user-name>`` references in the Swift config files::
#. Update ``<your-user-name>`` references in the Swift config files::
find /etc/swift/ -name \*.conf | xargs sudo sed -i "s/<your-user-name>/${USER}/"
The contents of the configuration files provided by executing the above
commands are as follows:
#. ``/etc/swift/swift.conf``
#. ``/etc/swift/swift.conf``
.. literalinclude:: /../saio/swift/swift.conf
:language: ini
#. ``/etc/swift/proxy-server.conf``
#. ``/etc/swift/proxy-server.conf``
.. literalinclude:: /../saio/swift/proxy-server.conf
:language: ini
#. ``/etc/swift/object-expirer.conf``
#. ``/etc/swift/object-expirer.conf``
.. literalinclude:: /../saio/swift/object-expirer.conf
:language: ini
#. ``/etc/swift/container-reconciler.conf``
#. ``/etc/swift/container-reconciler.conf``
.. literalinclude:: /../saio/swift/container-reconciler.conf
:language: ini
#. ``/etc/swift/container-sync-realms.conf``
#. ``/etc/swift/container-sync-realms.conf``
.. literalinclude:: /../saio/swift/container-sync-realms.conf
:language: ini
#. ``/etc/swift/account-server/1.conf``
#. ``/etc/swift/account-server/1.conf``
.. literalinclude:: /../saio/swift/account-server/1.conf
:language: ini
#. ``/etc/swift/container-server/1.conf``
#. ``/etc/swift/container-server/1.conf``
.. literalinclude:: /../saio/swift/container-server/1.conf
:language: ini
#. ``/etc/swift/object-server/1.conf``
#. ``/etc/swift/object-server/1.conf``
.. literalinclude:: /../saio/swift/object-server/1.conf
:language: ini
#. ``/etc/swift/account-server/2.conf``
#. ``/etc/swift/account-server/2.conf``
.. literalinclude:: /../saio/swift/account-server/2.conf
:language: ini
#. ``/etc/swift/container-server/2.conf``
#. ``/etc/swift/container-server/2.conf``
.. literalinclude:: /../saio/swift/container-server/2.conf
:language: ini
#. ``/etc/swift/object-server/2.conf``
#. ``/etc/swift/object-server/2.conf``
.. literalinclude:: /../saio/swift/object-server/2.conf
:language: ini
#. ``/etc/swift/account-server/3.conf``
#. ``/etc/swift/account-server/3.conf``
.. literalinclude:: /../saio/swift/account-server/3.conf
:language: ini
#. ``/etc/swift/container-server/3.conf``
#. ``/etc/swift/container-server/3.conf``
.. literalinclude:: /../saio/swift/container-server/3.conf
:language: ini
#. ``/etc/swift/object-server/3.conf``
#. ``/etc/swift/object-server/3.conf``
.. literalinclude:: /../saio/swift/object-server/3.conf
:language: ini
#. ``/etc/swift/account-server/4.conf``
#. ``/etc/swift/account-server/4.conf``
.. literalinclude:: /../saio/swift/account-server/4.conf
:language: ini
#. ``/etc/swift/container-server/4.conf``
#. ``/etc/swift/container-server/4.conf``
.. literalinclude:: /../saio/swift/container-server/4.conf
:language: ini
#. ``/etc/swift/object-server/4.conf``
#. ``/etc/swift/object-server/4.conf``
.. literalinclude:: /../saio/swift/object-server/4.conf
:language: ini
.. _setup_scripts:
@ -505,17 +531,18 @@ commands are as follows:
Setting up scripts for running Swift
------------------------------------
#. Copy the SAIO scripts for resetting the environment::
#. Copy the SAIO scripts for resetting the environment::
mkdir -p $HOME/bin
cd $HOME/swift/doc; cp saio/bin/* $HOME/bin; cd -
chmod +x $HOME/bin/*
#. Edit the ``$HOME/bin/resetswift`` script
#. Edit the ``$HOME/bin/resetswift`` script
The template ``resetswift`` script looks like the following:
.. literalinclude:: /../saio/bin/resetswift
:language: bash
If you are using a loopback device add an environment var to
substitute ``/dev/sdb1`` with ``/srv/swift-disk``::
@ -528,39 +555,44 @@ Setting up scripts for running Swift
sed -i "/find \/var\/log\/swift/d" $HOME/bin/resetswift
#. Install the sample configuration file for running tests::
#. Install the sample configuration file for running tests::
cp $HOME/swift/test/sample.conf /etc/swift/test.conf
The template ``test.conf`` looks like the following:
.. literalinclude:: /../../test/sample.conf
:language: ini
#. Add an environment variable for running tests below::
#. Add an environment variable for running tests below::
echo "export SWIFT_TEST_CONFIG_FILE=/etc/swift/test.conf" >> $HOME/.bashrc
#. Be sure that your ``PATH`` includes the ``bin`` directory::
#. Be sure that your ``PATH`` includes the ``bin`` directory::
echo "export PATH=${PATH}:$HOME/bin" >> $HOME/.bashrc
#. Source the above environment variables into your current environment::
#. Source the above environment variables into your current environment::
. $HOME/.bashrc
#. Construct the initial rings using the provided script::
#. Construct the initial rings using the provided script::
remakerings
The ``remakerings`` script looks like the following:
.. literalinclude:: /../saio/bin/remakerings
:language: bash
You can expect the output from this command to produce the following. Note
that 3 object rings are created in order to test storage policies and EC in
the SAIO environment. The EC ring is the only one with all 8 devices.
There are also two replication rings, one for 3x replication and another
for 2x replication, but those rings only use 4 devices::
for 2x replication, but those rings only use 4 devices:
.. code-block:: console
Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0
Device d1r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb2_"" with 1.0 weight got id 1
@ -593,15 +625,15 @@ Setting up scripts for running Swift
Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00
#. Read more about Storage Policies and your SAIO :doc:`policies_saio`
#. Read more about Storage Policies and your SAIO :doc:`policies_saio`
#. Verify the unit tests run::
#. Verify the unit tests run::
$HOME/swift/.unittests
Note that the unit tests do not require any swift daemons running.
#. Start the "main" Swift daemon processes (proxy, account, container, and
#. Start the "main" Swift daemon processes (proxy, account, container, and
object)::
startmain
@ -612,27 +644,28 @@ Setting up scripts for running Swift
The ``startmain`` script looks like the following:
.. literalinclude:: /../saio/bin/startmain
:language: bash
#. Get an ``X-Storage-Url`` and ``X-Auth-Token``::
#. Get an ``X-Storage-Url`` and ``X-Auth-Token``::
curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://127.0.0.1:8080/auth/v1.0
#. Check that you can ``GET`` account::
#. Check that you can ``GET`` account::
curl -v -H 'X-Auth-Token: <token-from-x-auth-token-above>' <url-from-x-storage-url-above>
#. Check that ``swift`` command provided by the python-swiftclient package works::
#. Check that ``swift`` command provided by the python-swiftclient package works::
swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat
#. Verify the functional tests run::
#. Verify the functional tests run::
$HOME/swift/.functests
(Note: functional tests will first delete everything in the configured
accounts.)
#. Verify the probe tests run::
#. Verify the probe tests run::
$HOME/swift/.probetests

8
swift/common/bufferedhttp.py
View File

@ -178,6 +178,14 @@ class BufferedHTTPConnection(HTTPConnection):
return ret
def putrequest(self, method, url, skip_host=0, skip_accept_encoding=0):
'''Send a request to the server.
:param method: specifies an HTTP request method, e.g. 'GET'.
:param url: specifies the object being requested, e.g. '/index.html'.
:param skip_host: if True does not add automatically a 'Host:' header
:param skip_accept_encoding: if True does not add automatically an
'Accept-Encoding:' header
'''
self._method = method
self._path = url
return HTTPConnection.putrequest(self, method, url, skip_host,

2
swift/common/middleware/s3api/controllers/s3_acl.py
View File

@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
from urllib import quote
from six.moves.urllib.parse import quote
from swift.common.utils import public
from swift.common.middleware.s3api.controllers.base import Controller