openstack/swift
Code Issues Proposed changes
39f46f552e
swift/setup.cfg
Christian Schwede e1140666d6 Add support to increase object ring partition power 
This patch adds methods to increase the partition power of an existing
object ring without downtime for the users using a 3-step process. Data
won't be moved to other nodes; objects using the new increased partition
power will be located on the same device and are hardlinked to avoid
data movement.

1. A new setting "next_part_power" will be added to the rings, and once
the proxy server reloaded the rings it will send this value to the
object servers on any write operation. Object servers will now create a
hard-link in the new location to the original DiskFile object. Already
existing data will be relinked using a new tool in the new locations
using hardlinks.

2. The actual partition power itself will be increased. Servers will now
use the new partition power to read from and write to. No longer
required hard links in the old object location have to be removed now by
the relinker tool; the relinker tool reads the next_part_power setting
to find object locations that need to be cleaned up.

3. The "next_part_power" flag will be removed.

This mostly implements the spec in [1]; however it's not using an
"epoch" as described there. The idea of the epoch was to store data
using different partition powers in their own namespace to avoid
conflicts with auditors and replicators as well as being able to abort
such an operation and just remove the new tree.  This would require some
heavy change of the on-disk data layout, and other object-server
implementations would be required to adopt this scheme too.

Instead the object-replicator is now aware that there is a partition
power increase in progress and will skip replication of data in that
storage policy; the relinker tool should be simply run and afterwards
the partition power will be increased. This shouldn't take that much
time (it's only walking the filesystem and hardlinking); impact should
be low therefore. The relinker should be run on all storage nodes at the
same time in parallel to decrease the required time (though this is not
mandatory). Failures during relinking should not affect cluster
operations - relinking can be even aborted manually and restarted later.

Auditors are not quarantining objects written to a path with a different
partition power and therefore working as before (though they are reading
each object twice in the worst case before the no longer needed hard
links are removed).

Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>

[1] https://specs.openstack.org/openstack/swift-specs/specs/in_progress/
increasing_partition_power.html

Change-Id: I7d6371a04f5c1c4adbb8733a71f3c177ee5448bb
2017-06-15 15:08:48 -07:00

135 lines
4.3 KiB
INI
Raw Blame History

[metadata]
name = swift
summary = OpenStack Object Storage
description-file =
README.rst
author = OpenStack
author-email = openstack-dev@lists.openstack.org
home-page = http://docs.openstack.org/developer/swift/
classifier =
Development Status :: 5 - Production/Stable
Environment :: OpenStack
Intended Audience :: Information Technology
Intended Audience :: System Administrators
License :: OSI Approved :: Apache Software License
Operating System :: POSIX :: Linux
Programming Language :: Python
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
[pbr]
skip_authors = True
skip_changelog = True
[files]
packages =
swift
scripts =
bin/swift-account-audit
bin/swift-account-auditor
bin/swift-account-info
bin/swift-account-reaper
bin/swift-account-replicator
bin/swift-account-server
bin/swift-config
bin/swift-container-auditor
bin/swift-container-info
bin/swift-container-replicator
bin/swift-container-server
bin/swift-container-sync
bin/swift-container-updater
bin/swift-container-reconciler
bin/swift-reconciler-enqueue
bin/swift-dispersion-populate
bin/swift-dispersion-report
bin/swift-drive-audit
bin/swift-form-signature
bin/swift-get-nodes
bin/swift-init
bin/swift-object-auditor
bin/swift-object-expirer
bin/swift-object-info
bin/swift-object-replicator
bin/swift-object-reconstructor
bin/swift-object-relinker
bin/swift-object-server
bin/swift-object-updater
bin/swift-oldies
bin/swift-orphans
bin/swift-proxy-server
bin/swift-recon
bin/swift-recon-cron
bin/swift-ring-builder
bin/swift-ring-builder-analyzer
bin/swift-temp-url
[entry_points]
paste.app_factory =
proxy = swift.proxy.server:app_factory
object = swift.obj.server:app_factory
mem_object = swift.obj.mem_server:app_factory
container = swift.container.server:app_factory
account = swift.account.server:app_factory
paste.filter_factory =
healthcheck = swift.common.middleware.healthcheck:filter_factory
crossdomain = swift.common.middleware.crossdomain:filter_factory
memcache = swift.common.middleware.memcache:filter_factory
ratelimit = swift.common.middleware.ratelimit:filter_factory
cname_lookup = swift.common.middleware.cname_lookup:filter_factory
catch_errors = swift.common.middleware.catch_errors:filter_factory
domain_remap = swift.common.middleware.domain_remap:filter_factory
staticweb = swift.common.middleware.staticweb:filter_factory
tempauth = swift.common.middleware.tempauth:filter_factory
keystoneauth = swift.common.middleware.keystoneauth:filter_factory
recon = swift.common.middleware.recon:filter_factory
tempurl = swift.common.middleware.tempurl:filter_factory
formpost = swift.common.middleware.formpost:filter_factory
name_check = swift.common.middleware.name_check:filter_factory
bulk = swift.common.middleware.bulk:filter_factory
container_quotas = swift.common.middleware.container_quotas:filter_factory
account_quotas = swift.common.middleware.account_quotas:filter_factory
proxy_logging = swift.common.middleware.proxy_logging:filter_factory
dlo = swift.common.middleware.dlo:filter_factory
slo = swift.common.middleware.slo:filter_factory
list_endpoints = swift.common.middleware.list_endpoints:filter_factory
gatekeeper = swift.common.middleware.gatekeeper:filter_factory
container_sync = swift.common.middleware.container_sync:filter_factory
xprofile = swift.common.middleware.xprofile:filter_factory
versioned_writes = swift.common.middleware.versioned_writes:filter_factory
copy = swift.common.middleware.copy:filter_factory
keymaster = swift.common.middleware.crypto.keymaster:filter_factory
encryption = swift.common.middleware.crypto:filter_factory
[build_sphinx]
all_files = 1
build-dir = doc/build
source-dir = doc/source
[egg_info]
tag_build =
tag_date = 0
tag_svn_revision = 0
[compile_catalog]
directory = swift/locale
domain = swift
[update_catalog]
domain = swift
output_dir = swift/locale
input_file = swift/locale/swift.pot
[extract_messages]
keywords = _ l_ lazy_gettext
mapping_file = babel.cfg
output_file = swift/locale/swift.pot
[nosetests]
exe = 1
verbosity = 2
detailed-errors = 1
cover-package = swift
cover-html = true
cover-erase = true
View Git Blame Copy Permalink