Enable to configure object-expirer in object-server.conf
To prepare for object-expirer's general task queue feature [1], this patch enables to configure object-expirer in object-server.conf. Object-expirer.conf can be used in the same manner as before, but deprecated. If both of object-server.conf with "object-expirer" section and object-expirer.conf are in a node, only object-server.conf is used. Object-expirer.conf is used only if all object-server.conf doesn't have "object-expirer" section. There are two differences between "object-expirer.conf" style and "object-server.conf" style. The first difference is `dequeue_from_legacy` default value. `dequeue_from_legacy` defines task queue mode. In "object-expirer.conf" style, the default mode is legacy queue. In "object-server.conf" style, the default mode is general queue. But general mode means no-op mode for now, because general task queue is not implemented yet. The second difference is internal client config. In "object-expirer.conf" style, config file of internal client is the object-expirer.conf itself. In "object-server.conf" style, config file of internal client is another file. [1]: https://review.openstack.org/#/c/517389/ Co-Authored-By: Matthew Oliver <matt@oliver.net.au> Change-Id: Ib21568f9b9d8547da87a99d65ae73a550e9c3230
This commit is contained in:
Kazuhiro MIYAHARA
Kota Tsuyuzaki
parent
b8824f052b
commit
443f029a58
@ -33,7 +33,8 @@
|
|||||||
.SH DESCRIPTION
|
.SH DESCRIPTION
|
||||||
.PP
|
.PP
|
||||||
This is the configuration file used by the object server and other object
|
This is the configuration file used by the object server and other object
|
||||||
background services, such as; replicator, reconstructor, updater and auditor.
|
background services, such as; replicator, reconstructor, updater, auditor, and
|
||||||
|
expirer.
|
||||||
|
|
||||||
The configuration file follows the python-pastedeploy syntax. The file is divided
|
The configuration file follows the python-pastedeploy syntax. The file is divided
|
||||||
into sections, which are enclosed by square brackets. Each section will contain a
|
into sections, which are enclosed by square brackets. Each section will contain a
|
||||||
@ -88,6 +89,7 @@ Disable pre-allocate disk space for a file. The default is false.
|
|||||||
.IP \fBexpiring_objects_container_divisor\fR
|
.IP \fBexpiring_objects_container_divisor\fR
|
||||||
The default is 86400.
|
The default is 86400.
|
||||||
.IP \fBexpiring_objects_account_name\fR
|
.IP \fBexpiring_objects_account_name\fR
|
||||||
|
Account name used for legacy style expirer task queue.
|
||||||
The default is 'expiring_objects'.
|
The default is 'expiring_objects'.
|
||||||
.IP \fBservers_per_port\fR
|
.IP \fBservers_per_port\fR
|
||||||
Make object-server run this many worker processes per unique port of "local"
|
Make object-server run this many worker processes per unique port of "local"
|
||||||
@ -581,6 +583,64 @@ Ignored if IOPRIO_CLASS_IDLE is set.
|
|||||||
.RE
|
.RE
|
||||||
|
|
||||||
|
|
||||||
|
.RS 0
|
||||||
|
.IP "\fB[object-expirer]\fR"
|
||||||
|
.RE
|
||||||
|
.RS 3
|
||||||
|
.IP \fBlog_name\fR
|
||||||
|
Label used when logging. The default is object-expirer.
|
||||||
|
.IP \fBlog_facility\fR
|
||||||
|
Syslog log facility. The default is LOG_LOCAL0.
|
||||||
|
.IP \fBlog_level\fR
|
||||||
|
Logging level. The default is INFO.
|
||||||
|
.IP \fBlog_address\fR
|
||||||
|
Logging address. The default is /dev/log.
|
||||||
|
.IP \fBinterval\fR
|
||||||
|
Minimum time for a pass to take. The default is 300 seconds.
|
||||||
|
.IP \fBreport_interval\fR
|
||||||
|
Minimum time for a pass to report. The default is 300 seconds.
|
||||||
|
.IP \fBrequest_tries\fR
|
||||||
|
The number of times the expirer's internal client will
|
||||||
|
attempt any given request in the event of failure. The default is 3.
|
||||||
|
.IP \fBconcurrency\fR
|
||||||
|
Number of expirer workers to spawn. The default is 1.
|
||||||
|
.IP \fBdequeue_from_legacy\fR
|
||||||
|
The flag to execute legacy style expirer tasks. The default is false.
|
||||||
|
.IP \fBprocesses\fR
|
||||||
|
Processes can only be used in conjunction with `dequeue_from_legacy`.
|
||||||
|
Processes is how many parts to divide the legacy work into, one part per process that will be doing the work.
|
||||||
|
Processes set 0 means that a single process will be doing all the legacy work.
|
||||||
|
Processes can also be specified on the command line and will override the config value.
|
||||||
|
The default is 0.
|
||||||
|
.IP \fBprocess\fR
|
||||||
|
Process can only be used in conjunction with `dequeue_from_legacy`.
|
||||||
|
Process is which of the parts a particular legacy process will work on process can also be specified
|
||||||
|
on the command line and will override the config value process is "zero based", if you want
|
||||||
|
to use 3 processes, you should run processes with process set to 0, 1, and 2. The default is 0.
|
||||||
|
.IP \fBreclaim_age\fR
|
||||||
|
The expirer will re-attempt expiring if the source object is not available up
|
||||||
|
to reclaim_age seconds before it gives up and deletes the task in the queue.
|
||||||
|
The default is 604800 seconds (= 1 week).
|
||||||
|
.IP \fBrecon_cache_path\fR
|
||||||
|
Path to recon cache directory. The default is /var/cache/swift
|
||||||
|
.IP \fBnice_priority\fR
|
||||||
|
Modify scheduling priority of server processes. Niceness values range from -20
|
||||||
|
(most favorable to the process) to 19 (least favorable to the process).
|
||||||
|
The default does not modify priority.
|
||||||
|
.IP \fBionice_class\fR
|
||||||
|
Modify I/O scheduling class of server processes. I/O niceness class values
|
||||||
|
are IOPRIO_CLASS_RT (realtime), IOPRIO_CLASS_BE (best-effort) and IOPRIO_CLASS_IDLE (idle).
|
||||||
|
The default does not modify class and priority.
|
||||||
|
Work only with ionice_priority.
|
||||||
|
.IP \fBionice_priority\fR
|
||||||
|
Modify I/O scheduling priority of server processes. I/O niceness priority
|
||||||
|
is a number which goes from 0 to 7. The higher the value, the lower
|
||||||
|
the I/O priority of the process. Work only with ionice_class.
|
||||||
|
Ignored if IOPRIO_CLASS_IDLE is set.
|
||||||
|
.RE
|
||||||
|
.PD
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
.SH DOCUMENTATION
|
.SH DOCUMENTATION
|
||||||
|
|||||||
@ -70,5 +70,6 @@ and also about OpenStack Swift as a whole can be found at
|
|||||||
|
|
||||||
|
|
||||||
.SH "SEE ALSO"
|
.SH "SEE ALSO"
|
||||||
|
.BR object-server.conf(5)
|
||||||
.BR object-expirer.conf(5)
|
.BR object-expirer.conf(5)
|
||||||
|
|
||||||
|
|||||||
@ -419,6 +419,7 @@ The following configuration sections are available:
|
|||||||
* `[object-reconstructor]`_
|
* `[object-reconstructor]`_
|
||||||
* `[object-updater]`_
|
* `[object-updater]`_
|
||||||
* `[object-auditor]`_
|
* `[object-auditor]`_
|
||||||
|
* `[object-expirer]`_
|
||||||
|
|
||||||
.. _object-server-default-options:
|
.. _object-server-default-options:
|
||||||
|
|
||||||
@ -1002,6 +1003,69 @@ ionice_priority None I/O scheduling priority of serve
|
|||||||
Ignored if IOPRIO_CLASS_IDLE is set.
|
Ignored if IOPRIO_CLASS_IDLE is set.
|
||||||
=========================== =================== ==========================================
|
=========================== =================== ==========================================
|
||||||
|
|
||||||
|
****************
|
||||||
|
[object-expirer]
|
||||||
|
****************
|
||||||
|
|
||||||
|
============================= =============================== ==========================================
|
||||||
|
Option Default Description
|
||||||
|
----------------------------- ------------------------------- ------------------------------------------
|
||||||
|
log_name object-expirer Label used when logging
|
||||||
|
log_facility LOG_LOCAL0 Syslog log facility
|
||||||
|
log_level INFO Logging level
|
||||||
|
log_address /dev/log Logging directory
|
||||||
|
interval 300 Time in seconds to wait between
|
||||||
|
expirer passes
|
||||||
|
report_interval 300 Frequency of status logs in seconds.
|
||||||
|
auto_create_account_prefix . Prefix used when automatically
|
||||||
|
creating accounts.
|
||||||
|
concurrency 1 Level of concurrency to use to do the work,
|
||||||
|
this value must be set to at least 1
|
||||||
|
expiring_objects_account_name expiring_objects name for legacy expirer task queue
|
||||||
|
dequeue_from_legacy False This service will look for jobs on the legacy expirer task queue.
|
||||||
|
processes 0 How many parts to divide the legacy work into,
|
||||||
|
one part per process that will be doing the work.
|
||||||
|
When set 0 means that a single legacy
|
||||||
|
process will be doing all the work.
|
||||||
|
This can only be used in conjunction with
|
||||||
|
`dequeue_from_legacy`.
|
||||||
|
process 0 Which of the parts a particular legacy process will
|
||||||
|
work on. It is "zero based", if you want to use 3
|
||||||
|
processes, you should run processes with process
|
||||||
|
set to 0, 1, and 2.
|
||||||
|
This can only be used in conjunction with
|
||||||
|
`dequeue_from_legacy`.
|
||||||
|
reclaim_age 604800 How long an un-processable expired object
|
||||||
|
marker will be retried before it is abandoned.
|
||||||
|
It is not coupled with the tombstone reclaim age
|
||||||
|
in the consistency engine.
|
||||||
|
request_tries 3 The number of times the expirer's internal client
|
||||||
|
will attempt any given request in the event
|
||||||
|
of failure
|
||||||
|
recon_cache_path /var/cache/swift Path to recon cache
|
||||||
|
nice_priority None Scheduling priority of server processes.
|
||||||
|
Niceness values range from -20 (most
|
||||||
|
favorable to the process) to 19 (least
|
||||||
|
favorable to the process). The default
|
||||||
|
does not modify priority.
|
||||||
|
ionice_class None I/O scheduling class of server processes.
|
||||||
|
I/O niceness class values are IOPRIO_CLASS_RT
|
||||||
|
(realtime), IOPRIO_CLASS_BE (best-effort),
|
||||||
|
and IOPRIO_CLASS_IDLE (idle).
|
||||||
|
The default does not modify class and
|
||||||
|
priority. Linux supports io scheduling
|
||||||
|
priorities and classes since 2.6.13 with
|
||||||
|
the CFQ io scheduler.
|
||||||
|
Work only with ionice_priority.
|
||||||
|
ionice_priority None I/O scheduling priority of server
|
||||||
|
processes. I/O niceness priority is
|
||||||
|
a number which goes from 0 to 7.
|
||||||
|
The higher the value, the lower the I/O
|
||||||
|
priority of the process. Work only with
|
||||||
|
ionice_class.
|
||||||
|
Ignored if IOPRIO_CLASS_IDLE is set.
|
||||||
|
============================= =============================== ==========================================
|
||||||
|
|
||||||
------------------------------
|
------------------------------
|
||||||
Container Server Configuration
|
Container Server Configuration
|
||||||
------------------------------
|
------------------------------
|
||||||
|
|||||||
@ -47,8 +47,109 @@ three processes with ``process`` set to 0, 1, and 2 for the three processes.
|
|||||||
If multiple processes are used, it's necessary to run one for each part of the
|
If multiple processes are used, it's necessary to run one for each part of the
|
||||||
work or that part of the work will not be done.
|
work or that part of the work will not be done.
|
||||||
|
|
||||||
The daemon uses the ``/etc/swift/object-expirer.conf`` by default, and here is
|
By default the daemon looks for two different config files. When launching,
|
||||||
a quick sample conf file::
|
the process searches for the ``[object-expirer]`` section in the
|
||||||
|
|
||||||
|
``/etc/swift/object-server.conf`` config. If the section or the config is missing
|
||||||
|
it will then look for and use the ``/etc/swift/object-expirer.conf`` config.
|
||||||
|
The latter config file is considered deprecated and is searched for to aid
|
||||||
|
in cluster upgrades.
|
||||||
|
|
||||||
|
Upgrading impact: General Task Queue vs Legacy Queue
|
||||||
|
----------------------------------------------------
|
||||||
|
|
||||||
|
The expirer daemon will be moving to a new general task-queue based design that
|
||||||
|
will divide the work across all object servers, as such only expirers defined
|
||||||
|
in the object-server config will be able to use the new system.
|
||||||
|
The parameters in both files are identical except for a new option in the
|
||||||
|
object-server ``[object-expirer]`` section, ``dequeue_from_legacy_queue``
|
||||||
|
which when set to ``True`` will tell the expirer that in addition to using
|
||||||
|
the new task queueing system to also check the legacy (soon to be deprecated)
|
||||||
|
queue.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
The new task-queue system has not been completed yet. So an expirer's with
|
||||||
|
``dequeue_from_legacy_queue`` set to ``False`` will currently do nothing.
|
||||||
|
|
||||||
|
By default ``dequeue_from_legacy_queue`` will be ``False``, it is necessary to
|
||||||
|
be set to ``True`` explicitly while migrating from the old expiring queue.
|
||||||
|
|
||||||
|
Any expirer using the old config ``/etc/swift/object-expirer.conf`` will not
|
||||||
|
use the new general task queue. It'll ignore the ``dequeue_from_legacy_queue``
|
||||||
|
and will only check the legacy queue. Meaning it'll run as a legacy expirer.
|
||||||
|
|
||||||
|
Why is this important? If you are currently running object-expirers on nodes
|
||||||
|
that are not object storage nodes, then for the time being they will still
|
||||||
|
work but only by dequeuing from the old queue.
|
||||||
|
When the new general task queue is introduced, expirers will be required to
|
||||||
|
run on the object servers so that any new objects added can be removed.
|
||||||
|
If you're in this situation, you can safely setup the new expirer
|
||||||
|
section in the ``object-server.conf`` to deal with the new queue and leave the
|
||||||
|
legacy expirers running elsewhere.
|
||||||
|
|
||||||
|
However, if your old expirers are running on the object-servers, the most
|
||||||
|
common topology, then you would add the new section to all object servers, to
|
||||||
|
deal the new queue. In order to maintain the same number of expirers checking
|
||||||
|
the legacy queue, pick the same number of nodes as you previously had and turn
|
||||||
|
on ``dequeue_from_legacy_queue`` on those nodes only. Also note on these nodes
|
||||||
|
you'd need to keep the legacy ``process`` and ``processes`` options to maintain
|
||||||
|
the concurrency level for the legacy queue.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
Be careful not to enable ``dequeue_from_legacy`` on too many expirers as
|
||||||
|
all legacy tasks are stored in a single hidden account and the same hidden
|
||||||
|
containers. On a large cluster one may inadvertently overload the
|
||||||
|
acccount/container servers handling the legacy expirer queue.
|
||||||
|
|
||||||
|
Here is a quick sample of the ``object-expirer`` section required in the
|
||||||
|
``object-server.conf``::
|
||||||
|
|
||||||
|
[object-expirer]
|
||||||
|
# log_name = object-expirer
|
||||||
|
# log_facility = LOG_LOCAL0
|
||||||
|
# log_level = INFO
|
||||||
|
# log_address = /dev/log
|
||||||
|
#
|
||||||
|
interval = 300
|
||||||
|
|
||||||
|
# If this true, expirer execute tasks in legacy expirer task queue
|
||||||
|
dequeue_from_legacy_queue = false
|
||||||
|
|
||||||
|
# processes can only be used in conjunction with `dequeue_from_legacy_queue`.
|
||||||
|
# So this option is ignored if dequeue_from_legacy_queue=false.
|
||||||
|
# processes is how many parts to divide the legacy work into, one part per
|
||||||
|
# process that will be doing the work
|
||||||
|
# processes set 0 means that a single legacy process will be doing all the work
|
||||||
|
# processes can also be specified on the command line and will override the
|
||||||
|
# config value
|
||||||
|
# processes = 0
|
||||||
|
|
||||||
|
# process can only be used in conjunction with `dequeue_from_legacy_queue`.
|
||||||
|
# So this option is ignored if dequeue_from_legacy_queue=false.
|
||||||
|
# process is which of the parts a particular legacy process will work on
|
||||||
|
# process can also be specified on the command line and will override the config
|
||||||
|
# value
|
||||||
|
# process is "zero based", if you want to use 3 processes, you should run
|
||||||
|
# processes with process set to 0, 1, and 2
|
||||||
|
# process = 0
|
||||||
|
|
||||||
|
report_interval = 300
|
||||||
|
|
||||||
|
# request_tries is the number of times the expirer's internal client will
|
||||||
|
# attempt any given request in the event of failure. The default is 3.
|
||||||
|
# request_tries = 3
|
||||||
|
|
||||||
|
# concurrency is the level of concurrency to use to do the work, this value
|
||||||
|
# must be set to at least 1
|
||||||
|
# concurrency = 1
|
||||||
|
|
||||||
|
# The expirer will re-attempt expiring if the source object is not available
|
||||||
|
# up to reclaim_age seconds before it gives up and deletes the entry in the
|
||||||
|
# queue.
|
||||||
|
# reclaim_age = 604800
|
||||||
|
|
||||||
|
And for completeness, here is a quick sample of the legacy
|
||||||
|
``object-expirer.conf`` file::
|
||||||
|
|
||||||
[DEFAULT]
|
[DEFAULT]
|
||||||
# swift_dir = /etc/swift
|
# swift_dir = /etc/swift
|
||||||
@ -76,6 +177,9 @@ a quick sample conf file::
|
|||||||
use = egg:swift#catch_errors
|
use = egg:swift#catch_errors
|
||||||
# See proxy-server.conf-sample for options
|
# See proxy-server.conf-sample for options
|
||||||
|
|
||||||
The daemon needs to run on a machine with access to all the backend servers in
|
|
||||||
the cluster, but does not need proxy server or public access. The daemon will
|
.. note::
|
||||||
use its own internal proxy code instance to access the backend servers.
|
When running legacy expirers, the daemon needs to run on a machine with
|
||||||
|
access to all the backend servers in the cluster, but does not need proxy
|
||||||
|
server or public access. The daemon will use its own internal proxy code
|
||||||
|
instance to access the backend servers.
|
||||||
|
|||||||
@ -473,6 +473,71 @@ use = egg:swift#recon
|
|||||||
# to 86400 (1 day).
|
# to 86400 (1 day).
|
||||||
# rsync_tempfile_timeout = auto
|
# rsync_tempfile_timeout = auto
|
||||||
|
|
||||||
|
# [object-expirer]
|
||||||
|
# Note: Be careful not to enable ``dequeue_from_legacy`` on too many expirers
|
||||||
|
# as all legacy tasks are stored in a single hidden account and the same hidden
|
||||||
|
# containers. On a large cluster one may inadvertently make the
|
||||||
|
# acccount/container server for the hidden ones busy.
|
||||||
|
#
|
||||||
|
# You can override the default log routing for this app here (don't use set!):
|
||||||
|
# log_name = object-expirer
|
||||||
|
# log_facility = LOG_LOCAL0
|
||||||
|
# log_level = INFO
|
||||||
|
# log_address = /dev/log
|
||||||
|
#
|
||||||
|
# interval = 300
|
||||||
|
#
|
||||||
|
# If this true, expirer execute tasks in legacy expirer task queue
|
||||||
|
# dequeue_from_legacy = false
|
||||||
|
#
|
||||||
|
# internal_client_conf_path = /etc/swift/internal-client.conf
|
||||||
|
#
|
||||||
|
# processes can only be used in conjunction with `dequeue_from_legacy`.
|
||||||
|
# So this option is ignored if dequeue_from_legacy=false.
|
||||||
|
# processes is how many parts to divide the legacy work into, one part per
|
||||||
|
# process that will be doing the work
|
||||||
|
# processes set 0 means that a single legacy process will be doing all the work
|
||||||
|
# processes can also be specified on the command line and will override the
|
||||||
|
# config value
|
||||||
|
# processes = 0
|
||||||
|
#
|
||||||
|
# process can only be used in conjunction with `dequeue_from_legacy`.
|
||||||
|
# So this option is ignored if dequeue_from_legacy=false.
|
||||||
|
# process is which of the parts a particular legacy process will work on
|
||||||
|
# process can also be specified on the command line and will override the config
|
||||||
|
# value
|
||||||
|
# process is "zero based", if you want to use 3 processes, you should run
|
||||||
|
# processes with process set to 0, 1, and 2
|
||||||
|
# process = 0
|
||||||
|
#
|
||||||
|
# report_interval = 300
|
||||||
|
#
|
||||||
|
# request_tries is the number of times the expirer's internal client will
|
||||||
|
# attempt any given request in the event of failure. The default is 3.
|
||||||
|
# request_tries = 3
|
||||||
|
#
|
||||||
|
# concurrency is the level of concurrency to use to do the work, this value
|
||||||
|
# must be set to at least 1
|
||||||
|
# concurrency = 1
|
||||||
|
#
|
||||||
|
# The expirer will re-attempt expiring if the source object is not available
|
||||||
|
# up to reclaim_age seconds before it gives up and deletes the entry in the
|
||||||
|
# queue.
|
||||||
|
# reclaim_age = 604800
|
||||||
|
#
|
||||||
|
# recon_cache_path = /var/cache/swift
|
||||||
|
#
|
||||||
|
# You can set scheduling priority of processes. Niceness values range from -20
|
||||||
|
# (most favorable to the process) to 19 (least favorable to the process).
|
||||||
|
# nice_priority =
|
||||||
|
#
|
||||||
|
# You can set I/O scheduling class and priority of processes. I/O niceness
|
||||||
|
# class values are realtime, best-effort and idle. I/O niceness
|
||||||
|
# priority is a number which goes from 0 to 7. The higher the value, the lower
|
||||||
|
# the I/O priority of the process. Work only with ionice_class.
|
||||||
|
# ionice_class =
|
||||||
|
# ionice_priority =
|
||||||
|
#
|
||||||
# Note: Put it at the beginning of the pipleline to profile all middleware. But
|
# Note: Put it at the beginning of the pipleline to profile all middleware. But
|
||||||
# it is safer to put this after healthcheck.
|
# it is safer to put this after healthcheck.
|
||||||
[filter:xprofile]
|
[filter:xprofile]
|
||||||
|
|||||||
@ -25,7 +25,7 @@ import re
|
|||||||
from swift import gettext_ as _
|
from swift import gettext_ as _
|
||||||
import tempfile
|
import tempfile
|
||||||
|
|
||||||
from swift.common.utils import search_tree, remove_file, write_file
|
from swift.common.utils import search_tree, remove_file, write_file, readconf
|
||||||
from swift.common.exceptions import InvalidPidFileException
|
from swift.common.exceptions import InvalidPidFileException
|
||||||
|
|
||||||
SWIFT_DIR = '/etc/swift'
|
SWIFT_DIR = '/etc/swift'
|
||||||
@ -48,7 +48,7 @@ GRACEFUL_SHUTDOWN_SERVERS = MAIN_SERVERS
|
|||||||
START_ONCE_SERVERS = REST_SERVERS
|
START_ONCE_SERVERS = REST_SERVERS
|
||||||
# These are servers that match a type (account-*, container-*, object-*) but
|
# These are servers that match a type (account-*, container-*, object-*) but
|
||||||
# don't use that type-server.conf file and instead use their own.
|
# don't use that type-server.conf file and instead use their own.
|
||||||
STANDALONE_SERVERS = ['object-expirer', 'container-reconciler']
|
STANDALONE_SERVERS = ['container-reconciler']
|
||||||
|
|
||||||
KILL_WAIT = 15 # seconds to wait for servers to die (by default)
|
KILL_WAIT = 15 # seconds to wait for servers to die (by default)
|
||||||
WARNING_WAIT = 3 # seconds to wait after message that may just be a warning
|
WARNING_WAIT = 3 # seconds to wait after message that may just be a warning
|
||||||
@ -475,6 +475,14 @@ class Server(object):
|
|||||||
self.server, '%s-server' % self.type, 1).replace(
|
self.server, '%s-server' % self.type, 1).replace(
|
||||||
'.pid', '.conf', 1)
|
'.pid', '.conf', 1)
|
||||||
|
|
||||||
|
def _find_conf_files(self, server_search):
|
||||||
|
if self.conf is not None:
|
||||||
|
return search_tree(SWIFT_DIR, server_search, self.conf + '.conf',
|
||||||
|
dir_ext=self.conf + '.conf.d')
|
||||||
|
else:
|
||||||
|
return search_tree(SWIFT_DIR, server_search + '*', '.conf',
|
||||||
|
dir_ext='.conf.d')
|
||||||
|
|
||||||
def conf_files(self, **kwargs):
|
def conf_files(self, **kwargs):
|
||||||
"""Get conf files for this server
|
"""Get conf files for this server
|
||||||
|
|
||||||
@ -482,17 +490,27 @@ class Server(object):
|
|||||||
|
|
||||||
:returns: list of conf files
|
:returns: list of conf files
|
||||||
"""
|
"""
|
||||||
if self.server in STANDALONE_SERVERS:
|
if self.server == 'object-expirer':
|
||||||
server_search = self.server
|
def has_expirer_section(conf_path):
|
||||||
|
try:
|
||||||
|
readconf(conf_path, section_name="object-expirer")
|
||||||
|
except ValueError:
|
||||||
|
return False
|
||||||
|
else:
|
||||||
|
return True
|
||||||
|
|
||||||
|
# config of expirer is preferentially read from object-server
|
||||||
|
# section. If all object-server.conf doesn't have object-expirer
|
||||||
|
# section, object-expirer.conf is used.
|
||||||
|
found_conf_files = [
|
||||||
|
conf for conf in self._find_conf_files("object-server")
|
||||||
|
if has_expirer_section(conf)
|
||||||
|
] or self._find_conf_files("object-expirer")
|
||||||
|
elif self.server in STANDALONE_SERVERS:
|
||||||
|
found_conf_files = self._find_conf_files(self.server)
|
||||||
else:
|
else:
|
||||||
server_search = "%s-server" % self.type
|
found_conf_files = self._find_conf_files("%s-server" % self.type)
|
||||||
if self.conf is not None:
|
|
||||||
found_conf_files = search_tree(SWIFT_DIR, server_search,
|
|
||||||
self.conf + '.conf',
|
|
||||||
dir_ext=self.conf + '.conf.d')
|
|
||||||
else:
|
|
||||||
found_conf_files = search_tree(SWIFT_DIR, server_search + '*',
|
|
||||||
'.conf', dir_ext='.conf.d')
|
|
||||||
number = kwargs.get('number')
|
number = kwargs.get('number')
|
||||||
if number:
|
if number:
|
||||||
try:
|
try:
|
||||||
@ -501,6 +519,13 @@ class Server(object):
|
|||||||
conf_files = []
|
conf_files = []
|
||||||
else:
|
else:
|
||||||
conf_files = found_conf_files
|
conf_files = found_conf_files
|
||||||
|
|
||||||
|
def dump_found_configs():
|
||||||
|
if found_conf_files:
|
||||||
|
print(_('Found configs:'))
|
||||||
|
for i, conf_file in enumerate(found_conf_files):
|
||||||
|
print(' %d) %s' % (i + 1, conf_file))
|
||||||
|
|
||||||
if not conf_files:
|
if not conf_files:
|
||||||
# maybe there's a config file(s) out there, but I couldn't find it!
|
# maybe there's a config file(s) out there, but I couldn't find it!
|
||||||
if not kwargs.get('quiet'):
|
if not kwargs.get('quiet'):
|
||||||
@ -511,10 +536,14 @@ class Server(object):
|
|||||||
else:
|
else:
|
||||||
print(_('Unable to locate config for %s') % self.server)
|
print(_('Unable to locate config for %s') % self.server)
|
||||||
if kwargs.get('verbose') and not kwargs.get('quiet'):
|
if kwargs.get('verbose') and not kwargs.get('quiet'):
|
||||||
if found_conf_files:
|
dump_found_configs()
|
||||||
print(_('Found configs:'))
|
elif any(["object-expirer" in name for name in conf_files]) and \
|
||||||
for i, conf_file in enumerate(found_conf_files):
|
not kwargs.get('quiet'):
|
||||||
print(' %d) %s' % (i + 1, conf_file))
|
print(_("WARNING: object-expirer.conf is deprecated. "
|
||||||
|
"Move object-expirers' configuration into "
|
||||||
|
"object-server.conf."))
|
||||||
|
if kwargs.get('verbose'):
|
||||||
|
dump_found_configs()
|
||||||
|
|
||||||
return conf_files
|
return conf_files
|
||||||
|
|
||||||
|
|||||||
@ -28,7 +28,7 @@ from eventlet.greenpool import GreenPool
|
|||||||
from swift.common.daemon import Daemon
|
from swift.common.daemon import Daemon
|
||||||
from swift.common.internal_client import InternalClient, UnexpectedResponse
|
from swift.common.internal_client import InternalClient, UnexpectedResponse
|
||||||
from swift.common.utils import get_logger, dump_recon_cache, split_path, \
|
from swift.common.utils import get_logger, dump_recon_cache, split_path, \
|
||||||
Timestamp
|
Timestamp, config_true_value
|
||||||
from swift.common.http import HTTP_NOT_FOUND, HTTP_CONFLICT, \
|
from swift.common.http import HTTP_NOT_FOUND, HTTP_CONFLICT, \
|
||||||
HTTP_PRECONDITION_FAILED
|
HTTP_PRECONDITION_FAILED
|
||||||
|
|
||||||
@ -50,6 +50,22 @@ class ObjectExpirer(Daemon):
|
|||||||
self.logger = logger or get_logger(conf, log_route='object-expirer')
|
self.logger = logger or get_logger(conf, log_route='object-expirer')
|
||||||
self.interval = int(conf.get('interval') or 300)
|
self.interval = int(conf.get('interval') or 300)
|
||||||
|
|
||||||
|
self.conf_path = \
|
||||||
|
self.conf.get('__file__') or '/etc/swift/object-expirer.conf'
|
||||||
|
# True, if the conf file is 'object-expirer.conf'.
|
||||||
|
is_legacy_conf = 'expirer' in self.conf_path
|
||||||
|
# object-expirer.conf supports only legacy queue
|
||||||
|
self.dequeue_from_legacy = \
|
||||||
|
True if is_legacy_conf else \
|
||||||
|
config_true_value(conf.get('dequeue_from_legacy', 'false'))
|
||||||
|
|
||||||
|
if is_legacy_conf:
|
||||||
|
self.ic_conf_path = self.conf_path
|
||||||
|
else:
|
||||||
|
self.ic_conf_path = \
|
||||||
|
self.conf.get('internal_client_conf_path') or \
|
||||||
|
'/etc/swift/internal-client.conf'
|
||||||
|
|
||||||
self.read_conf_for_queue_access(swift)
|
self.read_conf_for_queue_access(swift)
|
||||||
|
|
||||||
self.report_interval = int(conf.get('report_interval') or 300)
|
self.report_interval = int(conf.get('report_interval') or 300)
|
||||||
@ -75,8 +91,6 @@ class ObjectExpirer(Daemon):
|
|||||||
# This is for common parameter with general task queue in future
|
# This is for common parameter with general task queue in future
|
||||||
self.task_container_prefix = ''
|
self.task_container_prefix = ''
|
||||||
|
|
||||||
self.ic_conf_path = \
|
|
||||||
self.conf.get('__file__') or '/etc/swift/object-expirer.conf'
|
|
||||||
request_tries = int(self.conf.get('request_tries') or 3)
|
request_tries = int(self.conf.get('request_tries') or 3)
|
||||||
self.swift = swift or InternalClient(
|
self.swift = swift or InternalClient(
|
||||||
self.ic_conf_path, 'Swift Object Expirer', request_tries)
|
self.ic_conf_path, 'Swift Object Expirer', request_tries)
|
||||||
@ -251,6 +265,14 @@ class ObjectExpirer(Daemon):
|
|||||||
These will override the values from the config file if
|
These will override the values from the config file if
|
||||||
provided.
|
provided.
|
||||||
"""
|
"""
|
||||||
|
# This if-clause will be removed when general task queue feature is
|
||||||
|
# implemented.
|
||||||
|
if not self.dequeue_from_legacy:
|
||||||
|
self.logger.info('Until general task queue has been released '
|
||||||
|
'`dequeue_from_legacy == False` means an '
|
||||||
|
'object-expirer run is a no-op.')
|
||||||
|
return
|
||||||
|
|
||||||
self.get_process_values(kwargs)
|
self.get_process_values(kwargs)
|
||||||
pool = GreenPool(self.concurrency)
|
pool = GreenPool(self.concurrency)
|
||||||
self.report_first_time = self.report_last_time = time()
|
self.report_first_time = self.report_last_time = time()
|
||||||
|
|||||||
@ -503,6 +503,123 @@ class TestServer(unittest.TestCase):
|
|||||||
conf = self.join_swift_dir(server_name + '.conf')
|
conf = self.join_swift_dir(server_name + '.conf')
|
||||||
self.assertEqual(conf_file, conf)
|
self.assertEqual(conf_file, conf)
|
||||||
|
|
||||||
|
def _test_expirer_conf_files(self, files_and_contents, expected_files):
|
||||||
|
files, contents = zip(*files_and_contents)
|
||||||
|
with temptree(files, contents) as t:
|
||||||
|
manager.SWIFT_DIR = t
|
||||||
|
expected_files = [self.join_swift_dir(f) for f in expected_files]
|
||||||
|
|
||||||
|
def assert_results(quiet, verbose):
|
||||||
|
original_stdout = sys.stdout
|
||||||
|
try:
|
||||||
|
with open(os.path.join(t, 'output'), 'w+') as stdout:
|
||||||
|
sys.stdout = stdout
|
||||||
|
server = manager.Server('object-expirer')
|
||||||
|
conf_files = server.conf_files(verbose=verbose,
|
||||||
|
quiet=quiet)
|
||||||
|
messages = pop_stream(stdout)
|
||||||
|
finally:
|
||||||
|
sys.stdout = original_stdout
|
||||||
|
|
||||||
|
self.assertEqual(conf_files, expected_files)
|
||||||
|
|
||||||
|
if any(["expirer" in f for f in expected_files]) and not quiet:
|
||||||
|
self.assertIn(
|
||||||
|
"object-expirer.conf is deprecated.", messages)
|
||||||
|
if verbose:
|
||||||
|
for f in expected_files:
|
||||||
|
self.assertIn(f, messages)
|
||||||
|
elif not expected_files and not quiet:
|
||||||
|
self.assertIn("Unable to locate config", messages)
|
||||||
|
else:
|
||||||
|
self.assertEqual(messages, "")
|
||||||
|
|
||||||
|
assert_results(quiet=True, verbose=False)
|
||||||
|
assert_results(quiet=False, verbose=False)
|
||||||
|
assert_results(quiet=False, verbose=True)
|
||||||
|
|
||||||
|
def test_expirer_conf_files(self):
|
||||||
|
self._test_expirer_conf_files(
|
||||||
|
[('object-expirer.conf', '')], ['object-expirer.conf'])
|
||||||
|
|
||||||
|
self._test_expirer_conf_files(
|
||||||
|
[('object-server.conf', '')], [])
|
||||||
|
self._test_expirer_conf_files(
|
||||||
|
[('object-server.conf', '[object-expirer]')],
|
||||||
|
['object-server.conf'])
|
||||||
|
|
||||||
|
self._test_expirer_conf_files([
|
||||||
|
('object-server/1.conf', ''),
|
||||||
|
('object-server/2.conf', ''),
|
||||||
|
('object-server/3.conf', ''),
|
||||||
|
('object-server/4.conf', ''),
|
||||||
|
], [])
|
||||||
|
self._test_expirer_conf_files([
|
||||||
|
('object-server/1.conf', '[object-expirer]'),
|
||||||
|
('object-server/2.conf', ''),
|
||||||
|
('object-server/3.conf', ''),
|
||||||
|
('object-server/4.conf', ''),
|
||||||
|
], ['object-server/1.conf'])
|
||||||
|
self._test_expirer_conf_files([
|
||||||
|
('object-server/1.conf', '[object-expirer]'),
|
||||||
|
('object-server/2.conf', '[object-expirer]'),
|
||||||
|
('object-server/3.conf', '[object-expirer]'),
|
||||||
|
('object-server/4.conf', '[object-expirer]'),
|
||||||
|
], [
|
||||||
|
'object-server/1.conf',
|
||||||
|
'object-server/2.conf',
|
||||||
|
'object-server/3.conf',
|
||||||
|
'object-server/4.conf',
|
||||||
|
])
|
||||||
|
|
||||||
|
self._test_expirer_conf_files([
|
||||||
|
('object-server.conf', ''),
|
||||||
|
('object-expirer.conf', ''),
|
||||||
|
], ['object-expirer.conf'])
|
||||||
|
self._test_expirer_conf_files([
|
||||||
|
('object-server.conf', '[object-expirer]'),
|
||||||
|
('object-expirer.conf', ''),
|
||||||
|
], ['object-server.conf'])
|
||||||
|
|
||||||
|
self._test_expirer_conf_files([
|
||||||
|
('object-server/1.conf', ''),
|
||||||
|
('object-server/2.conf', ''),
|
||||||
|
('object-server/3.conf', ''),
|
||||||
|
('object-server/4.conf', ''),
|
||||||
|
('object-expirer.conf', ''),
|
||||||
|
], ['object-expirer.conf'])
|
||||||
|
self._test_expirer_conf_files([
|
||||||
|
('object-server/1.conf', '[object-expirer]'),
|
||||||
|
('object-server/2.conf', ''),
|
||||||
|
('object-server/3.conf', ''),
|
||||||
|
('object-server/4.conf', ''),
|
||||||
|
('object-expirer.conf', ''),
|
||||||
|
], ['object-server/1.conf'])
|
||||||
|
self._test_expirer_conf_files([
|
||||||
|
('object-server/1.conf', '[object-expirer]'),
|
||||||
|
('object-server/2.conf', '[object-expirer]'),
|
||||||
|
('object-server/3.conf', '[object-expirer]'),
|
||||||
|
('object-server/4.conf', '[object-expirer]'),
|
||||||
|
('object-expirer.conf', ''),
|
||||||
|
], [
|
||||||
|
'object-server/1.conf',
|
||||||
|
'object-server/2.conf',
|
||||||
|
'object-server/3.conf',
|
||||||
|
'object-server/4.conf',
|
||||||
|
])
|
||||||
|
|
||||||
|
self._test_expirer_conf_files([
|
||||||
|
('object-server/1.conf.d/20_setting.conf', '[object-expirer]'),
|
||||||
|
('object-server/2.conf.d/20_setting.conf', '[object-expirer]'),
|
||||||
|
('object-server/3.conf.d/20_setting.conf', '[object-expirer]'),
|
||||||
|
('object-server/4.conf.d/20_setting.conf', '[object-expirer]'),
|
||||||
|
], [
|
||||||
|
'object-server/1.conf.d',
|
||||||
|
'object-server/2.conf.d',
|
||||||
|
'object-server/3.conf.d',
|
||||||
|
'object-server/4.conf.d',
|
||||||
|
])
|
||||||
|
|
||||||
def test_proxy_conf_dir(self):
|
def test_proxy_conf_dir(self):
|
||||||
conf_files = (
|
conf_files = (
|
||||||
'proxy-server.conf.d/00.conf',
|
'proxy-server.conf.d/00.conf',
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user