x/timmy
Code Issues Proposed changes

Implement negative matching in rqfiles, doc updates, remove "priority" for by_id keys

Browse Source
This commit is contained in:
f3flight 2016-08-03 12:57:13 +00:00
parent 23f3756a09
commit 4705a23874
3 changed files with 53 additions and 30 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

31
doc/source/configuration.rst
View File

@ -13,8 +13,20 @@ Some of the parameters available in configuration file:
* **fuel_ip** the IP address of the master node in the environment
* **fuel_user** username to use for accessing Nailgun API
* **fuel_pass** password to access Nailgun API
* **rqdir** the path of *rqdir*, the directory containing scripts to execute and filelists to pass to rsync
* **out_dir** directory to store output data
* **fuel_tenant** Fuel Keystone tenant to use when accessing Nailgun API
* **fuel_port** port to use when connecting to Fuel Nailgun API
* **fuel_keystone_port** port to use when getting a Keystone token to access Nailgun API
* **fuelclient** True/False - whether to use fuelclient library to access Nailgun API
* **fuel_skip_proxy** True/False - ignore ``http(s)_proxy`` environment variables when connecting to Nailgun API
* **rqdir** the path to the directory containing rqfiles, scripts to execute, and filelists to pass to rsync
* **rqfile** path(s) to rqfile(s) containing actions and/or other configuration parameters. Use list if more than one file is specifed.
* **logs_days** how many past days of logs to collect. This option will set **start** parameter for each **logs** action if not defined in it.
* **logs_speed_limit** True/False - enable speed limiting of log transfers (total transfer speed limit, not per-node)
* **logs_speed_default** Mbit/s - used when autodetect fails
* **logs_speed** Mbit/s - manually specify max bandwidth
* **do_print_results** print outputs of commands and scripts to stdout
* **clean** True/False - erase previous results in outdir and archive_dir dir, if any
* **outdir** directory to store output data
* **archive_dir** directory to put resulting archives into
* **timeout** timeout for SSH commands and scripts in seconds
@ -35,7 +47,7 @@ The following actions are available for definition:
* **INFO**: Scripts are not copied to the destination system - script code is passed as stdin to `bash -s` executed via ssh or locally. Therefore passing parameters to scripts is not supported (unlike cmds where you can write any Bash string). You can use variables in your scripts instead. Scripts are executed in the following order: all scripts without variables, sorted by their full filename, then all scripts with variables, also sorted by full filename. Therefore if the order matters, it's better to put all scripts into the same folder and name them according to the order in which you want them executed on the same node. Mind that scripts with variables are executed after all scripts without variables. If you need to mix scripts with variables and without and maintain order, just use dict structure for all scripts, and set `null` as the value for those which do not need variables.
* **files** - a list of filenames to collect. passed to ``scp``. Supports wildcards.
* **filelists** - a list of filelist filenames located on a local system. Filelist is a text file containing files and directories to collect, passed to rsync. Does not support wildcards. If the filename does not contain path separator, the filelist is expected to be located inside ``rqdir/filelists``. Otherwise the provided path is used to read the filelist.
* **log_files**
* **logs**
* **path** - base path to scan for logs
* **include** - regexp string to match log files against for inclusion (if not set = include all)
* **exclude** - regexp string to match log files against. Excludes matched files from collection.
@ -82,6 +94,16 @@ It is possible to define special **by_<parameter-name>** dicts in config to (re)
In this example for any controller node, cmds setting will be reset to the value above. For nodes without controller role, default (none) values will be used.
Negative matches are possible via **not_** prefix:
::
by_roles:
not_fuel:
cmds: {'check-uptime': 'uptime'}
In this example **uptime** command will be executed on all nodes except Fuel server.
It is also possible to define a special **once_by_<parameter-name>** which works similarly, but will only result in attributes being assigned to a single (first in the list) matching node. Example:
::
@ -129,8 +151,7 @@ Configuration is assembled and applied in a specific order:
3. **rqfile**, if defined (default - ``rq.yaml``), is converted and injected into the configuration. At this stage the configuration is in its final form.
4. for every node, configuration is applied, except ``once_by_`` directives:
1. first the top-level attributes are set
2. then ``by_<attribute-name>`` parameters except ``by_id`` are iterated to override or append(accumulate) the attributes
3. then ``by_id`` is iterated to override any matching attributes, redefining what was set before
2. then ``by_<attribute-name>`` parameters are iterated to override settings and append(accumulate) actions
5. finally ``once_by_`<attribute-name>`` parameters are applied - only for one matching node for any set of matching values. This is useful, for example, if you want a specific file or command from only a single node matching a specific role, like running ``nova list`` only on one controller.
Once you are done with the configuration, you might want to familiarize yourself with :doc:`Usage </usage>`.

2
timmy/env.py
View File

@ -16,7 +16,7 @@
# under the License.
project_name = 'timmy'
version = '1.13.0'
version = '1.14.0'
if __name__ == '__main__':
exit(0)

50
timmy/nodes.py
View File

@ -65,7 +65,6 @@ class Node(object):
conf_once_prefix = 'once_'
conf_match_prefix = 'by_'
conf_default_key = '__default'
conf_priority_section = conf_match_prefix + 'id'
header = ['node-id', 'env', 'ip', 'mac', 'os',
'roles', 'online', 'status', 'name', 'fqdn']
@ -127,39 +126,38 @@ class Node(object):
else:
setattr(self, k, deepcopy(v))
def r_apply(el, p, p_s, c_a, k_d, o, d, clean=False):
def r_apply(el, p, c_a, k_d, o, d, clean=False):
# apply normal attributes
for k in [k for k in el if k != p_s and not k.startswith(p)]:
for k in [k for k in el if not k.startswith(p)]:
if el == conf and clean:
apply(k, el[k], c_a, k_d, o, default=True)
else:
apply(k, el[k], c_a, k_d, o)
# apply match attributes (by_xxx except by_id)
for k in [k for k in el if k != p_s and k.startswith(p)]:
# apply match attributes
for k in [k for k in el if k.startswith(p)]:
attr_name = k[len(p):]
if hasattr(self, attr_name):
attr = w_list(getattr(self, attr_name))
matching_keys = []
# negative matching ("not_")
for nk in [nk for nk in el[k] if nk.startswith('not_')]:
key = nk[4:]
if key not in attr:
matching_keys.append(nk)
# positive matching
for v in attr:
if v in el[k]:
subconf = el[k][v]
if d in el:
d_conf = el[d]
for a in d_conf:
apply(a, d_conf[a], c_a, k_d, o)
r_apply(subconf, p, p_s, c_a, k_d, o, d)
# apply priority attributes (by_id)
if p_s in el:
if self.id in el[p_s]:
p_conf = el[p_s][self.id]
if d in el[p_s]:
d_conf = el[p_s][d]
for k in d_conf:
apply(k, d_conf[k], c_a, k_d, o)
for k in [k for k in p_conf if k != d]:
apply(k, p_conf[k], c_a, k_d, o, default=True)
matching_keys.append(v)
# apply matching keys
for mk in matching_keys:
subconf = el[k][mk]
if d in el:
d_conf = el[d]
for a in d_conf:
apply(a, d_conf[a], c_a, k_d, o)
r_apply(subconf, p, c_a, k_d, o, d)
p = Node.conf_match_prefix
p_s = Node.conf_priority_section
c_a = Node.conf_appendable
k_d = Node.conf_keep_default
d = Node.conf_default_key
@ -169,7 +167,7 @@ class Node(object):
duplication if this function gets called more than once'''
for f in set(c_a).intersection(k_d):
setattr(self, f, [])
r_apply(conf, p, p_s, c_a, k_d, overridden, d, clean=clean)
r_apply(conf, p, c_a, k_d, overridden, d, clean=clean)
def get_release(self):
if self.id == 0:
@ -566,7 +564,11 @@ class NodeManager(object):
dst[k][attr] = el[k]
def merge_rq(rqfile, dst):
src = tools.load_yaml_file(rqfile)
if os.path.sep in rqfile:
src = tools.load_yaml_file(rqfile)
else:
f = os.path.join(self.rqdir, rqfile)
src = tools.load_yaml_file(f)
p = Node.conf_match_prefix
once_p = Node.conf_once_prefix + p
d = Node.conf_default_key