diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst index 18473d8..9acc0a2 100644 --- a/doc/source/configuration.rst +++ b/doc/source/configuration.rst @@ -1,31 +1,120 @@ -============= -Configuration -============= +===================== +General configuration +===================== -There is default configuration file ``config.yaml``, which can be used by the scripts. -If you wish to keep several configuration files, that is possible - just copy it and explicitly provide the name of it once you launch a script (``--config`` option). +All default configuration values are defined in ``timmy/conf.py``. Timmy works with these values if no configuration file is provided. +If a configuration file is provided via ``-c | --config`` option, it overlays the default configuration. +An example of a configuration file is ``config.yaml``. -Here is the description of available parameters in configuration file: - -* **ssh** parameters of *SSH* - - * **opts** parameters to send to ssh command directly (recommended to leave at default) - * **vars** environment variables to set for SSH +Some of the parameters available in configuration file: +* **ssh_opts** parameters to send to ssh command directly (recommended to leave at default), such as connection timeout, etc. See ``timmy/conf.py`` to review defaults. +* **env_vars** environment variables to pass to the commands and scripts - you can use this to expand variables in commands or scripts * **fuel_ip** the IP address of the master node in the environment -* **rqdir** the path of *rqdir*, the directory containing info about commands to execute and logs to gather -* **out-dir** directory to store output data -* **timeout** timeout for SSH commands in seconds -* **archives** directory to store the generated archives -* **log_files** path and filters for log files +* **fuel_user** username to use for accessing Nailgun API +* **fuel_pass** password to access Nailgun API +* **rqdir** the path of *rqdir*, the directory containing scripts to execute and filelists to pass to rsync +* **out_dir** directory to store output data +* **archive_dir** directory to put resulting archives into +* **timeout** timeout for SSH commands and scripts in seconds -Nodes which are stored in fuel database can be filtered by the following parameters: - * roles, - * online - * status the list of statuses ex. ['ready', 'discover'] - * **node_ids** the list of ids, ex. [0,5,6] +=================== +Configuring actions +=================== -* **hard_filter** hard filter for nodes -* **soft_filter** soft filters for nodes +Actions can be configured in a separate yaml file (by default ``rq.yaml`` is used) and / or defind in the main config file or passed via command line options ``-P``, ``-C``, ``-S``, ``-G``. + +The following actions are available for definition: +* **put** - a list of tuples / 2-element lists: [source, destination]. Passed to ``scp`` like so ``scp source :destination``. Wildcards supported for source. +* **cmds** - a list of dicts: {'command-name':'command-string'}. Example: {'command-1': 'uptime'}. Command string is a bash string. Commands are executed in a sorted order of their names. +* **scripts** - a list of script filenames located on a local system. If filename does not contain path separator, the script is expected ot be located inside ``rqdir/scripts``. Otherwise the provided path is used to read the script. +* **files** - a list of filenames to collect. passed to ``scp``. Supports wildcards. +* **filelists** - a list of filelist filenames located on a local system. Filelist is a text file containing files and directories to collect, passed to rsync. Does not support wildcards. If filename does not contain path separator, the filelist is expected to be located inside ``rqdir/filelists``. Otherwise the provided path is used to read the filelist. +* **log_files** +** **path** - base path to scan for logs +** **include** - regexp string to match log files against for inclusion (if not set = include all) +** **exclude** - regexp string to match log files against. Excludes matched files from collection. +** **start** - date or datetime string to collect only files modified on or after the specified time. Format - ``YYYY-MM-DD`` or ``YYYY-MM-DD HH:MM:SS`` + +=============== +Filtering nodes +=============== + +* **soft_filter** - use to skip any operations on non-matching nodes +* **hard_filter** - same as above but also removes non-matching nodes from NodeManager.nodes dict - useful when using timmy as a module + +Nodes can be filtered by the following parameters defined inside **soft_filter** and/or **hard_filter**: + * **roles** - the list of roles, ex. **['controller','compute']** + * **online** - enabled by default to skip non-accessible nodes + * **status** - the list of statuses. Default: **['ready', 'discover']** + * **ids** - the list of ids, ex. **[0,5,6]** + * any other attribute of Node object which is a simple type (int, float, str, etc.) or a list containing simple types + +Lists match **any**, meaning that if any element of the filter list matches node value (if value is a list - any element in it), node passes. + +Negative filters are possible by pretending filter parameter with **no_**, example: **no_id = [0]** will filter out Fuel. + +Negative lists also match **any** - if any match / collision found, node is skipped. + +You can combine any number of positive and negative filters as long as their names differ (since this is a dict). + +You can use both positive and negative parameters to match the same node parameter (though it does not make much sense): +**roles = ['controller', 'compute']** +**no_roles = ['compute']** +This will skip computes and run only on controllers. Like stated above, does not make much sense :) + +============================= +Parameter-based configuration +============================= + +It is possible to define special **by_** dicts in config to (re)define node parameters based on other parameters. For example: + +:: + + by_roles: + controller: + cmds: {'check-uptime': 'uptime'} + +In this example for any controller node, cmds setting will be reset to the value above. For nodes without controller role, default (none) values will be used. + +============= +rqfile format +============= + +``rqfile`` format is a bit different from config. The basic difference: + +**config:** + +:: + + scripts: [a ,b, c] + by_roles: + compute: + scripts: [d, e, f] + +**rqfile:** + +:: + + scripts: + __default: [a, b, c] + by_roles: + compute: [d, e, f] + +The **config** and **rqfile** definitions presented above are equivalent. It is possible to define config in a config file using the **config** format, or in an **rqfile** using **rqfile** format, linking to the **rqfile** in config with ``rqfile`` setting. It is also possible to define part here and part there. Mixing identical parameters in both places is not recommended - results may be unpredictable (such scenario was not thoroughly tested). In general **rqfile** is good for fewer settings with more parameter-based variations (``by_``), and main config for more different settings with less such variations. + +=============================== +Configuration application order +=============================== + +Configuration is assembled and applied in a specific order: +1. default configuration is initialized. See ``timmy/conf.py`` for details. +2. command line parameters, if defined, are used to modify the configuration. +3. **rqfile**, if defined (default - ``rq.yaml``), is converted and injected into the configuration. At this stage the configuration is in its final form. +4. for every node, configuration is applied, except ``once_by_`` directives: +4.1 first the top-level attributes are set +4.2 then ``by_`` parameters except ``by_id`` are iterated to override or append(accumulate) the attributes +4.3 then ``by_id`` is iterated to override any matching attributes, redefining what was set before +5. finally ``once_by_``` parameters are applied - only for one matching node for any set of matching values. This is useful for example if you want a specific file or command from only a single node matching a specific role, like running ``nova list`` only on one controller. Once you are done with the configuration, you might want to familiarize yourself with :doc:`Usage `.