Add attr-overview directive

This directive creates a bullet-point list of all the attributes
defined within a file.  The idea is to give a quick overview reference
for config file options.

There are two options to start with -- maxdepth is similar to the TOC
option and only shows certain levels of options; prefix allows to
filter down to a smaller set of options if required.

I've reworked the documentation examples as part of testing this.  The
various components are moved into separate files.  On the main page,
moved the config options into the main documentation (and use the zuul
attributes :) and pointed out that you can view the source of each
sample page to see how to generate what you see.

Change-Id: I6b0f414f50428c6e04b3aeb2a2c1f9196de80ce6

README.rst

@@ -1,18 +1,6 @@
 Zuul Sphinx
 ===========
 
-A Sphinx extension for documenting Zuul jobs.
+A `Sphinx <https://www.sphinx-doc.org>`__ extension for documenting
+`Zuul <https://zuul-ci.org>`__ jobs and configuration.
 
-Config options
---------------
-
-``zuul_role_paths``
-  (str list)
-  List of extra paths to examine for role documentation (other than
-  ``roles/``)
-
-``zuul_autoroles_warn_missing``
-  (boolean)
-  Default: True
-  Warn when a role found with ``autoroles`` does not have a
-  ``README.rst`` file.

doc/source/example-attributes.rst

@@ -0,0 +1,61 @@
+Configuration Attributes
+------------------------
+
+.. attr:: example-attr
+   :required:
+
+   This is an example configuration attribute.
+
+   .. attr:: foo
+      :default: bar
+      :example: sample_value_for_example_attr
+      :type: str
+
+      A sub attribute.
+
+      .. value:: bar
+
+         An attribute value.
+
+      .. value:: baz
+
+         Another attribute value.
+
+      .. attr:: moo
+
+         An even further nested attribute
+
+         .. attr:: boo
+
+            And one more for good luck
+
+.. attr:: another-example-attr
+
+   And back to the top level
+
+References
+==========
+
+This is an attribute role: :attr:`example-attr.foo`
+
+This is an attribute value role: :value:`example-attr.foo.bar`
+
+Summaries
+=========
+
+All attributes
+^^^^^^^^^^^^^^
+
+.. attr-overview::
+
+Only one level
+^^^^^^^^^^^^^^
+
+.. attr-overview::
+   :maxdepth: 1
+
+Only example-attr.foo prefix
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. attr-overview::
+   :prefix: example-attr.foo

doc/source/example-jobs.rst

@@ -1,8 +1,5 @@
-Example Jobs
-============
-
 Jobs
-----
+====
 
 .. job:: example
 

doc/source/example-roles.rst

@@ -1,8 +1,5 @@
-Example Roles
-=============
-
 Roles
------
+=====
 
 .. role:: example
 

doc/source/example-statistics.rst

@@ -0,0 +1,13 @@
+Statistics
+----------
+
+.. stat:: example-stat
+
+   This is an example statistic.
+
+   .. stat:: foo
+      :type: counter
+
+      A sub stat.
+
+This is a statistics role: :stat:`example-stat.foo`

doc/source/example-templates.rst

@@ -1,8 +1,5 @@
-Example Project Templates
-=========================
-
 Project Templates
------------------
+=================
 
 .. project_template:: example
 

doc/source/example-variables.rst

@@ -0,0 +1,25 @@
+Variables
+---------
+
+.. var:: example-variable
+
+   This is an example variable.
+
+   .. var:: foo
+
+      This is a variable.
+
+      .. var:: bar
+
+         This is a sub key.
+
+   .. var:: items
+      :type: list
+
+      This variable is a list.
+
+      .. var:: baz
+
+         This is an item in a list.
+
+This is a variable role: :var:`example-variable.items.baz`

doc/source/examples.rst

@@ -1,71 +0,0 @@
-Examples
-========
-
-Configuration Attributes
-------------------------
-
-.. attr:: example-attr
-   :required:
-
-   This is an example configuration attribute.
-
-   .. attr:: foo
-      :default: bar
-      :example: sample_value_for_example_attr
-      :type: str
-
-      A sub attribute.
-
-      .. value:: bar
-
-         An attribute value.
-
-      .. value:: baz
-
-         Another attribute value.
-
-This is an attribute role: :attr:`example-attr.foo`
-
-This is an attribute value role: :value:`example-attr.foo.bar`
-
-
-Job Variables
--------------
-
-.. var:: example-variable
-
-   This is an example variable.
-
-   .. var:: foo
-
-      This is a variable.
-
-      .. var:: bar
-
-         This is a sub key.
-
-   .. var:: items
-      :type: list
-
-      This variable is a list.
-
-      .. var:: baz
-
-         This is an item in a list.
-
-This is a variable role: :var:`example-variable.items.baz`
-
-
-Statistics
-----------
-
-.. stat:: example-stat
-
-   This is an example statistic.
-
-   .. stat:: foo
-      :type: counter
-
-      A sub stat.
-
-This is a statistics role: :stat:`example-stat.foo`

doc/source/index.rst

@@ -1,16 +1,52 @@
 .. include:: ../../README.rst
 
+Overview
+--------
+
+This documentation has full examples of how to use the zuul-sphinx
+features.
+
+Config options
+--------------
+
+The following options can be set
+
+.. attr:: zuul_role_paths
+   :type: str list
+
+   List of extra paths to examine for role documentation (other than
+   ``roles/``)
+
+.. attr:: zuul_roles_warn_missing
+   :type: bool
+   :default: True
+
+    Warn when a role found with ``autoroles`` does not have a
+    ``README.rst`` file.
+
+
+Examples
+--------
+
+.. note::
+
+   To see the commands that produces the rendered output for this page
+   or any of the examples below, use the ``Show Source`` link at the
+   bottom of the page.
+
 .. toctree::
    :maxdepth: 2
 
-   examples
+   example-variables
+   example-attributes
    example-jobs
    example-templates
    example-roles
    example-autodoc
+   example-statistics
 
 Indices and tables
-==================
+------------------
 
 * :ref:`genindex`
 * :ref:`search`

zuul_sphinx/zuul.py

@@ -17,7 +17,9 @@ import codecs
 import os
 
 from sphinx import addnodes
+from docutils import nodes
 from docutils.parsers.rst import Directive
+from docutils.parsers.rst import directives
 from sphinx.domains import Domain, ObjType
 from sphinx.errors import SphinxError
 from sphinx.roles import XRefRole
@@ -649,7 +651,104 @@ class ZuulDomain(Domain):
                 del self.data['objects'][fullname]
 
 
+######################################################################
+#
+# Attribute overview directives
+#
+
+# TODO(ianw)
+#
+# There are many ways this could be improved
+#  * fancy indentation of nested attrs in the overview
+#  * (related) stripping of prefixes for nesting
+#  * something better than a bullet list (table?)
+#  * add something to attributes so that they can list thier child
+#    attributes atuomatically.  Something like
+#
+#    .. attr:: foo
+#       :show_overview:
+#
+#       This is the foo option
+#
+#    and then
+#
+#       .. attr-overview::
+#          :maxdepth: 1
+#          :prefix: foo
+#
+#    gets automatically inserted for you, and then you should have a
+#    sensible overview of the sub-options of "foo" inside the
+#    top-level "foo" documentation
+#  * figure out if it could be added to TOC
+
+class attroverview(nodes.General, nodes.Element):
+    pass
+
+class AttrOverviewDirective(Directive):
+    option_arguments = 2
+    option_spec = {
+        'maxdepth': directives.positive_int,
+        'prefix': directives.unchanged
+    }
+
+    def run(self):
+        attr = attroverview('')
+        if 'maxdepth' in self.options:
+            attr._maxdepth = self.options['maxdepth']
+        if 'prefix' in self.options:
+            attr._prefix = self.options['prefix']
+        return [attr]
+
+
+def process_attr_overview(app, doctree, fromdocname):
+    objects = app.builder.env.domaindata['zuul']['objects']
+
+    for node in doctree.traverse(attroverview):
+        content = []
+
+        l = nodes.bullet_list()
+        content.append(l)
+        # The "..attr" calls have built up this dictionary, of the format
+        #
+        # {
+        #   attr-foo : (docname, attr),
+        #   attr-foo.bar : (docname, attr),
+        # }
+        #
+        # So, in words, we look at all items in this list that have
+        # our docname and the attr "type" (second argument) and build
+        # them into a bullet list.
+        for k,v in objects.items():
+            if v[0] == fromdocname and v[1] == 'attr':
+                # remove the leading "attr-" for the link name ... the
+                # whole thing is is the refid however.
+                name = k[5:]
+
+                # e.g. if we have foo.bar.baz that's considered 3
+                # levels
+                if getattr(node, '_maxdepth', None):
+                    maxdepth = node._maxdepth
+                    if len(name.split('.')) > maxdepth:
+                        continue
+
+                if getattr(node, '_prefix', None):
+                    prefix = node._prefix
+                    if not name.startswith(prefix.strip()):
+                        continue
+
+                item = nodes.list_item()
+                para = nodes.paragraph()
+                refnode = nodes.reference(name, name, internal=True, refid=k)
+                para.append(refnode)
+                item.append(para)
+                l.append(item)
+
+        node.replace_self(content)
+
+
 def setup(app):
     app.add_config_value('zuul_role_paths', [], 'html')
     app.add_config_value('zuul_autoroles_warn_missing', True, '')
+    app.add_directive('attr-overview', AttrOverviewDirective)
+    app.connect('doctree-resolved', process_attr_overview)
     app.add_domain(ZuulDomain)