diff --git a/README.rst b/README.rst index cb70931..b2ab948 100644 --- a/README.rst +++ b/README.rst @@ -2,23 +2,43 @@ subunit2SQL README ================== -subunit2SQL like it's name implies is a tool used for converting subunit -streams to data in a SQL database. The motivation is that for multiple -distributed test runs that are generating subunit output it is useful to -store the results in a unified repository. This is the motivation for the -testrepository project which does a good job for centralizing the results from -multiple test runs. +subunit2SQL is a tool for storing test results data in a SQL database. Like +it's name implies it was originally designed around converting `subunit`_ +streams to data in a SQL database and the packaged utilities assume a subunit +stream as the input format. However, the data model used for the DB does not +preclude using any test result format. Additionally the analysis tooling built +on top of a database is data format agnostic. However if you choose to use a +different result format as an input for the database additional tooling using +the DB api would need to be created to parse a different test result output +format. It's also worth pointing out that subunit has several language library +bindings available. So as a user you could create a small filter to convert a +different format to subunit. Creating a filter should be fairly easy and then +you don't have to worry about writing a tool like :ref:_`subunit2sql` to use a +different format. + +.. _subunit: https://github.com/testing-cabal/subunit/blob/master/README.rst + +For multiple distributed test runs that are generating subunit output it is +useful to store the results in a unified repository. This is the motivation for +the _`testrepository` project which does a good job for centralizing the +results from multiple test runs. + +.. _testrepository: http://testrepository.readthedocs.org/en/latest/ However, imagine something like the OpenStack CI system where the same basic test suite is normally run several hundreds of times a day. To provide useful introspection on the data from those runs and to build trends over time the test results need to be stored in a format that allows for easy querying. -Using a SQL database makes a lot of sense for doing this. +Using a SQL database makes a lot of sense for doing this, which was the +original motivation for the project. -subunit2SQL uses alembic migrations to setup a DB schema that can then be used -by the subunit2sql binary to parse subunit streams and populate the DB. -Additionally, it provides a DB API that can be used to query information from -the results stored to build other tooling. +At a high level subunit2SQL uses alembic migrations to setup a DB schema that +can then be used by the :ref:`subunit2sql` tool to parse subunit streams and +populate the DB. Then there are tools for interacting with the stored data in +the :ref:`subunit2sql-graph` command as well as the :ref:`sql2subunit` +command to create a subunit stream from data in the database. Additionally, +subunit2sql provides a Python DB API that can be used to query information from +the stored data to build other tooling. Usage ===== @@ -50,8 +70,10 @@ addition, the performance of running, even in a testing capacity, subunit2sql with MySQL or Postgres make it worth the effort of setting up one of them to use subunit2sql. -Running subunit2sql -------------------- +.. _subunit2sql: + +subunit2sql +----------- Once you have a database setup with the proper database schema you can then use the subunit2sql command to populate the database with data from your test runs. @@ -76,8 +98,10 @@ path that points to any logs or other external test artifacts related to the run being added. The run_meta option takes in a dictionary which will be added to the database as key value pairs associated with the run being added. -Creating a v2 Subunit Stream from the DB -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. _sql2subunit: + +sql2subunit +----------- The sql2subunit utility is used for taking a run_id and creating a subunit v2 stream from the data in the DB about that run. To create a new subunit @@ -90,10 +114,29 @@ file or the DB connection info. Running this command will print to stdout the subunit v2 stream for the run specified by $RUN_ID, unless the --out_path argument is specified to write it to a file instead. - Release Notes ============= +0.7.0 +----- + * Initial external plugin support for the subunit2sql-graph command + * Internal code cleanup around the use of the oslo namespace package + * A temporary version cap on oslo.db limiting it to < 2.0.0 because of a + subunit2sql dependency on an internal oslo.db interface (this will be removed + in a future release) + +0.6.0 +----- + * subunit2sql-graph improvements: + * Use setuptools extras to list graph requirements + * Adds documentation + * New graph type to show daily test count over time + * Graph cleanups + * Start of attachments support + * Adds a new table to store arbitrary attachments from a subunit stream + * Support to the subunit2sql utility to store attachments in the attachments + table + 0.5.1 ----- * Remove matplotlib from requirements file to avoid requiring additional C diff --git a/TODO.rst b/TODO.rst index 7f7a117..1e88435 100644 --- a/TODO.rst +++ b/TODO.rst @@ -3,6 +3,9 @@ Work Items for Subunit2SQL Short Term ---------- + * Add attachments support to the sql2subunit utility + * Enable sqlite support for the data migrations to enable using sqlite as + a database backend * Add more unit tests * DB API unit tests * write_subunit module @@ -16,6 +19,5 @@ Short Term Longer Term ----------- - * Add tooling to pull the data and visualize it in fun ways * Add some statistics functions on top of the DB api to perform analysis * Add a subunit2sql repository type to the testrepository project diff --git a/doc/source/conf.py b/doc/source/conf.py index 1b6de2c..e8b4308 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -199,8 +199,10 @@ latex_documents = [ # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ - ('index', 'subunit2sql', u'subunit2SQL Documentation', - [u'Matthew Treinish'], 1) + ('README', 'subunit2sql', u'Tooling for storing and interacting with test' + ' results databases', [u'Matthew Treinish'], 1), + ('graph', 'subunit2sql-graph', 'A tool for generating graphs from a ' + 'subunit2sql database', ['Matthew Treinish'], 1) ] # If true, show URL addresses after external links. diff --git a/doc/source/graph-dailycount.png b/doc/source/graph-dailycount.png new file mode 100644 index 0000000..6aeb983 Binary files /dev/null and b/doc/source/graph-dailycount.png differ diff --git a/doc/source/graph.rst b/doc/source/graph.rst index 7842aa2..7903d39 100644 --- a/doc/source/graph.rst +++ b/doc/source/graph.rst @@ -1,4 +1,6 @@ -Subunit2SQL Graphs +.. _subunit2sql-graph: + +subunit2sql-graph ================== subunit2sql includes a utility to generate various graphs from the data in a database. This is used to provide a more visual analysis of the data contained @@ -118,6 +120,18 @@ option list with something like:: subunit2sql-graph failures --help +Daily Test Counts +----------------- +This graph is used to show the the daily # of tests run as a time series line graph. + +For example, running something like:: + + subunit2sql-graph --database-connection mysql://test:test@localhost/subunit2sql --output test.png --title 'Daily Test Count' dailycount + +will generate a graph like: + +.. image:: graph-dailycount.png + subunit2sql-graph plugin interface ==================================