af9aa3678a
launchpad integration is nice, but it's incredibly slow. So there are times where it would be nice if it was optional. You can specify it on the command line with the -l flag. Change-Id: I7d5d57750c82efbc0936186625ceacecaf350085
elastic-recheck
"Classify tempest-devstack failures using ElasticSearch"
- Free software: Apache license
- Documentation: http://docs.openstack.org/developer/elastic-recheck
Idea
When a tempest job failure is detected, by monitoring gerrit (using gerritlib), a collection of logstash queries will be run on the failed job to detect what the bug was.
Eventually this can be tied into the rechecker tool and launchpad
queries/
All queries are stored in separate yaml files in a queries directory
at the top of the elastic_recheck code base. The format of these files
is ######.yaml (where ###### is the bug number), the yaml should have a
query keyword which is the query text for elastic
search.
Guidelines for good queries
- After a bug is resolved and has no more hits in elasticsearch, we should flag it with a resolved_at keyword. This will let us keep some memory of past bugs, and see if they come back. (Note: this is a forward looking statement, sorting out resolved_at will come in the future)
- Queries should get as close as possible to fingerprinting the root cause
- Queries should not return any hits for successful jobs, this is a sign the query isn't specific enough
In order to support rapidly added queries, it's considered socially acceptable to +A changes that only add 1 new bug query, and to even self approve those changes by core reviewers.
Future Work
- Move config files into a separate directory
- Make unit tests robust
- Add debug mode flag
- Expand gating testing
- Cleanup and document code better
- Sort out resolved_at stamping to remove active bugs
- Move away from polling ElasticSearch to discover if its ready or not
- Add nightly job to propose a patch to remove bug queries that return no hits -- Bug hasn't been seen in 2 weeks and must be closed
- implement resolved_at in loader
Main Dependencies
- gerritlib
- pyelasticsearch