User facing error Summary Messages
The purpose of this spec is to define a mechanism in Cinder that will allow a user to see error messages for asynchronous operations. This is more specific to failed operations and cause of their failure. Spec started in Newton APIImpact New /messages resource DocImpact Co-Authored-By: Alex Meade <mr.alex.meade@gmail.com> Implements : blueprint summarymessage Change-Id: If7f8b7f277863a6dbf9686dadab145590ec9567d
This commit is contained in:
Sheel Rana
parent
8afa9c447b
commit
48ddf3ff2a
432
specs/newton/summarymessage.rst
Normal file
432
specs/newton/summarymessage.rst
Normal file
@ -0,0 +1,432 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==============================================
|
||||
User facing "Failure" Message and Event Viewer
|
||||
==============================================
|
||||
|
||||
For quite some time, OpenStack services have wanted to be able to send
|
||||
messages, especially error messages, to API end users (by user I do not mean
|
||||
the operator, but the user that is interacting with the API).
|
||||
|
||||
If user performs some operation and operation fails or goes in
|
||||
hang state, then there must be some interface for user to see reason
|
||||
for this behavior.
|
||||
|
||||
So, this is basically regarding facilitating user to see error messages
|
||||
for asynchronous operations either directly through APIs or through new
|
||||
"Event Viewer" tab in horizon or horizon pluggin.
|
||||
|
||||
This is more specific to failed operations and cause of their failure.
|
||||
|
||||
https://blueprints.launchpad.net/cinder/+spec/summarymessage
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
If operations like create volume, create snapshot etc fails, user gets
|
||||
no detailed information in case operation fails.
|
||||
Sometime only operation status is updated to error, failed etc without
|
||||
any update to user.
|
||||
|
||||
In some case it is worse than this.
|
||||
For ex,
|
||||
1. In case rabbitmq is inactive and user tries to create volume,
|
||||
horizon hangs on waiting response from API forever.
|
||||
2. In case rabbitmq is active but recipient service is not running
|
||||
and user tries to create volume, API hangs on waiting response
|
||||
from rabbitMQ forever.
|
||||
|
||||
A few resources among all the OpenStack projects handle reporting errors
|
||||
to end users for asynchronous operations and those that do are inconsistent
|
||||
with each other.
|
||||
In addition to a mechanism to enable error reporting in a
|
||||
consistent way across OpenStack, the solution must also be able to
|
||||
accommodate a deployment of Cinder that contains no other OpenStack service.
|
||||
|
||||
Use Cases
|
||||
=========
|
||||
|
||||
Motivation for this Blue Print:
|
||||
|
||||
1. To help admin in debugging failed operations with
|
||||
less log filtering.
|
||||
2. To notify admin to start services which failed due to
|
||||
some abnormal conditions.
|
||||
3. To inform user about failure in case operation fails.
|
||||
4. To provide enough information to user about failure.
|
||||
|
||||
General Use Cases:
|
||||
|
||||
* Cinder volume/snapshot creation goes to ERROR status due
|
||||
to lack of capacity. (Scheduling error)
|
||||
* Cinder add volume to CG fails, how to tell user the volume
|
||||
and group are not on the same backend?
|
||||
* Cinder volume goes from attaching to available. why?
|
||||
* Volume retype fails.
|
||||
* Volume extend fails, I'd like to know why and be able to
|
||||
still use my volume instead of it being in error_extending.
|
||||
* ETC..
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
User can fetch operation failure details through direct API calls or
|
||||
through new horizon tab "Event Viewer".
|
||||
From CLI, cli client could be used to display same kind of information.
|
||||
|
||||
Suggested implementation is based on 2 way approach
|
||||
|
||||
1. Push Information:
|
||||
During user operations, messages will be pushed to database
|
||||
using component specific notifications.
|
||||
|
||||
These messages will be generated and pushed to database by
|
||||
component for operation start, operation completion or operation
|
||||
failure.
|
||||
|
||||
Message generation will be based on eventID to eventMessage
|
||||
mapping using message constant files which keeps different
|
||||
notification messages mapping in it.
|
||||
This way deployer can easily modify notification messages as per
|
||||
requirement.
|
||||
|
||||
2. Pull Information:
|
||||
In case user needs to check operation status, user can pull details
|
||||
using CLI client or Horizon tab.
|
||||
|
||||
Results may be shown in tabular way as shown below
|
||||
|
||||
+--------+---------+----------+-------+-------+----------+------+---------+
|
||||
| Tenant | EventID | NodeName | ReqID | Level | Resource | Time | Summary |
|
||||
+========+=========+==========+=======+=======+==========+======+=========+
|
||||
| Sheel | UKN_ERR | BS-cind1 | {...} | Error | Volume | {..} | {....} |
|
||||
+--------+---------+----------+-------+-------+----------+------+---------+
|
||||
|
||||
Every 'operation' initiated by the user has a request ID returned as
|
||||
an HTTP header in the context.
|
||||
These notification messages will be tied to operation request ID.
|
||||
(This request ID will be used for mapping a request to what happened in cinder
|
||||
for that operation.)
|
||||
|
||||
Summary message will contain operation specific failure message.
|
||||
For ex,
|
||||
"Volume create operation failed - {Reason of failure}"
|
||||
|
||||
|
||||
Filters:
|
||||
|
||||
Results can be filtered depending upon TenantID, HostID, UserID,
|
||||
Operation Outcome/Result(Fail/Pass) etc.
|
||||
|
||||
Type of Messages:
|
||||
|
||||
1. API Events : messages for failed operations.
|
||||
2. Service Logs: information for failed services either stopped by user
|
||||
or stopped due to any abnormal conditions.
|
||||
3. System Logs: any other logs than API and service logs.
|
||||
|
||||
**Suggested Architecture**:
|
||||
|
||||
The proposed change is to add a new /v3/<tenant>/messages API
|
||||
resource backed by a messages table in the Cinder DB.
|
||||
This endpoint will return a list of error messages that are
|
||||
intended for the end-user regarding failed asynchronous operations.
|
||||
|
||||
In short:
|
||||
* /v3/<tenant>/messages API resource, exposes notifications messages
|
||||
depending upon filters
|
||||
* message_ttl config option that dictates message minimum life in seconds
|
||||
* messages DB table
|
||||
|
||||
Questions
|
||||
---------
|
||||
None
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
* User facing notifications
|
||||
Use the existing notification framework in combination with an AMQP
|
||||
consumer to pull messages off and provide an endpoint for the user.
|
||||
Faults with this approach are that we do not want to display the current
|
||||
information in notifications to the user and it will require many more
|
||||
services as dependencies.
|
||||
|
||||
* Per resource faults
|
||||
This alternative suggests adding a sub-resource to each resource, such as
|
||||
volumes/<volume-id>/faults, similar to Nova's instance faults. This
|
||||
makes it difficult to poll for messages for more than a single resource
|
||||
or resource type. It also adds significant complexity to the api as
|
||||
every resource must add /faults in order to support messages.
|
||||
|
||||
* Exposing user messages via a separate service (such as Zaqar)
|
||||
This approach suggests storing user messages in another service that the
|
||||
user could query for messages or the service could utilize webhooks to
|
||||
notify the user. One major drawback to this approach is the complexity
|
||||
in writing bindings for the separate service(s) and the need for a
|
||||
separate service as a dependency.
|
||||
|
||||
What this specification does not solve
|
||||
--------------------------------------
|
||||
|
||||
* State change notifications.
|
||||
This solution does not intend to solve the use-case of alerting users when
|
||||
a volume or any other resource changes state. For example, when a volume
|
||||
changes from ``creating`` to ``available``.
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
New APIs:
|
||||
* GET /v3/<tenant>/messages
|
||||
With filters by attribute. Ex: GET /v3/<tenant>/messages?resource_type=volume
|
||||
* GET /v3/<tenant>/messages/<message-id>
|
||||
* DELETE /v3/<tenant>/messages/<message-id>
|
||||
|
||||
Message schema ::
|
||||
|
||||
Message:
|
||||
type: object
|
||||
required:
|
||||
- user_message
|
||||
- id
|
||||
- project_id
|
||||
- request_id
|
||||
- event_id
|
||||
- created_at
|
||||
- message_level
|
||||
- expires_at
|
||||
properties:
|
||||
id:
|
||||
type: string
|
||||
description: UUID will be stored in 'id' field.
|
||||
message_level:
|
||||
type: string
|
||||
enum:
|
||||
- ERROR
|
||||
description: The level of the message. In the future we may expand to
|
||||
sending information to the user that is not an error.
|
||||
user_message:
|
||||
type: string
|
||||
event_id:
|
||||
type: string
|
||||
description: Event ID can be used to
|
||||
a. update message text at deployer end for some specific situation.
|
||||
b. to report errors by user.
|
||||
c. to debug fast as it is easy to search where specific eventID is
|
||||
used for reporting error.
|
||||
resource_uuid:
|
||||
type: string
|
||||
description: The uuid of the offending resource.
|
||||
resource_type:
|
||||
type: string
|
||||
description: The type of resource this message pertains to.
|
||||
For ex, volume, snapshot, backup etc
|
||||
request_id:
|
||||
type: string
|
||||
created_at:
|
||||
type: string
|
||||
expires_at:
|
||||
type: string
|
||||
description: After this time the message may no longer exist
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
New messages table in the DB to store all messages. This table may prove to
|
||||
grow large in a cloud with lots of errors. The admin will be able to utilize
|
||||
the ``expires_at`` column to reap messages.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
Messages must be highly scrutinized before becoming visible to the user in
|
||||
order to avoid any sensitive data from being shown. This will be mitigated by
|
||||
having all user visible messages defined in a single module. The messaging
|
||||
mechanism will assert that any message it will create comes from the sanctioned
|
||||
location.
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
None
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
* New configuration option ``message_ttl`` that will dictate the number of
|
||||
seconds after the messages creation time to set the 'guaranteed_until'
|
||||
attribute on generated messages.
|
||||
|
||||
* New configuration option ``message_reap_interval`` that will dictate the
|
||||
number of seconds between calls to delete old messages. A value of -1
|
||||
will never run. DocImpact: This option should not be set on a large number
|
||||
of nodes, since too many nodes trying this delete at the same time will cause
|
||||
transaction bouncing and degraded DB performance.
|
||||
|
||||
* New configuration option ``message_reap_batch_size`` that dictates the number
|
||||
of expired messages to delete each interval. This allows a deployer to
|
||||
limit DB performance impact by setting a ceiling for the number of
|
||||
messages deleted at a time.
|
||||
|
||||
* The messages table will be potentially large and may be reaped based on
|
||||
the 'guaranteed_until' column. Where all messages with a
|
||||
``expires_at`` date earlier than the current time can be safely
|
||||
deleted.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
Developers should be aware of use-cases where the user needs information
|
||||
about an error. In these situations, an appropriate user message should be
|
||||
written and creation of the message added in the specific code path(s).
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
Sheel Rana
|
||||
Alex Meade
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
This whole implementation depends upon message generation, transport,
|
||||
collection, storage and analytics of different failure messages.
|
||||
|
||||
* cinder:
|
||||
Implementation to generate notification messages at the time of
|
||||
failure for all existing operations.
|
||||
|
||||
* cinder:
|
||||
notification listener is required which will serve as
|
||||
basis for handling event messages from different components.
|
||||
|
||||
* cinder:
|
||||
collector is required to collect, validate and store event
|
||||
messages to database.
|
||||
|
||||
* cinder:
|
||||
new API to fetch details form database depending upon filters.
|
||||
|
||||
* cinder:
|
||||
Add pagination to messages
|
||||
|
||||
* cinder-manage:
|
||||
Add mechanism to automatically, and via a cinder-manage command,
|
||||
reap expired messages in the db depending upon ttl value.
|
||||
|
||||
* cinder:
|
||||
Documentation for new API details.
|
||||
|
||||
* cinder:
|
||||
Update "Getting started Guide".
|
||||
|
||||
* cinder:
|
||||
Database schema preparation to store notification messages.
|
||||
|
||||
* cinder:
|
||||
Need to implement "delete messages as per message life" from database after
|
||||
message expiry time.
|
||||
For ex, if user has set ``message_ttl`` to 7 days, then all messages
|
||||
older than 7 days will be purged from database.
|
||||
|
||||
* horizon:
|
||||
Separate tab for cinder to display event messages.
|
||||
|
||||
* cinder-client:
|
||||
cinder cli to communicate with API and fetch event messages.
|
||||
|
||||
* cinder-client:
|
||||
Update to CLI reference Guide.
|
||||
|
||||
* Tempest tests
|
||||
|
||||
|
||||
Implementation Phases:
|
||||
----------------------
|
||||
This whole feature will be implemented in multiple phases:
|
||||
|
||||
Phase 1. Basic implementation regarding notification generation and storage
|
||||
into database with "/messages" exposed to view notification messages.
|
||||
This spec targets Phase 1 first, other phases will be implemented after
|
||||
acceptance of phase 1.
|
||||
|
||||
Phase 2. Implementation for facilitating admin to configure notification
|
||||
storage like db or zaqar or both.
|
||||
If both RPC/DB are configured by admin, notification message would be
|
||||
stored in zaqar along with storing information to database.
|
||||
|
||||
Phase 3. Implementation for consuming information from zaqar directly.
|
||||
|
||||
Phase 4. Horizon and CLI implementations to view notifications in more
|
||||
formatted manner.
|
||||
|
||||
Phase 5. Handling of some special cases where generation of notifications
|
||||
requires seperate handling like rabbitMQ related implementations for showing
|
||||
notifications in case rabbitMQ is in failed state or rabbitMQ recipient is
|
||||
in inactive state.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
None
|
||||
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Tempest tests should be written and run in the gate. It may prove difficult
|
||||
to implement complete functional testing of the feature as messages will not
|
||||
be created unless there is an error, which may be difficult to trigger.
|
||||
However, some operations are easy to trigger failure with unlimited quotas.
|
||||
One example is creating a thick provisioned volume too big to be stored on the
|
||||
backend.
|
||||
|
||||
Example Test Cases
|
||||
------------------
|
||||
|
||||
# List messages with no messages
|
||||
# Attempt creation of a TOO LARGE volume and verify appropriate scheduling
|
||||
error message is created
|
||||
# List messages with filters, especially resource_type
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
* REST API documentation
|
||||
* New config option, ``message_ttl`` (time to live)
|
||||
* New config option, ``message_reap_interval``
|
||||
(number of seconds between calls to delete old messages)
|
||||
* New config option, ``message_reap_batch_size``
|
||||
(number of messages which could be deleted in one batch)
|
||||
* New API policies for messages
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
Mitaka Midcycle discussion
|
||||
https://etherpad.openstack.org/p/mitaka-cinder-midcycle-user-notifications
|
||||
https://etherpad.openstack.org/p/mitaka-cinder-midcycle-day-1
|
||||
|
||||
Kilo Summit Discussion
|
||||
https://etherpad.openstack.org/p/kilo-cinder-async-reporting
|
||||
|
||||
Liberty Summit Discussion (in conjunction with HEAT) -
|
||||
https://etherpad.openstack.org/p/liberty-cross-project-user-notifications
|
||||
Loading…
Reference in New Issue
Block a user