diff --git a/doc/source/authentication_tokens.rst b/doc/source/authentication_tokens.rst new file mode 100644 index 000000000..3f709949b --- /dev/null +++ b/doc/source/authentication_tokens.rst @@ -0,0 +1,37 @@ +Generate an Authentication Token +================================ + +You can use `cURL `__ to try the authentication +process in two steps: get a token, and send the token to a service. + +1. Get an authentication token by providing your user name and either + your API key or your password. Here are examples of both approaches: + + You can request a token by providing your user name and your + password. + + :: + + $ curl -X POST https://localhost:5000/v2.0/tokens -d '{"auth":{"passwordCredentials":{"username": "joecool", "password":"coolword"}, "tenantId":"5"}}' -H 'Content-type: application/json' + + Successful authentication returns a token which you can use as + evidence that your identity has already been authenticated. To use + the token, pass it to other services as an ``X-Auth-Token`` header. + + Authentication also returns a service catalog, listing the endpoints + you can use for Cloud services. + +2. Use the authentication token to send a ``GET`` to a service you would + like to use. + +Authentication tokens are typically valid for 24 hours. Applications +should be designed to re-authenticate after receiving a 401 +(Unauthorized) response from a service endpoint. + + **Note** + + If you programmatically parse an authentication response, be aware + that service names are stable for the life of the particular service + and can be used as keys. You should also be aware that a user's + service catalog can include multiple uniquely-named services that + perform similar functions. diff --git a/doc/source/getting_started.rst b/doc/source/getting_started.rst new file mode 100644 index 000000000..ec72287f2 --- /dev/null +++ b/doc/source/getting_started.rst @@ -0,0 +1,387 @@ +===================== +Getting Started Guide +===================== + +Overview +-------- + +Messaging service is a RESTful API-based messaging +service. It supports distributed web applications,and is based on the +OpenStack Zaqar project. + +Messaging service is a vital component of large, distributed +web applications. You can use Messaging service for public, +private, and hybrid cloud environments. + +As you develop distributed web applications, you often have multiple +agents set up to complete sets of tasks for those applications. These +tasks can be anything from creating users to deleting blocks of storage. +Messaging service provides a simple interface that creates these tasks as +queues, messages, and claims. The interface then posts, claims, reads, +and deletes them as the tasks are needed and performed. + +Messaging service handles the distribution of tasks, but it does not +necessarily manage the order of the tasks. Applications handle the +workflow at a higher level. + +This guide explains how to access and start using the API so that you +can begin to use Messaging service for your applications. Instructions are +given for how to properly enter the necessary URLs, using cURL, to set +up and use a basic set of Messaging service operations. + +Prerequisites for Running Examples +---------------------------------- + +In order to run the examples in this guide, you must have the following +prerequisites: + +- A Cloud account + +- A username and password, as specified during registration + +- Prior knowledge of HTTP/1.1 conventions + +- Basic familiarity with Cloud and RESTful APIs + +How Messaging service Works +------------------------- + +Following is an overview of how Messaging service works. For definitions +of Messaging service terms, see the below glossary. + +1. You create a queue to which producers or publishers post messages. + +2. Workers (consumers or subscribers) claim or get a message from the + queue, complete the work in that message, and delete the message. + + If a worker will be off-line before it completes the work in a + message, the worker can retire the claim's time to live (TTL), + putting the message back into the queue for another worker to claim. + +3. Subscribers monitor the claims from these queues to track activity + and help troubleshoot errors. + +For the majority of use cases, Messaging service is not responsible for +the ordering of messages. However, if there is only a single producer, +Messaging service ensures that messages are handled in a First In, First +Out (FIFO) order. + +Messaging Patterns +------------------ + +The Messaging service API supports a variety of messaging patterns +including the following: + +- Task distribution + +- Event broadcasting + +- Point-to-point messaging + +Task distribution +----------------- + +The task distribution pattern has the following characteristics: + +- A producer is programmed to send messages to a queue. + +- Multiple workers (or consumers) are programmed to monitor a queue. + +- Only one worker can claim a message so that no other worker can claim + the message and duplicate the work. + +- The worker must delete the message when work is done. + +- TTL restores a message to an unclaimed state if the worker never + finishes. + +This pattern is ideal for dispatching jobs to multiple processors. + +Event Broadcasting +------------------ + +Characteristics of the event broadcasting pattern are: + +- The publisher sends messages to a queue. + +- Multiple observers (or subscribers) get the messages in the queue. + +- Multiple observers take action on each message. + +- Observers send a marker to skip messages already seen. + +- TTL eventually deletes messages. + +This pattern is ideal for notification of events to multiple observers +at once. + +Point-to-point messaging +------------------------ + +Characteristics of the point-to-point messaging pattern are: + +- The publisher sends messages to a queue. + +- The consumer gets the messages in the queue. + +- The consumer can reply with the result of processing a message by + sending another message to the same queue (queues are duplex by + default). + +- The publisher gets replies from the queue. + +- The consumer sends a marker to skip messages already seen. + +- TTL eventually deletes messages. + +This pattern is ideal for communicating with a specific client, +especially when a reply is desired from that client. + +Messaging service Operations +-------------------------- + +This section lists all of the operations that are available in the +Messaging service API. This document uses some of the most common +operations in `OpenStack API Reference `__.. + +For details about all of the operations, see the Messaging service API v2 +Reference. + +Home Document +~~~~~~~~~~~~~ + +The following operation is available for the home document: + +- Get Home Document + +Queues +~~~~~~ + +The following operations are available for queues: + +- Create Queue + +- List Queues + +- Get Queue + +- Update Queue + +- Get Queue Stats + +- Delete Queue + +Messages +~~~~~~~~ + +The following operations are available for messages: + +- Post Message + +- Get Messages + +- Get a Specific Message + +- Get a Set of Messages by ID + +- Delete Message + +- Delete a Set of Messages by ID + +Claims +~~~~~~ + +The following operations are available for claims: + +- Claim Messages + +- Get Claim + +- Update Claim + +- Release Claim + +Subscriptions +~~~~~~~~~~~~~ + +The following operations are available for subscriptions: + +- Create Subscriptions + +- List Subscriptions + +- Get Subscription + +- Update Subscription + +- Delete Subscription + + +Pools +~~~~~ + +The following operations are available for Pools: + +- Create Pools + +- List Pools + +- Get Pool + +- Update Pool + +- Delete Pool + +Flavors +~~~~~~~ + +The following operations are available for Flavors: + +- Create Flavors + +- List Flavors + +- Get Flavor + +- Update Flavors + +- Delete Flavors + + +Health +~~~~~~ + +The following operations are available for Health: + +- Ping for basic health status + +- Get detailed health status + + +Use Cases +--------- + +Queuing systems are used to coordinate tasks within an application. Here +are some examples: + +- **Backup**: A backup application might use a queuing system to + connect the actions that users do in the a control panel to the + customer's backup agent on a server. When a customer wants to start a + backup, they simply choose "start backup" on a panel. Doing so causes + the producer to put a "startBackup" message into the queue. Every few + minutes, the agent on the customers server (the worker) checks the + queue to see if it has any new messages to act on. The agent claims + the "startBackup" message and kicks off the backup on the customer's + server. + +- **Storage**: Gathering statistics for a large, distributed storage + system can be a long process. The storage system can use a queuing + system to ensure that jobs complete, even if one initially fails. + Since messages are not deleted until after the worker has completed + the job, the storage system can make sure that no job goes undone. If + the worker fails to complete the job, the message stays in the queue + to be completed by another server. In this case, a worker claims a + message to perform a statistics job, but the claim's TTL expired and + the message is put back into the queue when the job took too long to + complete (meaning that it most likely failed). By giving the claim a + TTL, applications can protect themselves from workers going off-line + while processing a message. After a claim's TTL expires, the message + is put back into the queue for another worker to claim. + +- **Email**: The team for an email application is constantly migrating + customer email from old versions to newer ones, so they develop a + tool to let customers do it themselves. The migrations take a long + time, so they cannot be done with single API calls, or by a single + server. When a user starts a migration job from their portal, the + migration tool sends messages to the queue with details of how to run + the migration. A set of migration engines, the consumers in this + case, periodically check the queues for new migration tasks, claim + the messages, perform the migration, and update a database with the + migration details. This process allows a set of servers to work + together to accomplish large migrations in a timely manner. + +Following are some generic use cases for Messaging service: + +- Distribute tasks among multiple workers (transactional job queues) + +- Forward events to data collectors (transactional event queues) + +- Publish events to any number of subscribers (event broadcasting) + +- Send commands to one or more agents (point-to-point messaging or + event broadcasting) + +- Request an action or get information from a Remote Procedure Call + (RPC) agent (point-to-point messaging) + +Additional Resources +==================== + +For more information about using the API, see the Messaging service API v2 +Reference. All you need to get started with Messaging service is the +getting started guide, the reference, and your Cloud account. + +For information about the OpenStack Zaqar API, see +`OpenStack API Reference `__. + +This API uses standard HTTP 1.1 response codes as documented at +`www.w3.org/Protocols/rfc2616/rfc2616-sec10.html `__. + +Glossary +======== + +**Claim** +The process of a worker checking out a message to perform a task. +Claiming a message prevents other workers from attempting to process the +same messages. + +**Claim TTL** +Defines how long a message will be in claimed state. A message can be +claimed by one worker at a time. + +**Consumer** +A server that claims messages from the queue. + +**Message** +A task, a notification, or any meaningful data that a producer or +publisher sends to the queue. A message exists until it is deleted by a +recipient or automatically by the system based on a TTL (time-to-live) +value. + +**Message TTL** +Defines how long a message will be accessible. + +**Producer** +A server or application that sends messages to the queue. + +**Producer - Consumer** +A pattern where each worker application that reads the queue has to +claim the message in order to prevent duplicate processing. Later, when +work is done, the worker is responsible for deleting the message. If +message is not deleted in a predefined time, it can be claimed by other +workers. + +**Publisher** +A server or application that posts messages to the queue with the intent +to distribute information or updates to multiple subscribers. + +**Publisher - Subscriber** +A pattern where all worker applications have access to all messages in +the queue. Workers cannot delete or update messages. + +**Queue** +The entity that holds messages. Ideally, a queue is created per work +type. For example, if you want to compress files, you would create a +queue dedicated to this job. Any application that reads from this queue +would only compress files. + +**Subscriber** +An observer that watches messages like an RSS feed but does not claim +any messages. + +**TTL** +Time-to-live value. + +**Worker** +A client that claims messages from the queue and performs actions based +on those messages. diff --git a/doc/source/headers_queue_api_working.rst b/doc/source/headers_queue_api_working.rst new file mode 100644 index 000000000..15e025b5e --- /dev/null +++ b/doc/source/headers_queue_api_working.rst @@ -0,0 +1,356 @@ +Common Headers +============== + +Each request to the Message Queuing API must include certain standard +and extended HTTP headers (as shown in the following table). These +headers provide host, agent, authentication, and other pertinent +information to the server. The following table provides the common +headers used by the API. + +.. list-table:: + :widths: 50 50 + :header-rows: 1 + + * - Header + - Description + * - Host + - Host name of the API + * - Date + - Current date and time + * - Accept + - Media type to use. Initially, only ``application/json`` is + supported. **Note: The "Accept" header is required.** + * - Accept-Encoding + - Specifies that the agent accepts gzip-encoded response bodies + * - Content-Type + - ``application/json`` + * - Content-Length + - For ``POST`` or ``PUT`` requests, the length in bytes of the + message document being submitted + * - X-Auth-Token + - Authorization token + * - X-Project-Id + - An ID for a project to which the value of X-Auth-Token grants + access. Queues are created under this project. The project ID + is the same as the account ID (also sometimes called tenant ID). + * - Client-ID + - A UUID for each client instance. The UUID must be submitted in + its canonical form (for example, 3381af92-2b9e-11e3-b191-71861300734c). + The client generates the Client-ID once. Client-ID persists + between restarts of the client so the client should + reuse that same Client-ID. + +**Note: All message-related operations require the use of "Client-ID" in +the headers to ensure that messages are not echoed back to the client that +posted them, unless the client explicitly requests this.** + +Working with the Message Queuing API +==================================== + +This chapter contains a simple exercise with some basic Message Queuing +requests that you will commonly use. Example requests are provided in +cURL, followed by the response. + +For a complete list of operations available for Message Queuing, see :doc:`getting_started` +Each operation is fully described in the `Message Queuing API v2 +Reference `_. + +Create Queue +------------ + +The Create Queue operation creates a queue in the region of your choice. + +The body of the PUT request is empty. + +The template is as follows: + +.. code:: json + + PUT {endpoint}/queues/{queue_name} + +The ``queue_name`` parameter specifies the name to give the queue. The +name *must not* exceed 64 bytes in length and is limited to US-ASCII +letters, digits, underscores, and hyphens. + +Following are examples of a Create Queue request and response: + +.. code-block:: bash + + curl -i -X PUT https://queues.api.openstack.org/v2/queues/samplequeue \ + -H "X-Auth-Token: " \ + -H "Accept: application/json" \ + -H "X-Project-Id: " + +.. code:: json + + HTTP/1.1 201 Created + Content-Length: 0 + Location: /v2/queues/samplequeue + +Post Message +------------ + +The Post Message operation inserts one or more messages in a queue. + +You can submit up to 10 messages in a single request, but you must +encapsulate them in a collection container (an array in JSON, even for a +single message - without the JSON array, you receive an "Invalid body +request" error message). You can use the resulting value of the location +header or response body to retrieve the created messages for further +processing if needed. + +The template is as follows: + +.. code:: json + + POST {endpoint}/queues/{queue_name}/messages + +The client specifies only the body and ttl attributes for the message. +Metadata, such as id and age, is added. + +The response body contains a list of resource paths that correspond to +each message submitted in the request, in the same order as they were +submitted. + +If a server-side error occurs during the processing of the submitted +messages, a partial list is returned. The ``partial`` attribute is set +to ``true``, and the client tries to post the remaining messages again. + + **Important** + + The ``partial`` attribute has been deprecated in the v1.0 API and is + not available in the v1.1 API. Drivers are now required to operate + in a transactional manner. In other words, either all messages must + be posted, or none of them. + +The ``body`` attribute specifies an arbitrary document that constitutes +the body of the message being sent. + +The following rules apply for the maximum size: + +- The size is limited to 256 KB for the entire request body (as-is), + including whitespace. + +- The maximum size of posted messages is the maximum size of the entire + request document (rather than the sum of the individual message + ``body`` field values as it was earlier releases). On error, the + client is notified of by how much the request exceeded the limit. + +The document *must* be valid JSON. (The Message Queuing service +validates it.) + +The ``ttl`` attribute specifies the lifetime of the message. When the +lifetime expires, the server deletes the message and removes it from the +queue. Valid values are 60 through 1209600 seconds (14 days). + + **Note** + + The server might not actually delete the message until its age + reaches (ttl + 60) seconds. So there might be a delay of 60 seconds + after the message expires before it is deleted. + +The following are examples of a Post Message request and response: + +.. code:: bash + + curl -i -X POST https://queues.api.openstack.org/v1/queues/samplequeue/messages -d \ + '[{"ttl": 300,"body": {"event": "BackupStarted"}},{"ttl": 60,"body": {"play": "hockey"}}]' \ + -H "Content-type: application/json" \ + -H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \ + -H "X-Auth-Token: " \ + -H "Accept: application/json" \ + -H "X-Project-Id: " + +.. code:: json + + HTTP/1.1 201 Created + Content-Length: 153 + Content-Type: application/json; charset=utf-8 + Location: /v1/queues/samplequeue/messages?ids=51ca00a0c508f154c912b85c,51ca00a0c508f154c912b85d + + {"partial": false, "resources": ["/v1/queues/samplequeue/messages/51ca00a0c508f154c912b85c", "/v1/queues/samplequeue/messages/51ca00a0c508f154c912b85d"]} + +Claim Messages +-------------- + +The Claim Messages operation claims a set of messages (up to the value +of the ``limit`` parameter) from oldest to newest and skips any messages +that are already claimed. If there are no messages available to claim, +the Message Queuing service returns an HTTP ``204 No Content`` response +code. + +The template is as follows: + +.. code:: json + + POST {endpoint}/queues/{queue_name}/claims{?limit} + Content-Type: application/json + + { + "ttl": {claim_ttl}, + "grace": {message_grace} + } + +The client (worker) needs to delete the message when it has finished +processing it. The client deletes the message before the claim expires +to ensure that the message is processed only once. If a client needs +more time, the Cloud Service provides the Update Claim operation to make +changes. See the Message Queuing API v1 Reference for a description of +this operation. As part of the delete operation, workers specify the +claim ID (which is best done by simply using the provided href). If +workers perform these actions, then if a claim simply expires, the +server can return an error and notify the worker of a possible race +condition. This action gives the worker a chance to roll back its own +processing of the given message because another worker can claim the +message and process it. + +The age given for a claim is relative to the server's clock. The claim's +age is useful for determining how quickly messages are getting processed +and whether a given message's claim is about to expire. + +When a claim expires, it is released back to the queue for other workers +to claim. (If the original worker failed to process the message, another +client worker can then claim the message.) + +The ``limit`` parameter specifies the number of messages to claim. The +``limit`` parameter is configurable. The default is 20. Messages are +claimed based on the number of messages available. The server might +claim and return less than the requested number of messages. + +The ``ttl`` attribute specifies the lifetime of the claim. While +messages are claimed, they are not available to other workers. The value +must be between 60 and 43200 seconds (12 hours). + +The ``grace`` attribute specifies the message grace period in seconds. +Valid values are between 60 and 43200 seconds (12 hours). To deal with +workers that have stopped responding (for up to 1209600 seconds or 14 +days, including claim lifetime), the server extends the lifetime of +claimed messages to be at least as long as the lifetime of the claim +itself, plus the specified grace period. If a claimed message normally +lives longer than the grace period, its expiration is not adjusted. it + +Following are examples of a Claim Messages request and response: + +.. code:: bash + + curl -i -X POST https://queues.api.openstack.org/v1/queues/samplequeue/claims -d \ + '{"ttl": 300,"grace":300}' \ + -H "Content-type: application/json" \ + -H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \ + -H "X-Auth-Token: " \ + -H "Accept: application/json" \ + -H "X-Project-Id: " + +.. code:: http + + HTTP/1.1 201 OK + Content-Length: 164 + Content-Type: application/json; charset=utf-8 + Location: /v1/queues/samplequeue/claims/51ca011c821e7250f344efd6 + X-Project-Id: + + [ + { + "body": { + "event": "BackupStarted" + }, + "age": 124, + "href": "\/v1\/queues\/samplequeue\/messages\/51ca00a0c508f154c912b85c?claim_id=51ca011c821e7250f344efd6", + "ttl": 300 + } + ] + +Delete Message with Claim ID +---------------------------- + +The Delete Message operations deletes messages. + +The template is as follows: + +.. code:: http + + DELETE {endpoint}/queues/{queue_name}/messages/{message_id}{?claim_id} + +The ``message_id`` parameter specifies the message to delete. + +The ``claim_id`` parameter specifies that the message is deleted only if +it has the specified claim ID and that claim has not expired. This +specification is useful for ensuring that only one worker processes any +given message. When a worker's claim expires before it deletes a message +that it has processed, the worker must roll back any actions it took +based on that message because another worker can now claim and process +the same message. + +Following are examples of a Delete Message request and response: + +.. code:: bash + + curl -i -X DELETE https://queues.api.openstack.org/v1/queues/samplequeue/messages/51ca00a0c508f154c912b85c?claim_id=51ca011c821e7250f344efd6 \ + -H "Content-type: application/json" \ + -H "X-Auth-Token: " \ + -H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \ + -H "Accept: application/json" \ + -H "X-Project-Id: " + +.. code:: http + + HTTP/1.1 204 No Content + +Release Claim +------------- + +The Release Claim operation immediately releases a claim, making any +remaining, undeleted) messages associated with the claim available to +other workers. + +The template is as follows: + +.. code:: http + + DELETE {endpoint}/queues/{queue_name}/claims/{claim_id} + +This operation is useful when a worker is performing a graceful +shutdown, fails to process one or more messages, or is taking longer +than expected to process messages and wants to make the remainder of the +messages available to other workers. + +Following are examples of a Release Claim request and response: + +.. code:: bash + + curl -i -X DELETE https://queues.api.openstack.org/v1/queues/samplequeue/claims/51ca011c821e7250f344efd6 \ + -H "Content-type: application/json" \ + -H "X-Auth-Token: " \ + -H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \ + -H "Accept: application/json" \ + -H "X-Project-Id: " + +.. code:: http + + HTTP/1.1 204 No Content + +Delete Queue +------------ + +The Delete Queue operation immediately deletes a queue and all of its +existing messages. + +The template is as follows: + +.. code:: http + + DELETE {endpoint}/queues/{queue_name} + +Following are examples of a Delete Queue request and response: + +.. code:: bash + + curl -i -X DELETE https://queues.api.openstack.org/v1/queues/samplequeue \ + -H "Content-type: application/json" \ + -H "X-Auth-Token: " \ + -H "Accept: application/json" \ + -H "X-Project-Id: " + +.. code:: http + + HTTP/1.1 204 No Content diff --git a/doc/source/index.rst b/doc/source/index.rst index 99b2a1015..fa808d9bc 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -126,6 +126,17 @@ In order to keep these layers decoupled, we have established that transport drivers must guarantee that the incoming data is well-formed and storage drivers must enforce their data model stays consistent. +User Guide +========== + +.. toctree:: + :maxdepth: 1 + + getting_started + send_request_api.rst + authentication_tokens.rst + headers_queue_api_working.rst + Setting up Zaqar in development environment =========================================== diff --git a/doc/source/send_request_api.rst b/doc/source/send_request_api.rst new file mode 100644 index 000000000..22e35a406 --- /dev/null +++ b/doc/source/send_request_api.rst @@ -0,0 +1,89 @@ +Send Requests to the API +======================== + +You have several options for sending requests through an API: + +- Developers and testers may prefer to use cURL, the command-line tool + from http://curl.haxx.se/. + + With cURL you can send HTTP requests and receive responses back from + the command line. + +- If you like to use a more graphical interface, the REST client for + Firefox also works well for testing and trying out commands, see + https://addons.mozilla.org/en-US/firefox/addon/restclient/. + +- You can also download and install rest-client, a Java application to + test RESTful web services, from + https://github.com/wiztools/rest-client. + +Sending API Requests Using cURL +------------------------------- + +cURL is a command-line tool that is available in UNIX® system-based +environments and Apple Mac OS X® systems, and can be downloaded for +Microsoft Windows® to interact with the REST interfaces. For more +information about cURL, visit http://curl.haxx.se/. + +cURL enables you to transmit and receive HTTP requests and responses +from the command line or from within a shell script. As a result, you +can work with the REST API directly without using one of the client +APIs. + +The following cURL command-line options are used in this guide to run +the examples. + +.. list-table:: + :widths: 50 50 + :header-rows: 1 + + * - Option + - Description + * - ``-d`` + - Sends the specified data in a ``POST`` request to the HTTP server. + * - ``-i`` + - Includes the HTTP header in the output. + * - ``-H HEADER`` + - Specifies an HTTP header in the request. + * - ``-X`` + - Specifies the request method to use when communicating with the HTTP + server. The specified request is used instead of the default + method, which is GET. For example, ``-X PUT`` specifies to use + the ``PUT`` request method. + +**Note** If you have the tools, you can run the cURL JSON request examples +with the following options to format the output from cURL: +`` | python -mjson.tool``. + +Copying and Pasting cURL Request Examples into a Terminal Window +---------------------------------------------------------------- + +To run the cURL request examples shown in this guide on Linux or Mac +systems, perform the following actions: + +1. Copy and paste each example from the HTML version of this guide into + an ASCII text editor (for example, vi or TextEdit). You can click on + the small document icon to the right of each request example to + select it. + +2. Modify each example with your required account information and so + forth, as detailed in this guide. + +3. After you are finished modifying the text for the cURL request + example with your information (for example, ``your_username`` + and ``your_api_key``), paste it into your terminal window. + +4. Press Enter to run the cURL command. + + **Note** + + The carriage returns in the cURL request examples that are part of + the cURL syntax are escaped with a backslash (\\) in order to avoid + prematurely terminating the command. However, you should not escape + carriage returns inside the JSON message within the command. + + **Tip** + + If you have trouble copying and pasting the examples as described, + try typing the entire example on one long line, removing all the + backslash line continuation characters. diff --git a/doc/user-guide/zaqar-get-started/os-zaqar-apiGettingStarted.xml b/doc/user-guide/zaqar-get-started/os-zaqar-apiGettingStarted.xml deleted file mode 100644 index f093d8707..000000000 --- a/doc/user-guide/zaqar-get-started/os-zaqar-apiGettingStarted.xml +++ /dev/null @@ -1,1153 +0,0 @@ - - - - - - -GET'> -PUT'> -POST'> -DELETE'> - - - - - - '> - - - - - '> -]> - - Message Queuing API v1 Getting Started Guide - Message Queuing Getting Started Guide - - - - - - - - - - API v1 - Message Queuing - 2014-02-20 - - - - Copyright details are filled in by the build - system. - - - - This document is intended for software developers interested in developing - applications using the Message Queuing Application Programming Interface - (API). - - - - 2014-02-20 - - Initial document for OpenStack incubation - incubation. - - - - 2014-08-30 - - The partial attribute is no longer supported. - - - - - - Document Change History - This version of the document replaces and obsoletes all - earlier versions. The most recent changes are described in - the following table: - - - - Overview - Message Queuing is a RESTful API-based messaging service that supports distributed web - applications. A message service such as Message Queuing is a vital component of large, - distributed web applications. You can use Message Queuing for public, private, and - hybrid cloud environments. - As you develop distributed web applications, you often have multiple agents set up to - complete sets of tasks for those applications. These tasks can be anything from creating - users to deleting blocks of storage. Message Queuing provides a simple interface to - create these tasks as queues, messages, and claims and to post, claim, read, and delete - them as the tasks are needed and performed. - Message Queuing handles the distribution of tasks, but it does not necessarily manage - the order of the tasks. Applications handle the workflow at a higher level. - Message Queuing is based on the OpenStack Zaqar project. - This guide explains how to access and start using the API so that you can begin to use - Message Queuing for your applications. Instructions are given for how to properly enter - the necessary URLs, using cURL, to set up and use a basic set of Message Queuing - operations. -
- Prerequisites for Running Examples - In - order to run the examples in this guide, you must have - the following prerequisites: - - - A Cloud account - - - A username and password, as - specified during registration - - - Prior knowledge of HTTP/1.1 - conventions - - - Basic familiarity with Cloud and RESTful - APIs - - -
-
- How Message Queuing Works - Following is an overview of how Message Queuing works. For definitions of Message - Queuing terms, see the . - - - You create a queue to which producers or - publishers post messages. - - - Workers (consumers or subscribers) claim or - get a message from the queue, complete the - work in that message, and delete the - message. - If a worker will be off-line before it - completes the work in a message, the worker - can retire the claim's time to live (TTL), - putting the message back into the queue for - another worker to claim. - - - Subscribers monitor the claims from these - queues to track activity and help troubleshoot - errors. - - - For the majority of use cases, Message Queuing is not responsible for the ordering - of messages. However, if there is only a single producer, Message Queuing ensures - that messages are handled in a First In, First Out (FIFO) order. -
- Messaging Patterns - The Message Queuing API supports a variety of - messaging patterns including the following: - - - Task distribution - - - Event broadcasting - - - Point-to-point messaging - - -
-
- Task distribution - The task distribution pattern has the following - characteristics: - - - A producer is programmed to send messages to a - queue. - - - Multiple workers (or consumers) are programmed to - monitor a queue. - - - Only one worker can claim a message so that - no other worker can claim the message and - duplicate the work. - - - The worker must delete the message when work - is done. - - - TTL restores a message to an unclaimed state - if the worker never finishes. - - - This pattern is ideal for dispatching jobs to multiple - processors. -
-
- Event Broadcasting - Characteristics of the event broadcasting pattern - are: - - - The publisher sends messages to a - queue. - - - Multiple observers (or subscribers) get the - messages in the queue. - - - Multiple observers take action on each - message. - - - Observers send a marker to skip messages - already seen. - - - TTL eventually deletes messages. - - - This pattern is ideal for notification of events to - multiple observers at once. -
-
- Point-to-point messaging - Characteristics of the point-to-point messaging - pattern are: - - - The publisher sends messages to a queue. - - - The consumer gets the messages in the queue. - - - The consumer can reply with the result of - processing a message by sending another message - to the same queue (queues are duplex by - default). - - - The publisher gets replies from the queue. - - - The consumer sends a marker to skip messages - already seen. - - - TTL eventually deletes messages. - - - This pattern is ideal for communicating with a - specific client, especially when a reply is desired - from that client. -
-
- Message Queuing Operations - This section lists all of the operations that are available in the Message - Queuing API. This document uses some of the most common operations in . - For details about all of the operations, see the Message Queuing - API v1 Reference. -
- Home Document - The following operation is available for the - home document: - - - Get Home Document - - -
-
- Queues - The following operations are available for - queues: - - - Create Queue - - - List Queues - - - Check Queue Existence - - - Set Queue Metadata - - - Get Queue Metadata - - - Get Queue Stats - - - Delete Queue - - -
-
- Messages - The following operations are available for - messages: - - - Post Message - - - Get Messages - - - Get a Specific Message - - - Get a Set of Messages by ID - - - Delete Message - - - Delete a Set of Messages by - ID - - -
-
- Claims - The following operations are available for - claims: - - - Claim Messages - - - Query Claim - - - Update Claim - - - Release Claim - - -
-
-
- Use Cases - Queuing systems are used to coordinate tasks - within an application. Here are some - examples: - - - Backup: - A backup application might use a queuing - system to connect the actions that users - do in the a control panel to the - customer's backup agent on a server. When - a customer wants to start a backup, they - simply choose "start backup" on a panel. - Doing so causes the producer to put a - "startBackup" message into the queue. - Every few minutes, the agent on the - customers server (the worker) checks the - queue to see if it has any new messages to - act on. The agent claims the "startBackup" - message and kicks off the backup on the - customer's server. - - - Storage: Gathering statistics - for a large, distributed storage system - can be a long process. The storage system - can use a queuing system to ensure that - jobs complete, even if one initially - fails. Since messages are not deleted - until after the worker has completed the - job, the storage system can make sure that - no job goes undone. If the worker fails to - complete the job, the message stays in the - queue to be completed by another server. - In this case, a worker claims a message to - perform a statistics job, but the claim's - TTL expired and the message is put back - into the queue when the job took too long - to complete (meaning that it most likely - failed). By giving the claim a TTL, - applications can protect themselves from - workers going off-line while processing a - message. After a claim's TTL expires, the - message is put back into the queue for - another worker to claim. - - - Email: - The team for an email application is - constantly migrating customer email from - old versions to newer ones, so they - develop a tool to let customers do it - themselves. The migrations take a long - time, so they cannot be done with single - API calls, or by a single server. When a - user starts a migration job from their - portal, the migration tool sends messages - to the queue with details of how to run - the migration. A set of migration engines, - the consumers in this case, periodically - check the queues for new migration tasks, - claim the messages, perform the migration, - and update a database with the migration - details. This process allows a set of - servers to work together to accomplish - large migrations in a timely manner. - - - - Following are some generic use cases for Message Queuing: - - - Distribute tasks among multiple workers (transactional - job queues) - - - Forward events to data collectors (transactional event - queues) - - - Publish events to any number of subscribers - (event broadcasting) - - - Send commands to one or more agents - (point-to-point messaging or event broadcasting) - - - Request an action or get information from a Remote - Procedure Call (RPC) agent (point-to-point - messaging) - - -
-
- - - Send Requests to the API - You have several options for sending requests through an - API: - - - Developers and testers may prefer to use cURL, - the command-line tool from http://curl.haxx.se/. - With cURL you can send HTTP requests and receive - responses back from the command line. - - - If you like to use a more graphical interface, - the REST client for Firefox also works well for - testing and trying out commands, see https://addons.mozilla.org/en-US/firefox/addon/restclient/. - - - - You can also download and install rest-client, a - Java application to test RESTful web services, - from http://code.google.com/p/rest-client/. - - - -
- Sending API Requests Using cURL - cURL is a command-line tool that is available in - UNIX® system-based environments and Apple Mac OS X® - systems, and can be downloaded for Microsoft Windows® - to interact with the REST interfaces. For more - information about cURL, visit http://curl.haxx.se/. - cURL enables you to transmit and receive HTTP - requests and responses from the command line or from - within a shell script. As a result, you can work with - the REST API directly without using one of the client - APIs. - The following cURL command-line options are used in - this guide to run the examples. - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
cURL Command-Line Options
OptionDescription
-dSends the specified data in a - POST request to the HTTP - server.
-iIncludes the HTTP header in the - output.
-H HEADERSpecifies an HTTP header in the - request.
-XSpecifies the request method to use when - communicating with the HTTP server. The - specified request is used instead of the - default method, which is GET. For example, - -X PUT specifies to use - the PUT request method.
- - If you have the tools, you can run the cURL JSON - request examples with the following options to - format the output from cURL: <curl - JSON request example> | python - -mjson.tool. - -
-
- Copying and Pasting cURL Request Examples into a - Terminal Window - To run the cURL request examples shown in this guide - () on - Linux or Mac systems, perform the following - actions: - - - Copy and paste each example from the HTML - version of this guide into an ASCII text - editor (for example, vi or TextEdit). You can - click on the small document icon to the right - of each request example to select it. - - - Modify each example with your required - account information and so forth, as detailed - in this guide. - - - After you are finished modifying the text - for the cURL request example with your - information (for example,your_username - and your_api_key), - paste it into your terminal window. - - - Press Enter to run the - cURL command. - - - - The carriage returns in the cURL request - examples that are part of the cURL syntax are - escaped with a backslash (\) in order to avoid - prematurely terminating the command. However, you - should not escape carriage returns inside the JSON - message within the command. - - - If you have trouble copying and pasting the - examples as described, try typing the entire - example on one long line, removing all the - backslash line continuation characters. - -
- - - Generate an Authentication Token - You can use cURL to try the - authentication process in two steps: get a token, and send the token to a service. - - Get an authentication token by providing your user name and either your - API key or your password. Here are examples of both approaches: - You can request a token by providing your user name and your - password. - $ curl -X POST https://localhost:5000/v2.0/tokens -d '{"auth":{"passwordCredentials":{"username": "joecool", "password":"coolword"}, "tenantId":"5"}}' -H 'Content-type: application/json' - Successful authentication returns a token which you can use as evidence - that your identity has already been authenticated. To use the token, pass it - to other services as an X-Auth-Token header. - Authentication also returns a service catalog, listing the endpoints you - can use for Cloud services. - - - Use the authentication token to send a GET to a service - you would like to use. - - - Authentication tokens are typically valid for 24 hours. Applications should be - designed to re-authenticate after receiving a 401 (Unauthorized) response from a service - endpoint. - - If you programmatically parse an authentication response, be aware that service - names are stable for the life of the particular service and can be used as keys. You - should also be aware that a user's service catalog can include multiple - uniquely-named services that perform similar functions. - - - - Common Headers - Each request to the Message Queuing API must include certain standard and extended - HTTP headers (as shown in the following table). These headers provide host, agent, - authentication, and other pertinent information to the server. The following table - provides the common headers used by the API. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Common Headers
HeaderDescription
HostHost name of the API
DateCurrent date and time
AcceptMedia type to use. Initially, only - application/json is - supported. Note: The - Accept header is - required.
Accept-EncodingSpecifies that the agent accepts gzip-encoded - response bodies
Content-Typeapplication/json
Content-LengthFor POST or - PUT requests, the - length in bytes of the message document being - submitted
X-Auth-TokenAuthorization token
X-Project-IdAn ID for a project to which the value of - X-Auth-Token grants access. Queues are created - under this project. The project ID is the same - as the account ID (also sometimes called - tenant ID).
Client-IDA UUID for each client instance. The UUID must - be submitted in its canonical form (for - example, - 3381af92-2b9e-11e3-b191-71861300734c). The - client generates the Client-ID once. Client-ID - persists between restarts of the client so the - client should reuse that same Client-ID. - Note: All - message-related operations require the use - of Client-ID in the headers - to ensure that messages are not echoed - back to the client that posted them, - unless the client explicitly requests - this.
- - - Working with the Message Queuing API - This chapter contains a simple exercise with some basic Message Queuing requests that - you will commonly use. Example requests are provided in cURL, followed by the - response. - For a complete list of operations available for Message Queuing, see . Each operation is fully described in the - Message Queuing API v1 Reference. -
- Create Queue - The Create Queue operation creates a queue in the - region of your choice. - The body of the PUT request is empty. - The template is as follows: - - PUT {endpoint}/queues/{queue_name} - - The queue_name parameter - specifies the name to give the queue. The name - must not exceed - 64 bytes in length and is limited to US-ASCII letters, - digits, underscores, and hyphens. - Following are examples of a Create Queue request and - response: - - cURL Create Queue Request - curl -i -X PUT https://queues.api.openstack.org/v1/queues/samplequeue \ --H "X-Auth-Token: your_auth_token" \ --H "Accept: application/json" \ --H "X-Project-Id: your_project_ID" - - - Create Queue Response - HTTP/1.1 201 Created -Content-Length: 0 -Location: /v1/queues/samplequeue - -
-
- Post Message - The Post Message operation inserts one or more - messages in a queue. - You can submit up to 10 messages in a single - request, but you must encapsulate them in a collection - container (an array in JSON, even for a single message - - without the JSON array, you receive an "Invalid body - request" error message). You can use the resulting - value of the location header or response body to - retrieve the created messages for further processing - if needed. - The template is as - follows:POST {endpoint}/queues/{queue_name}/messages - The client specifies only the body and ttl - attributes for the message. Metadata, such as id and age, is added. - The response body contains a list of resource paths - that correspond to each message submitted in the - request, in the same order as they were submitted. - If a server-side error occurs during the processing - of the submitted messages, a partial list is returned. - The partial attribute is set to - true, and the client tries - to post the remaining messages again. - - - The partial attribute has been deprecated in the v1.0 API - and is not available in the v1.1 API. - Drivers are now required to operate in a transactional manner. - In other words, either all messages must be posted, or none of them. - - - - The body attribute specifies an - arbitrary document that constitutes the body of the - message being sent. - The following rules apply for the maximum size: - - The size is limited to 256 KB for the - entire request body (as-is), including - whitespace. - - - The maximum size of posted messages is - the maximum size of the entire request - document (rather than the sum of the - individual message body field - values as it was earlier releases). On - error, the client is notified of by how - much the request exceeded the - limit. - - - - The document must be valid JSON. (The Message - Queuing service validates it.) - The ttl attribute specifies - the lifetime of the message. When the lifetime - expires, the server deletes the message and removes it - from the queue. Valid values are 60 through 1209600 - seconds (14 days). - - The server might not actually delete the - message until its age reaches (ttl + 60) seconds. So - there might be a delay of 60 seconds after the - message expires before it is deleted. - - The following are examples of a Post Message request - and response: - - cURL Post Message Request - curl -i -X POST https://queues.api.openstack.org/v1/queues/samplequeue/messages -d \ -'[{"ttl": 300,"body": {"event": "BackupStarted"}},{"ttl": 60,"body": {"play": "hockey"}}]' \ --H "Content-type: application/json" \ --H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \ --H "X-Auth-Token: your_auth_token" \ --H "Accept: application/json" \ --H "X-Project-Id: your_project_ID" - - - Post Message Response - HTTP/1.1 201 Created -Content-Length: 153 -Content-Type: application/json; charset=utf-8 -Location: /v1/queues/samplequeue/messages?ids=51ca00a0c508f154c912b85c,51ca00a0c508f154c912b85d - -{"partial": false, "resources": ["/v1/queues/samplequeue/messages/51ca00a0c508f154c912b85c", "/v1/queues/samplequeue/messages/51ca00a0c508f154c912b85d"]} - - -
-
- Claim Messages - The Claim Messages operation claims a set of messages (up to the value of the - limit parameter) from oldest to newest and skips any - messages that are already claimed. If there are no messages available to claim, the - Message Queuing service returns an HTTP 204 No Content response - code. - The template is as follows: - POST {endpoint}/queues/{queue_name}/claims{?limit} -Content-Type: application/json - -{ - "ttl": {claim_ttl}, - "grace": {message_grace} -} - The client (worker) needs to delete the message when it has finished processing - it. The client deletes the message before the claim expires to ensure that the - message is processed only once. If a client needs more time, the Cloud Service - provides the Update Claim operation to make changes. See the Message - Queuing API v1 Reference for a description of this operation. As - part of the delete operation, workers specify the claim ID (which is best done by - simply using the provided href). If workers perform these actions, then if a claim - simply expires, the server can return an error and notify the worker of a possible - race condition. This action gives the worker a chance to roll back its own - processing of the given message because another worker can claim the message and - process it. - The age given for a claim is relative to the - server's clock. The claim's age is useful for - determining how quickly messages are getting processed - and whether a given message's claim is about to - expire. - When a claim expires, it is released back to the - queue for other workers to claim. (If the original - worker failed to process the message, another client - worker can then claim the message.) - The limit parameter specifies - the number of messages to claim. The limit parameter is configurable. - The default is 20. Messages - are claimed based on the number of messages available. - The server might claim and return less than the - requested number of messages. - The ttl attribute specifies - the lifetime of the claim. While messages are claimed, - they are not available to other workers. The value - must be between 60 and 43200 seconds (12 - hours). - The grace attribute specifies - the message grace period in seconds. Valid values are - between 60 and 43200 seconds (12 hours). To deal with - workers that have stopped responding (for up to - 1209600 seconds or 14 days, including claim lifetime), - the server extends the lifetime of claimed messages to - be at least as long as the lifetime of the claim - itself, plus the specified grace period. If a claimed - message normally lives longer than the grace period, - its expiration is not adjusted. it - Following are examples of a Claim Messages request - and response: - - cURL Claim Messages Request - curl -i -X POST https://queues.api.openstack.org/v1/queues/samplequeue/claims -d \ -'{"ttl": 300,"grace":300}' \ --H "Content-type: application/json" \ --H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \ --H "X-Auth-Token: your_auth_token" \ --H "Accept: application/json" \ --H "X-Project-Id: your_project_ID" - - - Claim Messages Response - HTTP/1.1 201 OK -Content-Length: 164 -Content-Type: application/json; charset=utf-8 -Location: /v1/queues/samplequeue/claims/51ca011c821e7250f344efd6 -X-Project-Id: your_project_ID - -[ - { - "body": { - "event": "BackupStarted" - }, - "age": 124, - "href": "\/v1\/queues\/samplequeue\/messages\/51ca00a0c508f154c912b85c?claim_id=51ca011c821e7250f344efd6", - "ttl": 300 - } -] - -
-
- Delete Message with Claim ID - The Delete Message operations deletes - messages. - The template is as - follows:DELETE {endpoint}/queues/{queue_name}/messages/{message_id}{?claim_id} - - The message_id parameter - specifies the message to delete. - The claim_id parameter - specifies that the message is deleted only if it has - the specified claim ID and that claim has not expired. - This specification is useful for ensuring that only - one worker processes any given message. When a - worker's claim expires before it deletes a message - that it has processed, the worker must roll back any - actions it took based on that message because another - worker can now claim and process the same - message. - Following are examples of a Delete Message request - and response: - - cURL Delete Message Request - curl -i -X DELETE https://queues.api.openstack.org/v1/queues/samplequeue/messages/51ca00a0c508f154c912b85c?claim_id=51ca011c821e7250f344efd6 \ --H "Content-type: application/json" \ --H "X-Auth-Token: your_auth_token" \ --H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \ --H "Accept: application/json" \ --H "X-Project-Id: your_project_ID" - - - Delete Message Response - HTTP/1.1 204 No Content - -
-
- Release Claim - The Release Claim operation immediately releases a - claim, making any remaining, undeleted) messages - associated with the claim available to other - workers. - The template is as follows: - DELETE {endpoint}/queues/{queue_name}/claims/{claim_id} - This operation is useful when a worker is performing - a graceful shutdown, fails to process one or more - messages, or is taking longer than expected to process - messages and wants to make the remainder of the - messages available to other workers. - Following are examples of a Release Claim request - and response: - - cURL Release Claim Request - curl -i -X DELETE https://queues.api.openstack.org/v1/queues/samplequeue/claims/51ca011c821e7250f344efd6 \ --H "Content-type: application/json" \ --H "X-Auth-Token: your_auth_token" \ --H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \ --H "Accept: application/json" \ --H "X-Project-Id: your_project_ID" - - - Release Claim Response - HTTP/1.1 204 No Content - -
-
- Delete Queue - The Delete Queue operation immediately deletes a - queue and all of its existing messages. - The template is as - follows:DELETE {endpoint}/queues/{queue_name} - Following are examples of a Delete Queue request and - response: - - cURL Delete Queue Request - curl -i -X DELETE https://queues.api.openstack.org/v1/queues/samplequeue \ --H "Content-type: application/json" \ --H "X-Auth-Token: your_auth_token" \ --H "Accept: application/json" \ --H "X-Project-Id: your_project_ID" - - - Delete Queue Response - HTTP/1.1 204 No Content - - -
- - - Additional Resources - For more information about using the API, see the Message Queuing API v1 - Reference. All you need to get started with Message Queuing is the - getting started guide, the reference, and your Cloud account. - For information about the OpenStack Zaqar API, see - wiki.openstack.org/wiki/Zaqar/specs/api/v1. - This API uses standard HTTP 1.1 response codes as - documented at www.w3.org/Protocols/rfc2616/rfc2616-sec10.html. - - - - Glossary - - Claim - - The process of a worker checking out a message - to perform a task. Claiming a message prevents - other workers from attempting to process the same - messages. - - - - Claim TTL - - Defines how long a message will be in claimed - state. A message can be claimed by one worker at a - time. - - - - Consumer - - A server that claims messages from the - queue. - - - - Message - - A task, a notification, or any meaningful data - that a producer or publisher sends to the queue. A - message exists until it is deleted by a recipient - or automatically by the system based on a TTL - (time-to-live) value. - - - - Message TTL - - Defines how long a message will be accessible. - - - - - Producer - - A server or application that sends messages to - the queue. - - - - Producer - Consumer - - A pattern where each worker application that - reads the queue has to claim the message in order - to prevent duplicate processing. Later, when work - is done, the worker is responsible for deleting - the message. If message is not deleted in a - predefined time, it can be claimed by other - workers. - - - - Publisher - - A server or application that posts messages to - the queue with the intent to distribute - information or updates to multiple - subscribers. - - - - Publisher - Subscriber - - A pattern where all worker applications have - access to all messages in the queue. Workers - cannot delete or update messages. - - - - Queue - - The entity that holds messages. Ideally, a queue - is created per work type. For example, if you want - to compress files, you would create a queue - dedicated to this job. Any application that reads - from this queue would only compress files. - - - - Subscriber - - An observer that watches messages like an RSS - feed but does not claim any messages. - - - - TTL - - Time-to-live value. - - - - Worker - - A client that claims messages from the queue and - performs actions based on those messages. - - - - diff --git a/doc/user-guide/zaqar-get-started/pom.xml b/doc/user-guide/zaqar-get-started/pom.xml deleted file mode 100644 index 6a2724ed2..000000000 --- a/doc/user-guide/zaqar-get-started/pom.xml +++ /dev/null @@ -1,68 +0,0 @@ - - - org.openstack.docs - parent-pom - 1.0.0-SNAPSHOT - ../pom.xml - - 4.0.0 - zaqar-get-started - jar - Message Queuing API v1 Getting Started - - - local - 1 - - - - - com.rackspace.cloud.api - clouddocs-maven-plugin - - - - generate-webhelp - - generate-webhelp - - generate-sources - - - os-zaqar-apiGettingStarted.xml - appendix toc,title - article/appendix nop - article toc,title - book toc,title,figure,table,example,equation - chapter toc,title - section toc - part toc,title - qandadiv toc - qandaset to - reference toc,title - set toc,title - zaqar-get-started - zaqar-get-started - - - - - 1 - 0 - builtforOpenStack - true - 1 - 0 - 1 - 0 - false - true - . - http://docs.openstack.org/zaqar-getting-started/content/ - - - - -