GET'> PUT'> POST'> DELETE'> '> '> ]> Message Queuing Getting Started Guide API v1 Message Queuing 2014-02-20 Copyright details are filled in by the build system. This document is intended for software developers interested in developing applications using the Message Queuing Application Programming Interface (API). 2014-02-20 Initial document for OpenStack incubation incubation. 2014-08-30 The partial attribute is no longer supported. This version of the document replaces and obsoletes all earlier versions. The most recent changes are described in the following table: Message Queuing is a RESTful API-based messaging service that supports distributed web applications. A message service such as Message Queuing is a vital component of large, distributed web applications. You can use Message Queuing for public, private, and hybrid cloud environments. As you develop distributed web applications, you often have multiple agents set up to complete sets of tasks for those applications. These tasks can be anything from creating users to deleting blocks of storage. Message Queuing provides a simple interface to create these tasks as queues, messages, and claims and to post, claim, read, and delete them as the tasks are needed and performed. Message Queuing handles the distribution of tasks, but it does not necessarily manage the order of the tasks. Applications handle the workflow at a higher level. Message Queuing is based on the OpenStack Zaqar project. This guide explains how to access and start using the API so that you can begin to use Message Queuing for your applications. Instructions are given for how to properly enter the necessary URLs, using cURL, to set up and use a basic set of Message Queuing operations.

In order to run the examples in this guide, you must have the following prerequisites: A Cloud account A username and password, as specified during registration Prior knowledge of HTTP/1.1 conventions Basic familiarity with Cloud and RESTful APIs

Following is an overview of how Message Queuing works. For definitions of Message Queuing terms, see the . You create a queue to which producers or publishers post messages. Workers (consumers or subscribers) claim or get a message from the queue, complete the work in that message, and delete the message. If a worker will be off-line before it completes the work in a message, the worker can retire the claim's time to live (TTL), putting the message back into the queue for another worker to claim. Subscribers monitor the claims from these queues to track activity and help troubleshoot errors. For the majority of use cases, Message Queuing is not responsible for the ordering of messages. However, if there is only a single producer, Message Queuing ensures that messages are handled in a First In, First Out (FIFO) order.

The Message Queuing API supports a variety of messaging patterns including the following: Task distribution Event broadcasting Point-to-point messaging

The task distribution pattern has the following characteristics: A producer is programmed to send messages to a queue. Multiple workers (or consumers) are programmed to monitor a queue. Only one worker can claim a message so that no other worker can claim the message and duplicate the work. The worker must delete the message when work is done. TTL restores a message to an unclaimed state if the worker never finishes. This pattern is ideal for dispatching jobs to multiple processors.

Characteristics of the event broadcasting pattern are: The publisher sends messages to a queue. Multiple observers (or subscribers) get the messages in the queue. Multiple observers take action on each message. Observers send a marker to skip messages already seen. TTL eventually deletes messages. This pattern is ideal for notification of events to multiple observers at once.

Characteristics of the point-to-point messaging pattern are: The publisher sends messages to a queue. The consumer gets the messages in the queue. The consumer can reply with the result of processing a message by sending another message to the same queue (queues are duplex by default). The publisher gets replies from the queue. The consumer sends a marker to skip messages already seen. TTL eventually deletes messages. This pattern is ideal for communicating with a specific client, especially when a reply is desired from that client.

This section lists all of the operations that are available in the Message Queuing API. This document uses some of the most common operations in . For details about all of the operations, see the Message Queuing API v1 Reference.

The following operation is available for the home document: Get Home Document

The following operations are available for queues: Create Queue List Queues Check Queue Existence Set Queue Metadata Get Queue Metadata Get Queue Stats Delete Queue

The following operations are available for messages: Post Message Get Messages Get a Specific Message Get a Set of Messages by ID Delete Message Delete a Set of Messages by ID

The following operations are available for claims: Claim Messages Query Claim Update Claim Release Claim

Queuing systems are used to coordinate tasks within an application. Here are some examples: Backup: A backup application might use a queuing system to connect the actions that users do in the a control panel to the customer's backup agent on a server. When a customer wants to start a backup, they simply choose "start backup" on a panel. Doing so causes the producer to put a "startBackup" message into the queue. Every few minutes, the agent on the customers server (the worker) checks the queue to see if it has any new messages to act on. The agent claims the "startBackup" message and kicks off the backup on the customer's server. Storage: Gathering statistics for a large, distributed storage system can be a long process. The storage system can use a queuing system to ensure that jobs complete, even if one initially fails. Since messages are not deleted until after the worker has completed the job, the storage system can make sure that no job goes undone. If the worker fails to complete the job, the message stays in the queue to be completed by another server. In this case, a worker claims a message to perform a statistics job, but the claim's TTL expired and the message is put back into the queue when the job took too long to complete (meaning that it most likely failed). By giving the claim a TTL, applications can protect themselves from workers going off-line while processing a message. After a claim's TTL expires, the message is put back into the queue for another worker to claim. Email: The team for an email application is constantly migrating customer email from old versions to newer ones, so they develop a tool to let customers do it themselves. The migrations take a long time, so they cannot be done with single API calls, or by a single server. When a user starts a migration job from their portal, the migration tool sends messages to the queue with details of how to run the migration. A set of migration engines, the consumers in this case, periodically check the queues for new migration tasks, claim the messages, perform the migration, and update a database with the migration details. This process allows a set of servers to work together to accomplish large migrations in a timely manner. Following are some generic use cases for Message Queuing: Distribute tasks among multiple workers (transactional job queues) Forward events to data collectors (transactional event queues) Publish events to any number of subscribers (event broadcasting) Send commands to one or more agents (point-to-point messaging or event broadcasting) Request an action or get information from a Remote Procedure Call (RPC) agent (point-to-point messaging)

You have several options for sending requests through an API: Developers and testers may prefer to use cURL, the command-line tool from http://curl.haxx.se/. With cURL you can send HTTP requests and receive responses back from the command line. If you like to use a more graphical interface, the REST client for Firefox also works well for testing and trying out commands, see https://addons.mozilla.org/en-US/firefox/addon/restclient/. You can also download and install rest-client, a Java application to test RESTful web services, from http://code.google.com/p/rest-client/.

cURL is a command-line tool that is available in UNIX® system-based environments and Apple Mac OS X® systems, and can be downloaded for Microsoft Windows® to interact with the REST interfaces. For more information about cURL, visit http://curl.haxx.se/. cURL enables you to transmit and receive HTTP requests and responses from the command line or from within a shell script. As a result, you can work with the REST API directly without using one of the client APIs. The following cURL command-line options are used in this guide to run the examples.

cURL Command-Line Options

Option	Description
-d	Sends the specified data in a POST request to the HTTP server.
-i	Includes the HTTP header in the output.
-H HEADER	Specifies an HTTP header in the request.
-X	Specifies the request method to use when communicating with the HTTP server. The specified request is used instead of the default method, which is GET. For example, -X PUT specifies to use the PUT request method.

If you have the tools, you can run the cURL JSON request examples with the following options to format the output from cURL: <curl JSON request example> | python -mjson.tool.

To run the cURL request examples shown in this guide () on Linux or Mac systems, perform the following actions: Copy and paste each example from the HTML version of this guide into an ASCII text editor (for example, vi or TextEdit). You can click on the small document icon to the right of each request example to select it. Modify each example with your required account information and so forth, as detailed in this guide. After you are finished modifying the text for the cURL request example with your information (for example,your_username and your_api_key), paste it into your terminal window. Press Enter to run the cURL command. The carriage returns in the cURL request examples that are part of the cURL syntax are escaped with a backslash (\) in order to avoid prematurely terminating the command. However, you should not escape carriage returns inside the JSON message within the command. If you have trouble copying and pasting the examples as described, try typing the entire example on one long line, removing all the backslash line continuation characters.

You can use cURL to try the authentication process in two steps: get a token, and send the token to a service. Get an authentication token by providing your user name and either your API key or your password. Here are examples of both approaches: You can request a token by providing your user name and your password. $ curl -X POST https://localhost:5000/v2.0/tokens -d '{"auth":{"passwordCredentials":{"username": "joecool", "password":"coolword"}, "tenantId":"5"}}' -H 'Content-type: application/json' Successful authentication returns a token which you can use as evidence that your identity has already been authenticated. To use the token, pass it to other services as an X-Auth-Token header. Authentication also returns a service catalog, listing the endpoints you can use for Cloud services. Use the authentication token to send a GET to a service you would like to use. Authentication tokens are typically valid for 24 hours. Applications should be designed to re-authenticate after receiving a 401 (Unauthorized) response from a service endpoint. If you programmatically parse an authentication response, be aware that service names are stable for the life of the particular service and can be used as keys. You should also be aware that a user's service catalog can include multiple uniquely-named services that perform similar functions. Each request to the Message Queuing API must include certain standard and extended HTTP headers (as shown in the following table). These headers provide host, agent, authentication, and other pertinent information to the server. The following table provides the common headers used by the API.

Common Headers

Header	Description
Host	Host name of the API
Date	Current date and time
Accept	Media type to use. Initially, only application/json is supported. Note: The Accept header is required.
Accept-Encoding	Specifies that the agent accepts gzip-encoded response bodies
Content-Type	application/json
Content-Length	For POST or PUT requests, the length in bytes of the message document being submitted
X-Auth-Token	Authorization token
X-Project-Id	An ID for a project to which the value of X-Auth-Token grants access. Queues are created under this project. The project ID is the same as the account ID (also sometimes called tenant ID).
Client-ID	A UUID for each client instance. The UUID must be submitted in its canonical form (for example, 3381af92-2b9e-11e3-b191-71861300734c). The client generates the Client-ID once. Client-ID persists between restarts of the client so the client should reuse that same Client-ID. Note: All message-related operations require the use of Client-ID in the headers to ensure that messages are not echoed back to the client that posted them, unless the client explicitly requests this.

This chapter contains a simple exercise with some basic Message Queuing requests that you will commonly use. Example requests are provided in cURL, followed by the response. For a complete list of operations available for Message Queuing, see . Each operation is fully described in the Message Queuing API v1 Reference.

The Create Queue operation creates a queue in the region of your choice. The body of the PUT request is empty. The template is as follows: PUT {endpoint}/queues/{queue_name} The queue_name parameter specifies the name to give the queue. The name must not exceed 64 bytes in length and is limited to US-ASCII letters, digits, underscores, and hyphens. Following are examples of a Create Queue request and response: curl -i -X PUT https://queues.api.openstack.org/v1/queues/samplequeue \ -H "X-Auth-Token: your_auth_token" \ -H "Accept: application/json" \ -H "X-Project-Id: your_project_ID" HTTP/1.1 201 Created Content-Length: 0 Location: /v1/queues/samplequeue

The Post Message operation inserts one or more messages in a queue. You can submit up to 10 messages in a single request, but you must encapsulate them in a collection container (an array in JSON, even for a single message - without the JSON array, you receive an "Invalid body request" error message). You can use the resulting value of the location header or response body to retrieve the created messages for further processing if needed. The template is as follows:POST {endpoint}/queues/{queue_name}/messages The client specifies only the body and ttl attributes for the message. Metadata, such as id and age, is added. The response body contains a list of resource paths that correspond to each message submitted in the request, in the same order as they were submitted. If a server-side error occurs during the processing of the submitted messages, a partial list is returned. The partial attribute is set to true, and the client tries to post the remaining messages again. The partial attribute has been deprecated in the v1.0 API and is not available in the v1.1 API. Drivers are now required to operate in a transactional manner. In other words, either all messages must be posted, or none of them. The body attribute specifies an arbitrary document that constitutes the body of the message being sent. The following rules apply for the maximum size: The size is limited to 256 KB for the entire request body (as-is), including whitespace. The maximum size of posted messages is the maximum size of the entire request document (rather than the sum of the individual message body field values as it was earlier releases). On error, the client is notified of by how much the request exceeded the limit. The document must be valid JSON. (The Message Queuing service validates it.) The ttl attribute specifies the lifetime of the message. When the lifetime expires, the server deletes the message and removes it from the queue. Valid values are 60 through 1209600 seconds (14 days). The server might not actually delete the message until its age reaches (ttl + 60) seconds. So there might be a delay of 60 seconds after the message expires before it is deleted. The following are examples of a Post Message request and response: curl -i -X POST https://queues.api.openstack.org/v1/queues/samplequeue/messages -d \ '[{"ttl": 300,"body": {"event": "BackupStarted"}},{"ttl": 60,"body": {"play": "hockey"}}]' \ -H "Content-type: application/json" \ -H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \ -H "X-Auth-Token: your_auth_token" \ -H "Accept: application/json" \ -H "X-Project-Id: your_project_ID" HTTP/1.1 201 Created Content-Length: 153 Content-Type: application/json; charset=utf-8 Location: /v1/queues/samplequeue/messages?ids=51ca00a0c508f154c912b85c,51ca00a0c508f154c912b85d {"partial": false, "resources": ["/v1/queues/samplequeue/messages/51ca00a0c508f154c912b85c", "/v1/queues/samplequeue/messages/51ca00a0c508f154c912b85d"]}

The Claim Messages operation claims a set of messages (up to the value of the limit parameter) from oldest to newest and skips any messages that are already claimed. If there are no messages available to claim, the Message Queuing service returns an HTTP 204 No Content response code. The template is as follows: POST {endpoint}/queues/{queue_name}/claims{?limit} Content-Type: application/json { "ttl": {claim_ttl}, "grace": {message_grace} } The client (worker) needs to delete the message when it has finished processing it. The client deletes the message before the claim expires to ensure that the message is processed only once. If a client needs more time, the Cloud Service provides the Update Claim operation to make changes. See the Message Queuing API v1 Reference for a description of this operation. As part of the delete operation, workers specify the claim ID (which is best done by simply using the provided href). If workers perform these actions, then if a claim simply expires, the server can return an error and notify the worker of a possible race condition. This action gives the worker a chance to roll back its own processing of the given message because another worker can claim the message and process it. The age given for a claim is relative to the server's clock. The claim's age is useful for determining how quickly messages are getting processed and whether a given message's claim is about to expire. When a claim expires, it is released back to the queue for other workers to claim. (If the original worker failed to process the message, another client worker can then claim the message.) The limit parameter specifies the number of messages to claim. The limit parameter is configurable. The default is 20. Messages are claimed based on the number of messages available. The server might claim and return less than the requested number of messages. The ttl attribute specifies the lifetime of the claim. While messages are claimed, they are not available to other workers. The value must be between 60 and 43200 seconds (12 hours). The grace attribute specifies the message grace period in seconds. Valid values are between 60 and 43200 seconds (12 hours). To deal with workers that have stopped responding (for up to 1209600 seconds or 14 days, including claim lifetime), the server extends the lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus the specified grace period. If a claimed message normally lives longer than the grace period, its expiration is not adjusted. it Following are examples of a Claim Messages request and response: curl -i -X POST https://queues.api.openstack.org/v1/queues/samplequeue/claims -d \ '{"ttl": 300,"grace":300}' \ -H "Content-type: application/json" \ -H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \ -H "X-Auth-Token: your_auth_token" \ -H "Accept: application/json" \ -H "X-Project-Id: your_project_ID" HTTP/1.1 201 OK Content-Length: 164 Content-Type: application/json; charset=utf-8 Location: /v1/queues/samplequeue/claims/51ca011c821e7250f344efd6 X-Project-Id: your_project_ID [ { "body": { "event": "BackupStarted" }, "age": 124, "href": "\/v1\/queues\/samplequeue\/messages\/51ca00a0c508f154c912b85c?claim_id=51ca011c821e7250f344efd6", "ttl": 300 } ]

The Delete Message operations deletes messages. The template is as follows:DELETE {endpoint}/queues/{queue_name}/messages/{message_id}{?claim_id} The message_id parameter specifies the message to delete. The claim_id parameter specifies that the message is deleted only if it has the specified claim ID and that claim has not expired. This specification is useful for ensuring that only one worker processes any given message. When a worker's claim expires before it deletes a message that it has processed, the worker must roll back any actions it took based on that message because another worker can now claim and process the same message. Following are examples of a Delete Message request and response: curl -i -X DELETE https://queues.api.openstack.org/v1/queues/samplequeue/messages/51ca00a0c508f154c912b85c?claim_id=51ca011c821e7250f344efd6 \ -H "Content-type: application/json" \ -H "X-Auth-Token: your_auth_token" \ -H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \ -H "Accept: application/json" \ -H "X-Project-Id: your_project_ID" HTTP/1.1 204 No Content

The Release Claim operation immediately releases a claim, making any remaining, undeleted) messages associated with the claim available to other workers. The template is as follows: DELETE {endpoint}/queues/{queue_name}/claims/{claim_id} This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages and wants to make the remainder of the messages available to other workers. Following are examples of a Release Claim request and response: curl -i -X DELETE https://queues.api.openstack.org/v1/queues/samplequeue/claims/51ca011c821e7250f344efd6 \ -H "Content-type: application/json" \ -H "X-Auth-Token: your_auth_token" \ -H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \ -H "Accept: application/json" \ -H "X-Project-Id: your_project_ID" HTTP/1.1 204 No Content

The Delete Queue operation immediately deletes a queue and all of its existing messages. The template is as follows:DELETE {endpoint}/queues/{queue_name} Following are examples of a Delete Queue request and response: curl -i -X DELETE https://queues.api.openstack.org/v1/queues/samplequeue \ -H "Content-type: application/json" \ -H "X-Auth-Token: your_auth_token" \ -H "Accept: application/json" \ -H "X-Project-Id: your_project_ID" HTTP/1.1 204 No Content

For more information about using the API, see the Message Queuing API v1 Reference. All you need to get started with Message Queuing is the getting started guide, the reference, and your Cloud account. For information about the OpenStack Zaqar API, see wiki.openstack.org/wiki/Zaqar/specs/api/v1. This API uses standard HTTP 1.1 response codes as documented at www.w3.org/Protocols/rfc2616/rfc2616-sec10.html. Claim The process of a worker checking out a message to perform a task. Claiming a message prevents other workers from attempting to process the same messages. Claim TTL Defines how long a message will be in claimed state. A message can be claimed by one worker at a time. Consumer A server that claims messages from the queue. Message A task, a notification, or any meaningful data that a producer or publisher sends to the queue. A message exists until it is deleted by a recipient or automatically by the system based on a TTL (time-to-live) value. Message TTL Defines how long a message will be accessible. Producer A server or application that sends messages to the queue. Producer - Consumer A pattern where each worker application that reads the queue has to claim the message in order to prevent duplicate processing. Later, when work is done, the worker is responsible for deleting the message. If message is not deleted in a predefined time, it can be claimed by other workers. Publisher A server or application that posts messages to the queue with the intent to distribute information or updates to multiple subscribers. Publisher - Subscriber A pattern where all worker applications have access to all messages in the queue. Workers cannot delete or update messages. Queue The entity that holds messages. Ideally, a queue is created per work type. For example, if you want to compress files, you would create a queue dedicated to this job. Any application that reads from this queue would only compress files. Subscriber An observer that watches messages like an RSS feed but does not claim any messages. TTL Time-to-live value. Worker A client that claims messages from the queue and performs actions based on those messages.