11 KiB
Making it durable
For later versions of the guide: Extend the Fractals app to use Swift directly, and show the actual code from there.
Explain how to get objects back out again.
Large object support in Swift http://docs.openstack.org/developer/swift/overview_large_objects.html
This section introduces object storage. OpenStack Object Storage (code-named swift) is open source software for creating redundant, scalable data storage using clusters of standardized servers to store petabytes of accessible data. It is a long-term storage system for large amounts of static data that can be retrieved, leveraged, and updated. Access is via an API, not through a file-system like more traditional storage.
There are a two key concepts to understand in the Object Storage API. The Object Storage API is organized around two types of entities:
- Objects
- Containers
Similar to the Unix programming model, an object is a "bag of bytes" that contains data, such as documents and images. Containers are used to group objects. You can make many objects inside a container, and have many containers inside your account.
If you think about how you traditionally make what you store durable, very quickly you should come to the conclusion that keeping multiple copies of your objects on separate systems is a good way to do that. However, keeping track of multiple copies of objects is a pain, and building that into an app requires a lot of logic. OpenStack Object Storage does this automatically for you behind-the-scenes -replicating each object at least twice before returning 'write success' to your API call. It will always work to ensure that there are three copies of your objects (by default) at all times -replicating them around the system in case of hardware failure, maintenance, network outage or any other kind of breakage. This is very convenient for app creation - you can just dump objects into object storage and not have to care about any of this additional work to keep them safe.
Using Object Storage to store fractals
The Fractals app currently uses the local file system on the instance to store the images it generates. This is not scalable or durable, for a number of reasons.
Because the local file system is ephemeral storage, if the instance
is terminated, the fractal images will be lost along with the instance.
Block based storage, which we will discuss in /block_storage, avoids that
problem, but like local file systems, it requires administration to
ensure that it does not fill up, and immediate attention if disks
fail.
The Object Storage service manages many of these tasks that normally would require the application owner to manage them, and presents a scalable and durable API that you can use for the fractals app, without having to be concerned with the low level details of how the objects are stored and replicated, and growing the storage pool. In fact, Object Storage handles replication intrinsically, storing multiple copies of each object and returning one of them on demand using the API.
First, let's learn how to connect to the Object Storage endpoint:
dotnet
Warning
This section has not yet been completed for the .NET SDK.
fog
Warning
This section has not yet been completed for the fog SDK.
jclouds
Warning
This section has not yet been completed for the jclouds SDK.
libcloud
../samples/libcloud/durability.py
Warning
Libcloud 0.16 and 0.17 are afflicted with a bug that means authentication to a swift endpoint can fail with a Python exception. If you encounter this, you can upgrade your libcloud version, or apply a simple 2-line patch.
Note
Libcloud uses a different connector for Object Storage to all other
OpenStack services, so a conn object from previous sections will not
work here and we have to create a new one named swift.
pkgcloud
Warning
This section has not yet been completed for the pkgcloud SDK.
openstacksdk
Warning
This section has not yet been completed for the OpenStack SDK.
phpopencloud
Warning
This section has not yet been completed for the PHP-OpenCloud SDK.
To begin to store objects, we must first make a container. Call yours
fractals:
libcloud
../samples/libcloud/durability.py
You should see output such as:
<Container: name=fractals, provider=OpenStack Swift>You should now be able to see this container appear in a listing of all containers in your account:
libcloud
../samples/libcloud/durability.py
You should see output such as:
[<Container: name=fractals, provider=OpenStack Swift>]The next logical step is to upload an object. Find a photo of a goat
on line, name it goat.jpg, and upload it to your
fractals container:
libcloud
../samples/libcloud/durability.py
List objects in your fractals container to see if the
upload was successful. Then, download the file to verify that the md5sum
is the same:
libcloud
../samples/libcloud/durability.py
[<Object: name=an amazing goat, size=191874, hash=439884df9c1c15c59d2cf43008180048, provider=OpenStack Swift ...>]../samples/libcloud/durability.py
<Object: name=an amazing goat, size=954465, hash=7513986d3aeb22659079d1bf3dc2468b, provider=OpenStack Swift ...>../samples/libcloud/durability.py
7513986d3aeb22659079d1bf3dc2468bFinally, let's clean up by deleting our test object:
libcloud
../samples/libcloud/durability.py
Note
You need to pass in objects to the delete commands, not object names.
Now there should be no more objects be available in the container
fractals.
../samples/libcloud/durability.py
[]Backup the Fractals from the database on the Object Storage
So let's now use the knowledge from above to backup the images of the Fractals app, stored inside the database right now, on the Object Storage.
Use the fractals' container from above to put the images
in:
libcloud
../samples/libcloud/durability.py
Next, we backup all of our existing fractals from the database to our swift container. A simple for loop takes care of that:
libcloud
../samples/libcloud/durability.py
<Object: name=025fd8a0-6abe-4ffa-9686-bcbf853b71dc, size=61597, hash=b7a8a26e3c0ce9f80a1bf4f64792cd0c, provider=OpenStack Swift ...>
<Object: name=26ca9b38-25c8-4f1e-9e6a-a0132a7a2643, size=136298, hash=9f9b4cac16893854dd9e79dc682da0ff, provider=OpenStack Swift ...>
<Object: name=3f68c538-783e-42bc-8384-8396c8b0545d, size=27202, hash=e6ee0cd541578981c294cebc56bc4c35, provider=OpenStack Swift ...>Note
Replace IP_API_1 with the IP address of the API
instance.
Note
The example code uses the awesome Requests library. Before you try to run the previous script, make sure that it is installed on your system.
Configure the Fractals app to use Object Storage
Warning
Currently, you cannot directly store generated images in OpenStack Object Storage. Please revisit this section again in the future.
Extra features
Delete containers
One call we did not cover and that you probably need to know is how to delete a container. Ensure that you have removed all objects from the container before running this script. Otherwise, the script fails:
libcloud
../samples/libcloud/durability.py
Warning
It is not possible to restore deleted objects. Be careful.
Add metadata to objects
You can also do advanced things like uploading an object with metadata, such as in following example. For more information, see the documentation for your SDK. This option also uses a bit stream to upload the file, iterating bit by bit over the file and passing those bits to Object Storage as they come. Compared to loading the entire file in memory and then sending it, this method is more efficient, especially for larger files.
libcloud
../samples/libcloud/durability.py
It would be nice to have a pointer here to section 9.
Large objects
For efficiency, most Object Storage installations treat large objects
(say, > 5GB) differently than smaller objects.
libcloud
If you are working with large objects, use the
ex_multipart_upload_object call instead of the simpler
upload_object call. Behind the scenes, the call splits the
large object into chunks and creates a special manifest so that the
chunks can be recombined on download. Alter the chunk_size
parameter (in bytes) according to what your cloud can accept.
../samples/libcloud/durability.py
Next steps
You should now be fairly confident working with Object Storage. You can find more information about the Object Storage SDK calls at:
Or, try one of these steps in the tutorial:
/block_storage: Migrate the database to block storage, or use the database-as-a-service component./orchestration: Automatically orchestrate your application./networking: Learn about complex networking./advice: Get advice about operations./craziness: Learn some crazy things that you might not think to do ;)