x/swiftonfile
Code Issues Proposed changes
Swift Object Server implementation that enables objects created using the Swift API to be accessed as files on a POSIX filesystem.
368 Commits 5 Branches 9 Tags 1.8 MiB
Python 98.9%
Shell 0.8%
RPM Spec 0.3%
aa1bfb3e67
Go to file
Prashanth Pai aa1bfb3e67 Rebase to Swift 2.2.1 release 
NOTE:
The previous rebase was to Swift 2.1.0 and this rebase is to
Swift 2.2.1 (first release in kilo series). There was a
Swift 2.2.0 (last release in juno series) release in between.

Change-Id: Ibce2e299935e165db89a91a6fe8c4c5c027db098
Signed-off-by: Prashanth Pai <ppai@redhat.com>
2015-02-02 11:17:59 +05:30
bin changing references of gluster to swiftonfile 2014-07-08 11:04:31 -04:00
doc/markdown Workflow documentation is now in infra-manual 2014-12-05 03:30:47 +00:00
etc Fix bug in object name constraint 2014-12-01 13:05:29 +05:30
extras doc: Update developer documentation 2014-10-01 13:33:53 -04:00
swiftonfile Rebase to Swift 2.2.1 release 2015-02-02 11:17:59 +05:30
test Rebase to Swift 2.2.1 release 2015-02-02 11:17:59 +05:30
.functests Fix bug in object name constraint 2014-12-01 13:05:29 +05:30
.functests-ci changes references to swiftonfile in tests scripts 2014-07-08 11:04:31 -04:00
.gitignore Modify unit tests and func tests 2014-05-08 12:11:46 +05:30
.gitmodules Add OpenStack Swift as a git submodule 2013-09-05 04:23:50 -07:00
.gitreview Add .gitreview file 2014-09-22 12:19:08 +05:30
.travis.yml removing blank lines 2014-04-28 19:30:38 -04:00
.unittests changes references to swiftonfile in tests scripts 2014-07-08 11:04:31 -04:00
makerpm.sh changing references of gluster to swiftonfile 2014-07-08 11:04:31 -04:00
MANIFEST.in Add MANIFEST.in 2015-01-09 21:56:59 -07:00
pkgconfig.py changing references of gluster to swiftonfile 2014-07-08 11:04:31 -04:00
README.md Note limitation of swift geo-replication. 2015-01-30 19:03:08 -07:00
requirements.txt Rebase to Swift 2.2.1 release 2015-02-02 11:17:59 +05:30
setup.py new constraints middleware 2014-10-07 16:53:44 -04:00
swiftonfile.spec changing references of gluster to swiftonfile 2014-07-08 11:04:31 -04:00
test-requirements.txt Rebase to Swift 2.2.1 release 2015-02-02 11:17:59 +05:30
tox.ini Rebase to Swift 2.2.1 release 2015-02-02 11:17:59 +05:30

README.md

Build Status

Swift-on-File

Swift-on-File is a Swift Object Server implementation that enables users to access the same data, both as an object and as a file. Data can be stored and retrieved through Swift's REST interface or as files from NAS interfaces including native GlusterFS, GPFS, NFS and CIFS.

Swift-on-File is to be deployed as a Swift storage policy, which provides the advantages of being able to extend an existing Swift cluster and also migrating data to and from policies with different storage backends.

The main difference from the default Swift Object Server is that Swift-on-File stores objects following the same path hierarchy as the object's URL. In contrast, the default Swift implementation stores the object following the mapping given by the Ring, and its final file path is unkown to the user.

For example, an object with URL: https://swift.example.com/v1/acct/cont/obj, would be stored the following way by the two systems:

  • Swift: /mnt/sdb1/2/node/sdb2/objects/981/f79/f566bd022b9285b05e665fd7b843bf79/1401254393.89313.data
  • SoF: /mnt/swiftonfile/acct/cont/obj

Use cases

Swift-on-File can be especially useful in cases where access over multiple protocols is desired. For example, imagine a deployment where video files are uploaded as objects over Swift's REST interface and a legacy video transcoding software access those videos as files.

Along the same lines, data can be ingested over Swift's REST interface and then analytic software like Hadoop can operate directly on the data without having to move the data to a separate location.

Another use case is where users might need to migrate data from an existing file storage systems to a Swift cluster.

Similarly, scientific applications may process file data and then select some or all of the data to publish to outside users through the swift interface.

Limitations and Future plans

Swift-On-File currently works only with Filesystems with extended attributes support. It is also recommended that these Filesystems provide data durability as Swift-On-File should not use Swift's replication mechanisms.

GlusterFS and GPFS are good examples of Filesystems that work well with Swift-on-File. Both provide a posix interface, global namespace, scalability, data replication and support for extended attributes.

Currently, files added over a file interface (e.g., native GlusterFS), do not show up in container listings, still those files would be accessible over Swift's REST interface with a GET request. We are working to provide a solution to this limitation.

Because Swift-On-File relies on the data replication support of the filesystem the Swift Object replicator process does not have any role for containers using the Swift-on-File storage policy. This means that Swift geo replication is not available to objects in in containers using the Swift-on-File storage policy. Multi-site replication for these objects must be provided by the filesystem.

Future plans includes adding support for Filesystems without extended attributes, which should extend the ability to migrate data for legacy storage systems.

Get involved:

To learn more about Swift-On-File, you can watch the presentation given at the Paris OpenStack Summit: Deploying Swift on a File System. The Paris presentation slides can be found here Also see the presentation given at the Atlanta Openstack Summit: Breaking the Mold with Openstack Swift and GlusterFS. The Atlanta presentation slides can be found here.

Join us in contributing to the project. Feel free to file bugs, help with documentation or work directly on the code. You can file bugs or blueprints on launchpad

or find us in the #swiftonfile channel on Freenode.

Guides to get started:

  1. Quick Start Guide with XFS/GlusterFS
  2. Developer Guide