In my opinion, an easy to use distributed filesystem on Linux is long due. Today I came across XtreemFS which accomplishes just that. Setting up a basic cluster with 2 OSDs takes no more than 10 minutes. Let’s walk through the processes.

First, install the binary packages using repositories made available by the XtreemFS team. Once the repository file is registered, install these packages:

xtreemfs-client xtreemfs-server xtreemfs-tools

Now suppose my server to host the DIR and MRC service is called KDC01, and my OSDs will be running on KDC01 and WKS01. On KDC01, edit the following config files under /etc/xos/xtreemfs and set = kdc01. For the mrc and osd config files, set up hostname = kdc01


On WKS01 where a second OSD will be running, edit the osd config file, point = kdc01 and hostname = wks01


Then start up the services. On KDC01, start the following services:

/etc/init.d/xtreemfs-dir start
/etc/init.d/xtreemfs-mrc start
/etc/init.d/xtreemfs-osd start

On WKS01, start up the OSD service

/etc/init.d/xtreemfs-osd start

Now create the XtreemFS filesystem:

mkfs.xtreemfs kdc01/myVol # run once on KDC01
mount kdc01/myVol /data # run on KDC01 and WKS01

This is basically done. By default, 1 replica will be stored on the entire XtreenFS volume. XtreemFS client is smart enough to contact the best OSD for the file resource. In case it’s desirable to maintain more than 1 replica of the same file, it can be done with the xtfsutil tool. For example, to configure XtreemFS to replicate a file to all available OSDs:

$ xtfsutil -r WqRq vmlinuz-2.6.32-431.11.2.el6.x86_64
$ xtfsutil -a auto vmlinuz-2.6.32-431.11.2.el6.x86_64
$ xtfsutil vmlinuz-2.6.32-431.11.2.el6.x86_64
Path (on volume) /vmlinuz-2.6.32-431.11.2.el6.x86_64
XtreemFS file Id 2418e2d4-5852-4dee-9b95-706343424844:3
XtreemFS URL pbrpc://kdc01:32638/myVol/vmlinuz-2.6.32-431.11.2.el6.x86_64
Owner root
Group root
Type file
Replication policy WqRq
XLoc version 2
Replica 1
Striping policy STRIPING_POLICY_RAID0 / 1 / 128kB
OSD 1 5f130657-d2d4-41f4-a94a-7bbd61fdd4d9 (wks01:32640)
Replica 2
Striping policy STRIPING_POLICY_RAID0 / 1 / 128kB
OSD 1 274841c2-a997-412b-b5ef-44e5390cfa5b (kdc01:32640)

The actual files are stored under /var/lib/xtreemfs/objs/. That can be easily customized in osd’s config file.

This is a really quick guide on how to set up a simple cluster. For serious deployment, it’s a good idea to set up DIR replication to eliminate single point of failure. Check out the userguide from XtreemFS for more info.

Leave a Comment