In my opinion, an easy to use distributed filesystem on Linux is long due. Today I came across XtreemFS which accomplishes just that. Setting up a basic cluster with 2 OSDs takes no more than 10 minutes. Let’s walk through the processes.
First, install the binary packages using repositories made available by the XtreemFS team. Once the repository file is registered, install these packages:
xtreemfs-client xtreemfs-server xtreemfs-tools
Now suppose my server to host the DIR and MRC service is called KDC01, and my OSDs will be running on KDC01 and WKS01. On KDC01, edit the following config files under /etc/xos/xtreemfs and set dir_service.host = kdc01. For the mrc and osd config files, set up hostname = kdc01
- mrcconfig.properties
- osdconfig.properties
On WKS01 where a second OSD will be running, edit the osd config file, point dir_service.host = kdc01 and hostname = wks01
- osdconfig.properties
Then start up the services. On KDC01, start the following services:
/etc/init.d/xtreemfs-dir start
/etc/init.d/xtreemfs-mrc start
/etc/init.d/xtreemfs-osd start
On WKS01, start up the OSD service
/etc/init.d/xtreemfs-osd start
Now create the XtreemFS filesystem:
mkfs.xtreemfs kdc01/myVol # run once on KDC01
mount kdc01/myVol /data # run on KDC01 and WKS01
This is basically done. By default, 1 replica will be stored on the entire XtreenFS volume. XtreemFS client is smart enough to contact the best OSD for the file resource. In case it’s desirable to maintain more than 1 replica of the same file, it can be done with the xtfsutil tool. For example, to configure XtreemFS to replicate a file to all available OSDs:
$ xtfsutil -r WqRq vmlinuz-2.6.32-431.11.2.el6.x86_64 $ xtfsutil -a auto vmlinuz-2.6.32-431.11.2.el6.x86_64 $ xtfsutil vmlinuz-2.6.32-431.11.2.el6.x86_64 Path (on volume) /vmlinuz-2.6.32-431.11.2.el6.x86_64 XtreemFS file Id 2418e2d4-5852-4dee-9b95-706343424844:3 XtreemFS URL pbrpc://kdc01:32638/myVol/vmlinuz-2.6.32-431.11.2.el6.x86_64 Owner root Group root Type file Replication policy WqRq XLoc version 2 Replicas: Replica 1 Striping policy STRIPING_POLICY_RAID0 / 1 / 128kB OSD 1 5f130657-d2d4-41f4-a94a-7bbd61fdd4d9 (wks01:32640) Replica 2 Striping policy STRIPING_POLICY_RAID0 / 1 / 128kB OSD 1 274841c2-a997-412b-b5ef-44e5390cfa5b (kdc01:32640)
The actual files are stored under /var/lib/xtreemfs/objs/. That can be easily customized in osd’s config file.
This is a really quick guide on how to set up a simple cluster. For serious deployment, it’s a good idea to set up DIR replication to eliminate single point of failure. Check out the userguide from XtreemFS for more info.