HDFS XRootD Fallback Project Installation Twiki

Contents

This twiki contains installation instructions for each component of the HDFS XRootD Fallback system.

Installing hdfs-xrootd-fallback

Some assumptions made before installation:

• osg-release-3.2 is installed
• Hadoop 2.0 client with FUSE mount is installed

Procedure

1. Install UCSDT2 RPM repo:

   rpm -Uvh http://www.t2.ucsd.edu/repo/el6/x86_64/ucsdt2-repo-1.0-4.el6.noarch.rpm
   

2. Enable repo in /etc/yum.repos.d/ucsdt2.repo:

   [ucsdt2]
   name=UCSDT2 Repo for Enterprise Linux 6 - $basearch
   baseurl=http://www.t2.ucsd.edu/repo/el6/$basearch/
   priority=90
   enabled=1
   gpgcheck=1
   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-UCSDT2
   

3. Install hdfs-xrootd-fallback:

   yum install hdfs-xrootd-fallback
   

4. Copy config template:

   cp /etc/hadoop/conf.osg/xfbfs-site.xml /etc/hadoop/conf/
   

5. Modify /etc/hadoop/conf/xfbfs-site.xml as needed
   NOTE Fields are explained in the comments of the provided xfbfs-site.xml template file.
   At UCSD T2: scp uaf-9:~matevz/xrootd-ucsd/xfbfs/xfbfs-site.xml /etc/hadoop/conf/
6. Add the following lines to /etc/hadoop/conf/core-site.xml:

     <property>
       <name>fs.xfbfs.impl</name>
       <value>org.xrootd.hdfs.fallback.XrdFallBackFileSystem</value>
       <description>XRootd fallback filesystem for xfbfs protocol.</description>
     </property>
   

7. Unmount FUSE mount point.
8. In /etc/fstab, change the URI scheme from dfs:// to xfbfs://
   Example:

   hadoop-fuse-dfs#xfbfs://namenode_host:9000 /hadoop fuse rdbuffer=131072,rw,notrash,big_writes 0 0
   

9. Mount FUSE mount point.

Installing hdfs-xrootd-fbmon

To be written...

Installing hdfs-xrootd-healer

To be written...

-- JeffDost - 15 Feb 2014