27 January 2016

Xrootd for all

Xrootd provides, amongst other things, a convenient method to externally access files at a site anywhere in the world using your grid credentials.

Specialist storage systems such as DPM and dcache now include a xrootd server in their deployment.
If you use a stranded POSIX file system (e.g. Lustre, GPFS, NFS) it's possible to set up a standalone xrootd server to export all or part of the file system to external clients.

The LHC experiments have have gone a step further and have set up federated storage services combining storage from several separate sites in to one namespace allowing seamless client access to storage without having to worry where the data is stored. They have provided instructions to setup a service but only for their VO e.g.

Extending this to allow access to other VOs data via xrootd but without the federated storage service is simple.

The xrootd server runs as user xrootd. In order to access files it must have the correct permissions to the files. This can be done by making the xrootd user a member of the appropriate groups across the site (e.g. via NIS).

ypcat -k group 
dteam dteam:x:12345:user1,user2, …, xrootd
atlas atlas:x:13345:user1,user2, …, xrootd

For simplicity I'll make a symlink to the file system I want to export on the xrootd server, e.g.

ln -sf /mnt/lustre_2/storm_3/atlas/ /atlas
ln -sf /mnt/lustre_2/storm_3/dteam/ /dteam

The xrootd server configuration file is /etc/xrootd/xrootd-clustered.cfg. Within this file we need to define the file system to export and do so read only for security,

all.export /dteam r/o
all.export /atlas r/o

We also need to add the VOs to the X509 configuration.

sec.protparm gsi -vomsfun:/usr/lib64/libXrdSecgsiVOMS.so -vomsfunparms:certfmt=raw | vos=atlas,dteam | grps=/atlas,/dteam
acc.authdb /etc/xrootd/auth_file

The /etc/xrootd/auth_file specifies the group/user access rights. The following will give read and list right to members of the atlas group to file under /atlas and dteam group members for files under /dteam

g /atlas /atlas rl
g /dteam /dteam rl

The final configuration files look like

cat xrootd-clustered.cfg
frm.xfr.copycmd /bin/cp /dev/null $PFN

# atlas redirection
all.manager atlas-xrd-uk.cern.ch+:1098
xrootd.redirect atlas-xrd-uk.cern.ch:1094 ? /atlas
all.sitename SITENAME

all.export /dteam r/o
all.export /atlas r/o

all.role server
all.adminpath /var/run/xrootd
all.pidpath /var/run/xrootd
xrootd.async off

# atlas Monitoring
if exec xrootd
xrd.report atl-prod05.slac.stanford.edu:9931 every 60s all -buff -poll sync
# if your sites is in EU uncomment the next line
xrootd.monitor all flush 30s window 5s fstat 60 lfn ops xfr 5 dest redir files info user atlas-fax-eu-collector.cern.ch:9330

# N2N configuration. Please change for your site
oss.namelib /usr/lib64/XrdOucName2NameLFC.so

# X509 configuration, change nothing
xrootd.seclib /usr/lib64/libXrdSec.so
sec.protparm gsi -vomsfun:/usr/lib64/libXrdSecgsiVOMS.so -vomsfunparms:certfmt=raw|vos=atlas,dteam|grps=/atlas,/dteam
sec.protocol /usr/lib64 gsi -ca:1 -crl:3 -gridmap:/dev/null
acc.authdb /etc/xrootd/auth_file
acc.authrefresh 60

[root@xrootd02 xrootd]# cat auth_file
g /atlas /atlas rl
g /dteam /dteam rl

Additional file systems can be added in the same fashion for more VOs e.g. snoplus, t2k …..

It's possible to use argus server for the authentication http://londongrid.blogspot.co.uk/2014/10/xrootd-and-argus-authentication.html

The bandwidth to the file system will be limited by the performance of the xrootd server. For local file access it's still better to use native POSIX access, especially with parallel file systems like Lustre.

No comments: