Milestone Passed with Last Two Years of Transfers Through FTS

Over the Last Two years; the FTS system (as used by the WLCG VOs and others) have moved ~0.5EB of data (over a Billion files). What is an EB ? Well its 1000 PBytes or 10^6 TBytes or 10^9 GBytes or 10^12MBytes or 10^15kBytes or 10^18Bytes. This can be seen form the monitoring page:

Storage system consistency checking required a new methoid to dump nameserver entries from Castor database ( who ever said blog post titles hsould be short and pithy...)

So as to partly fulfil a (re)new request to consistency check our SEs for various WLCG VOs; and for data management of smaller VOs; we realise that the RAL Tier1 needed an improved method to acquire a list per VO if all the files we have in our castor storage system.
We first tried to naively just use the castor "nsfind" command (similar to your normal find command) on our storage system.  However we soon realised this caused problems with our production system. 
So................. We decided to setup a client host and  backup offline database to query. (This also means that we now are acquiring and storing monthly dumps of the storage nameservers for longitudinal analysis (but that is the story of a future post.).

From this we tried creating a dump for one particular VO; and it took 22 days to complete. (this lead to issues as we had wished to dump the database weekly.  He improved matters by two methods.
1-Deleted ~5million old empty directories (wish the VO would do this themselves.)
2- Gave the problem to our ORACLE DBAs to look at and try to improve nsfind or come up with another method.  (The DBAs gave us a new script; which when run on a smaller part of the namespace reduced the complete time from 8.5 hours to 100 minutes.)

The additional benefit from deleting of empty directories reduced the overall dump from 22 days to 3.5 hours. Hopefully this should mean that we can provide regular (time delayed) dumps of the fileslist of our storage.

What can a lossy fibre in the middle of you r network infrastructure do to your network rate.......

Just a quick post regarding how we noticed packet errors on one of a balanced pair of links within our network. By taking the link down (at  ~1700 Wednesday) and  re-seating the end connections lead to a mark improvement in rate and reduction in packet loss.

Here is a plot of the number of packets on the link:

Packets on the link

This clearly shows the period with no packets on the link corresponding to when the link was down.

Now look at the packet errors and discards:

Error/Discards on the link

 This clearly shows packet errors/discards before the change; and none after the change. The packet "loss" rate was ~ 6k/240k or ~14k/560k which equates to ~2.5 %!!The really interest comes when looking at the data rate through the combined link during this period:

Rate

The data rate through the link pair increases by ~10x when the link is removed (form ~0.6Gbps to 6Gbps.) Also note that there is no dramatic loss when the secondary link is re-nabled  (at ~1130 on Thursday.)

Community knowledge: Upgrading an (SL5) DPM disk node to xrootd4.x from xrootd3.x in place.

While developers try to avoid breaking backwards compatibility between versions, sometimes it is necessary. One such situation occurred for the xrootd protocol, as the long-awaited Xrootd 4 series was released earlier this year, bringing (amongst other changes) true IPv6 support.

Unfortunately, due to the changes involved, support for xrootd4 in DPM was not immediately available (and the various tools which provide assistance and conversion for paths for VOs similarly needed porting). As a result, most sites did not move from xrootd3 at the time.

Because of the complexity of the release process (with some packages built for xrootd4 being available from different dates, and multiple repositories involved), the DPM devs published a blog entry in February concerning special instructions for managing Xrootd4 transition.

Much of the complexity of that blog entry is no longer relevant, however, as all of the dependant packages are now available - but many sites still have systems running Xrootd 3 services, including Glasgow.

So, I took a look at the process for moving from Xrootd3 to 4, on a single disk server. (Xrootd3 and 4 based DPM disk servers can co-exist with each other, and a head node with either release, so there's no need to move them all at once.) We predominantly support ATLAS at Glasgow, so the instructions here are focussed on making sure that ATLAS support works.
[NOTE: this is not sufficient to upgrade a head node to xrootd4, which would require a few additional changes, and I have not tested this yet.]

yum update emi-dpm_disk dpm-xrootd xrootd-server-atlas-n2n-plugin dmlite-plugins-adapter

(
Upgrading the emi-dpm_mysql package just pulls in the core dpm/dmlite functionality, as xrootd is an optional protocol.
dpm-xrootd pulls in updates to xrootd and the dmlite interface to it (which is what we want)
xrootd-server-atlas-n2n-plugin is needed for translation of ATLAS VO surl paths into xrootd paths.
dmlite-plugins-adapter updates the adapter library for dmlite, which is used to allow xrootd to get authorisation/authentication from dpm, for example. For some reason, none of the above packages seem to update it automatically, but without it at a new enough release, dpm-xrootd stuff will not be able to properly talk to emi-dpm_mysql stuff.
)

Specifically, you'll need to ensure that:

dmlite-plugins-adapter >= 0.7.0
xrootd-server-atlas-n2n-plugin >= 0.2

You should also check that the dpm-xrootd package pulls in vomsxrd (>= 0.3) as a dependancy - if it doesn't you need to make sure that the WLCG repo is properly enabled.

You'll also need to open
/etc/sysconfig/xrootd
and add
-k fifo
to the contents of any and all variables with names ending _OPTIONS.

Once this is all done, you can happily restart the xrootd services on the node, and all should be well. (tail -f /var/log/xrootd/disk/xrootd.log can help to spot any issues if they do appear).

Castor Rebalancer a success at RAL... sort of...

We recently started using the re-balance feature of Castor Storage at RAL-LCG2

And this looked like good news to allow us to keep number of files and free space balanced across diskservers within a pool. However; a couple of days after we turned this feature on; our production team noticed a vast increase in the number of bad incomplete replicate files being produced. (Good news is that the original files still exist; so there is no loss of data. However we thought it might be good idea to effectively turn off re-balancing with a tweak to the settings on our stagerDB/ transfer management system within Castor. (I have since learned a lot more about the usage and output from our "printdbconfig" and "modifydbconfig" commands!) We have been making changes to various settings but the main settings of current interest for this an other issues have been:

CLASS          KEY                                     VALUE
--------------------------------------------------------------------
D2dCopy       MaxNbRetries                      0
Draining        MaxNbFilesScheduled         200
Draining        MaxNbSchedD2dPerDrain  200
Migration      MaxNbMounts                     7
Rebalancing  MaxNbFilesScheduled         5
Rebalancing  Sensitivity                            100

These current settings seem to have stopped the creation of new problematic files, now "just" need to work out why exactly it seems to have fixed it and see if we can re-enable re-balancing.