Beware the Kraken! What happens when you start plotting transfer rates.

FTS transfers  are how the WLCG moves alot of its data. I decided decided to look at what the instantaneous rates within the transfers were.Lines in the log files appear as:

I decided to plot the value of the instantaneous rate with respect to how often this value appeared. Plotting this for 2/18 FTS servers at RAL for ~1month of transfers gives:  



This has been described as a Kraken, the hand of Freddie Kruger, a sea anemone or a leafless tree's branches blowing in the wind . Please leave comments on your own suggestion!!

I also decided to look at the subset of data for FTS transfers to the new CEPH storage at the RAL Tier1 and saw this:



My first thought is that it is similar to the Cinderella castle by Disney.
https://www.pinterest.com/explore/disney-castle-silhouette/ :)

First look at IPV4/6 Dual mesh perfSonar results for the RAL-LCG2 Tier1

We now have the RAL-LCG2 perfSONAR production hosts dual stacked. I though I would have a look at difference between IPv4 and IPv6 values for throughput rates and round trip times from traceroute  measurements. I first decided to look at the bandwidth measurements between pairs of hosts, and got the following plot:


So I then decided to look at what we get if we compare the ratio of throughput measurements and compare it to the ratio of the round trip times of the traceroute measurements:

Current storage scope within GridPP

With a new project year upon us, I decided to review which site's storage support is used used by the WLCG VOs, what SRMs are used and the file systems used. On that last part , we now don't just have filesystems but also object stores with the usage of CEPH. Other filesystems are XFS, ZFS, HDFS, Spectrum Scale (or the artist formerly known as GPFS), and Lustre.

In terms of Storage elements/systems we have DPM, dCache, Castor, classic SE, stand alone xrootd, and stand alone gsiftp services. When it comes to the regional T2s and who they are used by, the following helps.

I thought about embedding SE system into the font used for each site, but thought that was too much overlay of information.

What rates can we get for single file transfers?

Recently has a conversation regarding what the expected rates we can see for single file data transfers; so I went to have a look... For recalling a single file 275GB ( we have then at the Tie1 one for some VOs) I got the following results. These are all just examples and have no statiscal basis to them. but as a first step it gives interesting results. When recalling back for our tape system i get the following graph showing over 300MB/s:

Of course, I am also interested to see what happens when I copy a file into castor.The floowing is an example of a similar 275Gb file being copied across the WAN  then is written to tape. As you can see the intial write phase (~75MB/s) is lower than the rate for the file to be written to tape (~260MB/s)    Copy across the network for similar size file and then uploading into Castor:

 
 The end of the log file for this transfer is shown here:

N.B This transfer was using four concurrent stream within the gsiftp transfer.

It is (I think) interesting to look at the theoretical rate limit for transfers for a single stream between the two hosts in this transfer using some predictions; (from website https://www.switch.ch/network/tools/tcp_throughput/) :

So we may need to work on this...
The higher level monitoring worried me for this file until I realised the display options greatly effect the perceived rate. In this (in what might be an atypical) example; by solely changing the bin size I was able to change the perceived rate from 75MB/s to 460MB/s as seen in the two pictures below:

So I will rely on log file info from now on... (well mainly)

Now my site has the advantage (hindrance) of two storage systems  using different hardware configurations and  separate implementation of a gsiftp server; so I decided to see what rate I could get between the two.... And I managed over 100MB/s. This 100MB/s was for a 5 section poll time in the transfer as seen here:

What I also find interesting is that there seems to be a systematic difference between the average rate for transfers depending on transfer direction. (70-80MB/s one direction 90-95MB/s in reverse.) whether this difference is worth investigating is a question I will leave to the reader to decide. Also of interest may be to see what the effect of changing data transfer protocol has; but that it for another day...

Happy Birthday To ME ! ( almost) The contiuing adventures of Dave the dataset

With my 6th birthday nearly open me, (how the last six years have flown by....) I thought I should have an update.I an my children still exist in 50 rooms across 29 houses.
The list of houses were my children and I reside are:
AGLT2
BNL-OSG2
CERN-PROD
CSCS-LCG2
DESY-HH
FZK-LCG2
GRIF-IRFU
GRIF-LAL
GRIF-LPNHE
IFIC-LCG2
IN2P3-CC
IN2P3-LPSC
INFN-T1
MPPMU
NCG-INGRID-PT
NDGF-T1
NERSC
NET2
NIKHEF-ELPROD
PRAGUELCG2
RAL-LCG2
SMU
SWT2
TOKYO-LCG2
TR-10-ULAKBIM
TRIUMF-LCG2
UKI-SOUTHGRID-OX-HEP
WEIZMANN-LCG2
WUPPERTALPROD

Nationalities are Canadian, Czech, Dutch, French, German, Israeli, Italian, Nordic, Japanese, Portuguese, Spanish, Swiss, Turkish, UK and USA
And the types of room are:
CPB
DAQ
DATADISK
DATATAPE
DET-LARG
GROUPTAPE
LOCALGROUPDISK
PERF-EGAMMA
PERF-FLAVTAG
PERF-MUONS
PHYS-BEAUTY
PHYS-SM
PHYS-SUSY
SCRATCHDISK
TZERO


There are 284 unique individuals in my family tree: 4 are triplets; 56 are twins. 224 individuals have no replicas; so are at risk of extinction if any particular room is destroyed. OF the 284;  64 are Ursulas', 7 are Gavins' and 195 are Dirks'.

Each child varies from 1 to 3531 files (103 are only a single file.) Size varies from 450B to 3.13TB.
In total, there are 7.96515TB of unique data spread across  22841 files. ( giving an average file size form the dataset and its children of 348MB.)

When not to optimise best network settings ( and you should be satified with good...)

Just a quick note.. So whilst reviewing the current recommendations for network settings our our disk servers from the advice of our friends at fasterdata.es.net , I noticed they also had a section for settings for network performance machines running tests similar to those which on our machines as part of our WLCG work using the perfSONAR monitoring tools. 

Firstly I was interested to see that there were possible differences in settings between these machine types, but it also go me thinking that should we apply the perfSONAR settings to our perfSONAR machines. Un-intuitively the answer is NO! The gist of the advice from my friendly sys-admin with greater knowledge on this matter than I on this matter put it as follows.  `"` You want to have similar settings on the performance node (perfSONAR) as you do on you production disk servers so that you have  a good representation of the expected performance of reality..  `"`