ATLAS Jamboree 2019 view from the offiste perspective.

I didn't go in person to the ATLAS Jamboree this year held at CERN. For those who are allowed to view I suggest looking at: https://indico.cern.ch/event/770307/

But I did join for some via vidyo!
Here is my musings about the talk givens I saw. ( Shame I couldn't get involved in coffee/dinner  discussions which are often the most fruitful moments of these meetings):

Even before the main meeting started, there is an interesting talk regarding HPC data access in US an ANL.


In particular, I  like the thought of globus usage and incorporating rucio into DTNs at the sites.  Similar to what was discussed at other sites at the rucio community workshop last week.

In the preview talk, I picked out the switch to Fast Sim rather than Full sim will increase output rate by a factor of 10. A good reminder that user workflow changes could drastically alter computing requirements.
From the main meeting , the following meetings will be of interest on a data storage:

Data Organization and Management Activities: Third Party Copy and (storage) Quality of Service
TPC: details on DPM

DOMA ACCESS: Caches 

DDM Ops overview

Diskless and lightweight sites: consolidation of storage

Data Carousel

Networking - best practice for sites, and evolution

WLCG evolution strategy

 

One thing it di was cause me to think what if; (and I stress the if is me not ATLAS musing,)  ATLAS  wanted to read 1PB of data a day from Tape at RAL and then distribute it across the world?

 

 

Understanding Globus connect/online... is it doing a lot??

I have made further progress in understanding Globus transfer tool (one thing I still struggle with is what to call it...) What I know I still need to understand is its authentication and authorization mechanisms. Of interest (to me at least) was to look at the usage of our Globus endpoints at RAL. 20TB in last 3 months. Now to work out if that is a lot or not compared to other  similar Globus endpoints and or other communities...

All things data transfery for UK ATLAS for 3 months.

As part of ruminations into how UK Tier2 sites for WLCG are going to evolve their storage. I decided to compare and contrast historical network and file usage for UK WLCG sites for  ATLAS VO for a three month period and relate this info to job completion and data deletion rates.

First we have file  I/O associated with Analysis jobs at the sites:

One should note that not all sites run analysis jobs. So I created a similar table for Production Jobs:

 

I then thought would be interesting to take the totals form the above two tables and see how they relate to the number of Jobs on computer resources. ( I am pretending that thee is no access from non-local worker nodes.

Below is a comparison Total WAN and LAN  data transfers rates with the Size of Storage Element with the total Volume of data which has been deleted from the SE (useful if you are interested in churn rates on your SE):

Caveat on all this info is that it is very much a measure of the VO usage at sites and not necessarily the capability of a site.

What rates can we get for single file transfers?

Recently has a conversation regarding what the expected rates we can see for single file data transfers; so I went to have a look... For recalling a single file 275GB ( we have then at the Tie1 one for some VOs) I got the following results. These are all just examples and have no statiscal basis to them. but as a first step it gives interesting results. When recalling back for our tape system i get the following graph showing over 300MB/s:

Of course, I am also interested to see what happens when I copy a file into castor.The floowing is an example of a similar 275Gb file being copied across the WAN  then is written to tape. As you can see the intial write phase (~75MB/s) is lower than the rate for the file to be written to tape (~260MB/s)    Copy across the network for similar size file and then uploading into Castor:

 
 The end of the log file for this transfer is shown here:

N.B This transfer was using four concurrent stream within the gsiftp transfer.

It is (I think) interesting to look at the theoretical rate limit for transfers for a single stream between the two hosts in this transfer using some predictions; (from website https://www.switch.ch/network/tools/tcp_throughput/) :

So we may need to work on this...
The higher level monitoring worried me for this file until I realised the display options greatly effect the perceived rate. In this (in what might be an atypical) example; by solely changing the bin size I was able to change the perceived rate from 75MB/s to 460MB/s as seen in the two pictures below:

So I will rely on log file info from now on... (well mainly)

Now my site has the advantage (hindrance) of two storage systems  using different hardware configurations and  separate implementation of a gsiftp server; so I decided to see what rate I could get between the two.... And I managed over 100MB/s. This 100MB/s was for a 5 section poll time in the transfer as seen here:

What I also find interesting is that there seems to be a systematic difference between the average rate for transfers depending on transfer direction. (70-80MB/s one direction 90-95MB/s in reverse.) whether this difference is worth investigating is a question I will leave to the reader to decide. Also of interest may be to see what the effect of changing data transfer protocol has; but that it for another day...