Understanding Globus connect/online... is it doing a lot??

I have made further progress in understanding Globus transfer tool (one thing I still struggle with is what to call it...) What I know I still need to understand is its authentication and authorization mechanisms. Of interest (to me at least) was to look at the usage of our Globus endpoints at RAL. 20TB in last 3 months. Now to work out if that is a lot or not compared to other  similar Globus endpoints and or other communities...

Learning about Globus Connect/Online and more when The globusWorld Tour came to RAL

RAL recently hosted a Globus World Tour event where ~50 researchers and system administrators from across the UK (and across research interest groups) gathered to learn about Globus Online.
A link to the agenda is here:
https://www.globusworld.org/tour/program?c=16

The sessions covered various areas from server system administration, transfer usage and monitoring, API/CLI access and app integration. As was pointed out, globus do more than just data transfer.  (I got use Jupyter notebooks again :) ) What I also found helpful was trying to understand tokens as authentication method. (Although most is still just a mystery to me.)
Useful pages to look at  are the globus how-to page:

 https://docs.globus.org/how-to/

 and the github repository:

https://github.com/globus

Time to go and see if I can setup a globus connect server on a linux box with no gui or browser.....

Data rates in the UK for last 12 months: Wow alot of data goes bentween the WNs and SEs...

So with me move to new data and work-flow models as a result of the idea to create further storageless sites and caching sites. I decided to take a look at how large data flows  within the UK. Caveat Emptor: I took this data from ATLAS dashboard and make the assumption that there is little WAN traffic from WNs to SEs. I am aware this is not correct, but is at the moment a small factor for ATLAS. (Hence why I reviewed ATLAS rather than CMS whom I know use AAA alot.)
 

In green are the data volumes in UK disk storage, in red are the rates out of the storage. (In blue is the rate for WAN transfers between UK SEs.) In purple shows the rates to and from the RAL tape system. Of notes is that during the month there was 49.1PB of data deleted from UK storage out of a disk cache of ~33PB. What I note form these rate that the 139PB of data ingest from storage into worker nodes  and the 11.4PB out from the completed jobs is data that would have had to go on the WAN if WNs were not co-located with SEs.

All things data transfery for UK ATLAS for 3 months.

As part of ruminations into how UK Tier2 sites for WLCG are going to evolve their storage. I decided to compare and contrast historical network and file usage for UK WLCG sites for  ATLAS VO for a three month period and relate this info to job completion and data deletion rates.

First we have file  I/O associated with Analysis jobs at the sites:

One should note that not all sites run analysis jobs. So I created a similar table for Production Jobs:

 

I then thought would be interesting to take the totals form the above two tables and see how they relate to the number of Jobs on computer resources. ( I am pretending that thee is no access from non-local worker nodes.

Below is a comparison Total WAN and LAN  data transfers rates with the Size of Storage Element with the total Volume of data which has been deleted from the SE (useful if you are interested in churn rates on your SE):

Caveat on all this info is that it is very much a measure of the VO usage at sites and not necessarily the capability of a site.

What rates can we get for single file transfers?

Recently has a conversation regarding what the expected rates we can see for single file data transfers; so I went to have a look... For recalling a single file 275GB ( we have then at the Tie1 one for some VOs) I got the following results. These are all just examples and have no statiscal basis to them. but as a first step it gives interesting results. When recalling back for our tape system i get the following graph showing over 300MB/s:

Of course, I am also interested to see what happens when I copy a file into castor.The floowing is an example of a similar 275Gb file being copied across the WAN  then is written to tape. As you can see the intial write phase (~75MB/s) is lower than the rate for the file to be written to tape (~260MB/s)    Copy across the network for similar size file and then uploading into Castor:

 
 The end of the log file for this transfer is shown here:

N.B This transfer was using four concurrent stream within the gsiftp transfer.

It is (I think) interesting to look at the theoretical rate limit for transfers for a single stream between the two hosts in this transfer using some predictions; (from website https://www.switch.ch/network/tools/tcp_throughput/) :

So we may need to work on this...
The higher level monitoring worried me for this file until I realised the display options greatly effect the perceived rate. In this (in what might be an atypical) example; by solely changing the bin size I was able to change the perceived rate from 75MB/s to 460MB/s as seen in the two pictures below:

So I will rely on log file info from now on... (well mainly)

Now my site has the advantage (hindrance) of two storage systems  using different hardware configurations and  separate implementation of a gsiftp server; so I decided to see what rate I could get between the two.... And I managed over 100MB/s. This 100MB/s was for a 5 section poll time in the transfer as seen here:

What I also find interesting is that there seems to be a systematic difference between the average rate for transfers depending on transfer direction. (70-80MB/s one direction 90-95MB/s in reverse.) whether this difference is worth investigating is a question I will leave to the reader to decide. Also of interest may be to see what the effect of changing data transfer protocol has; but that it for another day...