ATLAS Jamboree 2019 view from the offiste perspective.

I didn't go in person to the ATLAS Jamboree this year held at CERN. For those who are allowed to view I suggest looking at: https://indico.cern.ch/event/770307/

But I did join for some via vidyo!
Here is my musings about the talk givens I saw. ( Shame I couldn't get involved in coffee/dinner  discussions which are often the most fruitful moments of these meetings):

Even before the main meeting started, there is an interesting talk regarding HPC data access in US an ANL.


In particular, I  like the thought of globus usage and incorporating rucio into DTNs at the sites.  Similar to what was discussed at other sites at the rucio community workshop last week.

In the preview talk, I picked out the switch to Fast Sim rather than Full sim will increase output rate by a factor of 10. A good reminder that user workflow changes could drastically alter computing requirements.
From the main meeting , the following meetings will be of interest on a data storage:

Data Organization and Management Activities: Third Party Copy and (storage) Quality of Service
TPC: details on DPM

DOMA ACCESS: Caches 

DDM Ops overview

Diskless and lightweight sites: consolidation of storage

Data Carousel

Networking - best practice for sites, and evolution

WLCG evolution strategy

 

One thing it di was cause me to think what if; (and I stress the if is me not ATLAS musing,)  ATLAS  wanted to read 1PB of data a day from Tape at RAL and then distribute it across the world?

 

 

Understanding Globus connect/online... is it doing a lot??

I have made further progress in understanding Globus transfer tool (one thing I still struggle with is what to call it...) What I know I still need to understand is its authentication and authorization mechanisms. Of interest (to me at least) was to look at the usage of our Globus endpoints at RAL. 20TB in last 3 months. Now to work out if that is a lot or not compared to other  similar Globus endpoints and or other communities...

Learning about Globus Connect/Online and more when The globusWorld Tour came to RAL

RAL recently hosted a Globus World Tour event where ~50 researchers and system administrators from across the UK (and across research interest groups) gathered to learn about Globus Online.
A link to the agenda is here:
https://www.globusworld.org/tour/program?c=16

The sessions covered various areas from server system administration, transfer usage and monitoring, API/CLI access and app integration. As was pointed out, globus do more than just data transfer.  (I got use Jupyter notebooks again :) ) What I also found helpful was trying to understand tokens as authentication method. (Although most is still just a mystery to me.)
Useful pages to look at  are the globus how-to page:

 https://docs.globus.org/how-to/

 and the github repository:

https://github.com/globus

Time to go and see if I can setup a globus connect server on a linux box with no gui or browser.....

Big data picture

Not as in ((big data) picture) but (big (data picture)), if that makes sense.

I find myself in Edinburgh - it's been far too long since I last was here, I am embarrassed to say.

We are looking at data movement for EUDAT and PRACE, and by a natural extension (being at EPCC), GridPP and DiRAC. The main common data mover is GridFTP: useful because we can (more or less) all move data with GridFTP, it gets great performance, we know how to tune and monitor it, and it supports third party copying. We also need to see how to bridge in GlobusOnline, with the new delegated credentials. In fact both Contrail and NCSA developed OAuth-delegated certificates (and while the original CILogon work was OAuth1, the new stuff is OAuth2.)

One use case is data sharing (the link is to a nice little video Adam Carter from EPCC showed in the introduction). You might argue that users are not jumping up and down screaming for interdisciplinary collaborations, yet if they were possible they might happen! When data policies require data be made available, as a researcher producing data you really have no choice: your data must be shareable with other communities.