Learning about Globus Connect/Online and more when The globusWorld Tour came to RAL

RAL recently hosted a Globus World Tour event where ~50 researchers and system administrators from across the UK (and across research interest groups) gathered to learn about Globus Online.
A link to the agenda is here:
https://www.globusworld.org/tour/program?c=16

The sessions covered various areas from server system administration, transfer usage and monitoring, API/CLI access and app integration. As was pointed out, globus do more than just data transfer.  (I got use Jupyter notebooks again :) ) What I also found helpful was trying to understand tokens as authentication method. (Although most is still just a mystery to me.)
Useful pages to look at  are the globus how-to page:

 https://docs.globus.org/how-to/

 and the github repository:

https://github.com/globus

Time to go and see if I can setup a globus connect server on a linux box with no gui or browser.....

This is Dave; First of the ATLAS datasets at RAL-LCG2, signing off.

The last physics beams and collisions in the LHC for the foreseeable future finished last month. This has been described better than I can do it here:
https://www.symmetrymagazine.org/article/lhc-ends-second-season-of-data-taking
 I thought I would use this moment to finish my description. I and my children are still in 43 rooms spread over 24 houses. We are in Switzerland, USA, Germany, UK, France, Portugal, Netherlands, Japan,Turkey, Canada, Italy, Israel and Czech Republic. There are 268datasets of which 1 has three replicas , 56 have a twin, the remaining 211 are at risk of extinction if a room were to be destroyed. Only 54 datatsets are user derived rather than being centrally produced.

In general,  since my birth neatly nine years ago,  the number of primary brethren at my initial home of the RAL tape system.may also be of interest to people.
In the "DATATAPE" room of RAL-LCG2 there are 17855 people. It's amazing how we can fit all in one room!  4865 of these are unique data runs.(If you recall that my DNA value is 2.3.7.3623, then the last value at RAL is 2.3.3.20441). There are 7628774 files filling a space of 7.294 PB of RAW data  (out of a total 9.925PB)

Below is a plot of the distribution of dataset size (GB)  stored for data runs:


More datasets will be coming later (though still to be confirmed when.)  Data rates will be much larger and so greater volumes of data will need to be recorded. It's going to be an interesting couple of years working out how to do this and a challenge to prepare for this, whilst still providing  functionality and capacity to provide my current and future children to the Users who will continue to analyse me and my brethren in the coming years. This is Dave; first (but no last ) of the ATLAS raw datasets stored at RAL-LCG2, signing off.

Pretrospective

Traditionally at the end of the year it falls to the undersigned to look back at highlights of the past year and forward to issues in the coming year.  By which we mean technical issues, not political, economic, or peopleoligical - no mention of the B-word or the impact it has had on us already.

Looking back, without the benefit of the archive (an internal network error has, ironically, taken it temporarily offline), one of the significant developments of 2018 was the formation of what is now known as IRIS, an infrastructure for an amalgamation of STFC-funded research communities.  GridPP, of course, is a part of this, an example of how to provide an infrastructure even if the solution would not work directly for all user communities.  Also highlightworthy, the work on object stores, and ZFS.

Looking forward, expanding the work with IRIS will be interesting, in particular the competition for the middleware on top of the IaaS and storage, because the current infrastructure requires a fair bit of IaaS expertise.  Less new, but still not sufficiently followed up on, are opportunities to work on the integration of data centre automation and information systems, although even some storage vendors at CIUK knew little about this, and WLCG itself seems to have given up on this angle.  In fact, looking further back to the early naughties it is remarkable how often the wheels get reinvented, thing B gets replaced with thing C that is a reinvention of thing A which was originally replaced by B.

Coming back to 2019, more pertinent still for the GridPP T2s will be the ongoing evolution of sites to storage sites and cache-only sites, a process which continues to pose lots of interesting technical challenges.

When did we switch storage systems.... I'm not sure I can tell.....

So as part of the Storage evolution within the UK, the Birmingham site is switching SEs from DPM to EOS for the ATLAS experiment. However this blog isn't about that, this blog is about what is happening in the meantime... Similar to the way in which we pointed UCL WNs to use QMUL storage in London, ATLAS have switched the 'Brum' WNs to use storage at Manchester. This "greatly" increases the rtt between WN and SE from ~0.1ms to ~4ms. So I got to wandering, is this increase in latency noticeable in Job efficiency for the VOs. Here are the graphs from the VO for Birmingham and Manchester for the last month..
For Manchester a a control set for SE performance we have:

And for Birmingham we have:

I leave it to the reader to come to conclusions about wether Birmingham efficiencies have dropped since changing SEs. (Even if that conclusion is "you need more data to make a proper comparison",) but at least its something to get things going...

40/100G networks are here! (but also have been for a while...)

So hearing that RAL now has a 100G link into JANET started me thinking again about what kind of network tuning might be needed.. A quick bit of using a popular search engine (and knowledge that their are certain individuals I have found that can provide useful info on this area, I found:

https://fasterdata.es.net/host-tuning/100g-tuning/

https://www.es.net/assets/Uploads/100G-Tuning-TechEx2016.tierney.pdf

What surprised me (and shouldn't have) was how old these posts were! Of particular interest has been that there really is a benefit to get diskservers upgraded to SL/CENTOS 7 rather than 6. I also did not know there was a 2G limit for TCP window size.

Successful DPM/DOME BOF/Hackathon

As part  of a side meeting to the GRIDPP41 meeting held in Ambleside this year, we also had a successful BOF/Hackathon meeting. Now I probably should care more about the WN configuration/security hack session, but that really is SEP to me. However the discussion we had regarding Lancaster's DOME deployment  and DPM legacy mode in general was useful to understand the DPM roadmap and how we are going to roll out SRM less service at  tier 2 sites. This is germane at the moment as some of the features such as 3rd party copy (or TPC as it shall now be known as...) are needed for supporting data activities not just for the WLCG VOs but also some of our smaller communities. (DUNE is an example.)

Help, ZFS ate my storage server (kernel segfaults on SL6)

At Edinburgh our storage test server (sl6) just updated it's kernel and had to reboot. Unfortunately it did not come back and suffered a kernel segfault during the reboot.

This was spotted to be during the filesystem mounting stage in the init scripts and specifically was caused by modprobe-ing the zfs module which had just been built by dkms.

The newer sl6 redhat kernels (2.6.32-754....) appear to have broken part of the abi used by the ZFS modules built by dkms.

The solution to fix this was found to be:

1. Reboot into the old kernel (anything with a version 2.6.32-696... or older)
2. check dkms for builds of the zfs/spl modules:   dkms status
3. run:   dkms uninstall zfs/0.7.9; dkms uninstall spl/0.7.9
4. make sure dkms removed this for ALL kernel versions (if needed run dkms uninstall zfs/0.7.9 -k 2.6.32-754) to remove it for a specific kernel
5. remove all traces of these modules:
    for i in /lib/modules/*; do
  for j in extra weak-updates; do
   for k in avl icp nvpair spl splat unicode zcommon zfs zpios ; do
     rm -r ${i}/${j}/${k};
   done;
  done;
 done
6. reboot back into the new kernel and reinstall zfs:
   dkms install zfs/0.7.9; dkms install spl/0.7.9
7. Check that you've saved everything important.
8. Now load the new modules: modprobe zfs
9. re-import your pools: zpool import -a

Alternatively: Remove all of the zfs modules (steps 3 and 5) before you reboot your system after installing the new kernel and then dkms will re-install everything on the next reboot.

For more info: https://github.com/zfsonlinux/zfs/issues/7704

TL;DR  When building new kernel modules dkms doesn't always rebuild external modules safely, make sure you remove these dependencies when you perform a kernel update so that everything is rebuilt safely

CMS dark data volume after migrtaion to new SE at Tier1 RAL

CMS have recently moved all  full usage of disk only storage from CASTOR to CEPH at the RAL-LCG2 tier1. (They are still using CASTOR for tape services.) CMS had been using an approx 2PB of disk only CASTOR space on dedicated servers. What is of interest is that after CMS had moved their data and deleted the old data from CASTOR, there was still ~ 100TB of dark data left on the SE. WE have now been able to removed this data and have started the process of decommissioning the hardware. (Some hardware may be re-provisioned for other tasks, but most is creaking with old age.)

ATLAS are almost in a similar position. I wander how much dark data will be left for them.. ( Capacity is 3.65PB at the moement so my guess is ~150TB)

When all the dark data is removed , and the files removed form the namespace, we can also clean up the dark directory structure which is no longer needed.  I leave it to the read to guess how many dark directories CMS have left...

DPM Workshop 2018 Report

CESNET hosted the 2018 DPM Workshop in Prague, 31st May to 1st June.

As always, the Workshop was built around the announcement of a new DPM release - 1.10.x - and promotion of the aims and roadmaps of the DPM core development team represented by it.

Since the 1.9.x series, the focus of DPM development has been on the next-generation "DOME" codebase. The 1.10 release, therefore, shows performance improvements in managing requests for all supported transfer protocols - GridFTP, Xroot, HTTP - but only when the DOME adapter is managing them.
(DOME itself is an http/WebDAV based management protocol, implemented as a plugin to xrootd, and directly implementing the dmlite API as an adapter.)

By contrast, the old lcgdm code paths are increasingly obsolesced in 1.10 - the most significant work done on the SRM daemon supported via these paths was the fix to Centos7 SOAP handling*.

As a consequence of this, there was a floated suggestion that SRM (and the rest of the lcgdm legacy codebase for DPM) be marked as "unsupported" from 1 June 2019 - a year after this workshop. There was some lively debate about the consequences of this, and two presentations (from ATLAS and CMS) covering the possibility of using SRMless storage. [In short: this is probably not a problem, for those experiements.]
There was some significant concern mainly about historical dependancies on SRM - both for our transfer orchestration infrastructure, for which non-SRM transfers are less tested, and for historical file catalogues, which may have "srm://" paths embedded in them.

As an additional point, there was a discussion of longstanding configuration "issues" with Xrootd redirection into the CMS AAA hierarchy, as discovered and amended by Andrea Sartirana at the end of 2017.


Other presentations from the contributing regions had a significant focus on testing other new features of DPM in 1.9.x; the distributed site approach (using DOME to manage pool nodes at geographically remote locations relative to the head, securely), and the new "volatile pool" model for lightweight caching in DPM.

For Italy, Alessandra Doria reported on the "distributed" DPM configuration across Roma, Napoli and LNF (Frascati), implemented currently as a testbed. This is an interesting implementation of both distributed DPM, and the volatile pools - each site has both a local permanent storage pool, plus a volatile cache pool, enabling the global namespace across the entire distributed DPM to be transparent (as remote files are cached in the volatile pool from other sites).

For Belle 2, Silvio Pardi reported on some investigations and tests of volatile pools for caching of data for analysis.

We also presented, from the UK, work on implementing the old ARGUS-DPM bridge for DMLITE/DOME. This api bridge allows the ARGUS service - the component of the WLCG site framework which centralises authentication and authorisation decisions - to make user and group ban status available to a local DPM. (DPM, especially in DMLITE and DOME eras, does not perform account mapping in the way that compute elements do, so the most useful part of ARGUS's functionality is the binary authorisation component. As site ARGUS' are federated with the general WLCG ARGUS instance to allow "federated user banning" as a security feature, the ability to set storage policies via the same framework is useful.)

*Centos7 gSOAP libraries changed behaviour such that they handle connection timeouts poorly, resulting in erroneous errors being sent to clients to an SRM when they reopen a connection. A work-around was developed at the CGI-GSOAP level, and deployed initially at UK sites which had noticed the issue.

Impact of Firmware updates of Spectra and Meltdown Mitigation.

In order to address the security issues associated with the Spectra / Meltdown hardware bug found in many modern operating system AND CPUs firmware, CPU microcode updates are required. The microcode updates addresses the Spectre variant 2 attack. Spectre variant 2 attacks work by persuading a processor's branch predictor to make a specific bad prediction about which code will be executed and from which information can be obtained about the process.

Much has been said about the performance impact of Spectra / meltdown mitigation caused by the kernel patches. Less is known about the impact of the firmware updates on system performance. Most of the concern is about the performance impact on processes that switch between user and system calls. These are typically applications that perform disk or network operations.

After one abortive attempt Intel has released a new set of CPU microcode updates that promise to provide stability (https://newsroom.intel.com/wp-content/uploads/sites/11/2018/03/microcode-update-guidance.pdf). We have run some IO intensive benchmarks tests on our servers testing different firmware on our Intel Haswell CPUs (E5 2600 V3).

Our test setup up is made up of 3 HPE DL60 servers each with one OS disk and three data disks (1 TB SATA hard drives). One node is used for control while the other two will be involved in the actual benchmark process. The servers have Intel E5 2650 V3 CPUs and 128GB of RAM. Each server is connected at 10Gb/s SFP+ to a non blocking switch. All system are running scientific linux 6.9 (aka CentOS 6.9) with all the latests updates installed.

The manufacture, HPE, has provided a BIOS update which will deploy this new microbe version and we will investigate the impact of updating the microcode to 0x3C(BIOS 2.52) from previous version 0x3A(2.56) while keeping everything else constant. One nice feature of the HPE servers is the ability to swap to a backup BIOS so updates can be reverted.

Our first test uses a HDFS test called DFSIO with a  Hadoop setup (1 name node, 2 data nodes with 3 data disks each). The test will write 1TB of data across the 6 disks and then reads it back. The command run are

yarn jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-tests.jar TestDFSIO -D

test.build.data=mytestdfsio -write -nrFiles 1000 -fileSize 1000

yarn jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-tests.jar TestDFSIO -D

test.build.data=mytestdfsio -read -nrFiles 1000 -fileSize 1000

The results, in minutes taken, clearly show a major performance impact, of order 20%, in using the new microcode update! 

As a cross check we did a similar test using IOzone. Here we used the distributed mode of IOzone to run tests on the six disks of the two data nodes. The command run was 

iozone -+m clustre.cfg -r 4096k -s 85g -i 0 -i 1 -t 12 1TB, 12 threads, were clustre.cfg defines the nodes and disks used.

The results, in kb/s throughput, again show a measurable impact in performance of using the new firmware, although at a smaller scale (5%).

Instead of using local idisk (direct attached storage) we also did the test over the network, using our Lustre file system instead of the local disks, we saw no performance impact in either test however in this case the 10Gb/s link was a bottle neck and may have influenced the results. We will investigate further as time allows.