Co-evolving data nodes

Bing! a mail comes in from our friends in the States saying look! here's someone in New Zealand who has set up iRODS node to GridFTP data to/from their site. It is a very detailed document yet it looks a lot like the DiRAC/GridPP data node document. They have solved many of the same problems we have solved, independently.

The basic idea is to have a node outside your institute/organisation which can be used to transfer data to/from your datastore/cluster. With a GridFTP endpoint, you could move data with FTS (as we do with DiRAC), people can use Globus (used by STFC's facilities, for example), or data can be moved to/from other e-infrastructures (such as EUDAT's B2STAGE) or EGI. Regardless of the underlying storage, there will be common topics like security, monitoring, performance, how to (or not to) firewall it, how to make it discoverable, etc. It could be the data node in a Science DMZ.

The suggestion is that we (= GridPP, DiRAC, and in fact anyone else who is willing and able) contribute to a detailed writeup which can be published as an OGF document (open access publishing for free!, and because GridFTP is an OGF protocol), either community practice or experiences - and then have a less detailed paper which could be submitted to a conference or published in a journal. 

(Cloud) storage plugfest

A quick note to advertise the coming SNIA/CloudWatch cloud storage "plugfest" - physically in Santa Clara, CA, US, but remote attendance is possible: www.cloudplugfest.org. 19-22 Sep (19-21 is the SNIA storage developer event itself.)

These events are often very interesting, bringing together different components - and standards - and making them work together. Submit your work on any or all of CDMI, OCCI, CIMI, TOSCA, XACML/SAML/X.509.

EUDAT and GridPP

EUDAT2020 (the H2020 follow-up project to EUDAT) just finished its kick-off meeting at CSC. Might be useful to jot down a few thoughts on similarities and differences and such before it is too late.

Both EUDAT and GridPP are - as far as this blog is concerned - data e- (or cyber-) infrastructures. The infrastructure is distributed across sites, sites provide storage capacity or users, there is a common authentication and authorisation scheme, there are data discovery mechanisms, both use GOCDB for service availability.

• EUDAT will be using CDMI as its storage interface - just like EGI does - and CDMI is in many ways fairly SRM-like. We have previously done work comparing the two.
• EUDAT will also be doing HTTP "federations" (i.e. automatic failover when a replica is missing; this is confusingly referred to as "federation" by some people).
• Interoperation with EGI is useful/possible/thought-about (delete as applicable). EUDAT's B2STAGE will be interfacing to EGI - there is already a mailing list for discussions.
• GridPP's (or WLCG's) metadata management is probably a bit too confusing at the moment since there is no single file catalogue 
• B2ACCESS is the authentication and authorisation infrastructure in EUDAT; it could interoperate with GridPP via SARoNGS (ask us at OGF44 where we will also look at AARC's relation to GridPP and EUDAT). Jos tells us that KIT also have a SARoNGS type service.
• Referencing a file is done with a persistent identifier, rather like the LFN (Logical Filename) GridPP used to have.
• "Easy" access via WebDAV is an option for both projects. GlobusOnline is an option (sometimes) for both projects. In fact, B2STAGE is currently using GO, but will also be using FTS.

Using FTS is particularly interesting because it should then be possible to transfer files between EUDAT and GridPP. The differences between the projects are mainly that

• GridPP is more mature - has had 14-15 years now to build its infrastructure; EUDAT is of course a much younger project (but then again, EUDAT is not exactly starting from scratch)
• EUDAT is doing more "dynamic data" where the data might change later. Also looking at more support for the lifecycle.
• EUDAT and GridPP have distinct user communities, to a first approximation at least.
• The middleware is different; GridPP does of course offer compute where EUDAT will offer simpler server-side workflows. GridPP services are more integrated, where in EUDAT the B2 services are more separated (but will be unified by the discovery/lookup service and by B2ACCESS)
• Authorisation mechanisms will be very different (but might hopefully interface to each other; there are plans for this in B2ACCESS).

There is some overlap between data sites in WLCG and those in EUDAT. This could lead to some interesting collaborations and cross-pollinations. Come to OGF44 and the EGI conference and talk to us about it.

Highlights of ISGC 2014

ISGC 2014 is over. Lots of interesting discussions - on the infrastructure end, ASGC developing fanless machine room, interest in (and results on) CEPH and GLUSTER, dCache tutorial, and an hour of code with the DIRAC tutorial.

All countries and regions presented overviews of their work in e-/cyber-Infrastructure.

Interestingly, although this wasn't a HEP conference, practically everyone is doing >0 on LHC, so the LHC really is binding countries and researchers (well, at least physicist and infrastructureists) and e-Infrastructures together (and NRENs). When one day, someone sits down to tally up the benefit and impact of the LHC, this ought to be one of the top ones. The ability to work together and to (mostly) be able to move data to each other, and to trust each other's CAs.

Regarding the DIRAC tutorial, I was there and went through as much as I could ("I am not doing that to my private key")  Something to play with a bit more when I have time - an hour (of code) is not much time; there are always compromises between getting stuff done realistically and cheating in tutorials, but as long as there's something you can take away and play with later. As regards the key shenanigans, DIRAC say they will be working with EGI on SSO, so that's promising. Got the T-shirt, too. "Interware," though?

On the security side, OSG have been interfacing to DigiCert, following the planned termination of the ESNET CA. Once again grids have demands that are not seen in the commercial world, such as the need for bulk certificates (particularly cost effective ones - something a traditional Classic IGTF can do fairly well.) Other security questions (techie acronym alert, until end of paragraph) include how Argus and XACML compare for implementing security policies, and the EMI STS - CERN looking at linking with ADFS. And Malaysia are trialling an online CA based on a FIPS level three token with a Raspberry π.

EGI federated cloud got mentioned quite a few times - KISTI interested in offering IaaS, also Australia interested in joining. Philippines providing resources. EGI have a strategy for engagement. Interesting the extent to which they are driving the of CDMI.

I should mention Shaun gave a talk on "federated" access to data, comparing the protocols - which I missed - the talk, I mean - being in another session, but I understand it was well received and there was a lot of interest.

Software development - interesting experiences from the dCache team and building user communities with (for) DIRAC. How are people taught to develop code? The closing session was by Adam Lyon from Fermilab who talked about the lessons learned - the HEP vision of big data being different from the industry one. And yet HEP needs a culture shift to move away from the not-invented-here.

ISGC really had a great mix of Asian and European countries, as well as the US and Australia. This post was just a quick look through my notes; there'll be much more to pick up and ponder over the coming months. And I haven't even mentioned the actual science stuff ...

dCache workshop at (with) ISGC 2014

Shaun and I took part in the dCache workshop. Starting with a VM with a dCache RPM, the challenge was to set it up with two pools, NFS4, and WebDAV. A second VM got to access the data, mainly via NFS or HTTP(S) - security ranged from IP address to X.509 certificates. The overall impression was that it was pretty easy to get set up and configure the interfaces and get it to do something useful: dCache is not "an SRM" or "an NFS server" but rather storage middleware which provides a wide range of interfaces to storage. One of the things the dCache team is looking into is the cloud interface, via CDMI. This particular interface is not ready (as of March 2014) for production, but it's something we may want to look into and test with the EGI FC's version, Stoxy.