EUDAT and GridPP

EUDAT2020 (the H2020 follow-up project to EUDAT) just finished its kick-off meeting at CSC. Might be useful to jot down a few thoughts on similarities and differences and such before it is too late.

Both EUDAT and GridPP are - as far as this blog is concerned - data e- (or cyber-) infrastructures. The infrastructure is distributed across sites, sites provide storage capacity or users, there is a common authentication and authorisation scheme, there are data discovery mechanisms, both use GOCDB for service availability.

• EUDAT will be using CDMI as its storage interface - just like EGI does - and CDMI is in many ways fairly SRM-like. We have previously done work comparing the two.
• EUDAT will also be doing HTTP "federations" (i.e. automatic failover when a replica is missing; this is confusingly referred to as "federation" by some people).
• Interoperation with EGI is useful/possible/thought-about (delete as applicable). EUDAT's B2STAGE will be interfacing to EGI - there is already a mailing list for discussions.
• GridPP's (or WLCG's) metadata management is probably a bit too confusing at the moment since there is no single file catalogue 
• B2ACCESS is the authentication and authorisation infrastructure in EUDAT; it could interoperate with GridPP via SARoNGS (ask us at OGF44 where we will also look at AARC's relation to GridPP and EUDAT). Jos tells us that KIT also have a SARoNGS type service.
• Referencing a file is done with a persistent identifier, rather like the LFN (Logical Filename) GridPP used to have.
• "Easy" access via WebDAV is an option for both projects. GlobusOnline is an option (sometimes) for both projects. In fact, B2STAGE is currently using GO, but will also be using FTS.

Using FTS is particularly interesting because it should then be possible to transfer files between EUDAT and GridPP. The differences between the projects are mainly that

• GridPP is more mature - has had 14-15 years now to build its infrastructure; EUDAT is of course a much younger project (but then again, EUDAT is not exactly starting from scratch)
• EUDAT is doing more "dynamic data" where the data might change later. Also looking at more support for the lifecycle.
• EUDAT and GridPP have distinct user communities, to a first approximation at least.
• The middleware is different; GridPP does of course offer compute where EUDAT will offer simpler server-side workflows. GridPP services are more integrated, where in EUDAT the B2 services are more separated (but will be unified by the discovery/lookup service and by B2ACCESS)
• Authorisation mechanisms will be very different (but might hopefully interface to each other; there are plans for this in B2ACCESS).

There is some overlap between data sites in WLCG and those in EUDAT. This could lead to some interesting collaborations and cross-pollinations. Come to OGF44 and the EGI conference and talk to us about it.

ISGC 2015 Review and Musings..

The 2015 ISGC Conference is coming to a close; so I thought I would jot down some musings regarding some of the talks I have seen (and presented.) over the last week. Not surprisingly; since the G and C are grids and clouds, a lot of talks were regrading compute, however there were various talks on storage and data management (especially dCache). But most interesting talk was regarding new technology which sees a cpu and network interface incorporated into an individual HDD. this can be seen here:
http://indico3.twgrid.org/indico/contributionDisplay.py?sessionId=26&contribId=80&confId=593

There were also many site discussion from the various asian countries represented, of which network setup and storage was on particular interest (also including using infiniband between Singapore Seattle and Australia.) My perfSONAR talk seem to be well received.  It makes the distance our european dataflows have to travel seem trivial.

It was also interesting to listen to some of the Humanities and Arts themed talks. (First time I have ever heard post- modernism used at a conference!!) Their data volume may well be smaller than WLCG VOS;  but still complex and uses interesting visualisation methods.

Some thoughts on data in the cloud gathered at CloudScape VII

Some if-not-quite-live then certainly not at all dead notes from #CloudScapeVII on data in the cloud.

• How to establish trust in the cloud data centre?  Clouds can run pretty good security, which you'd otherwise only get in the large data centre.
• Clouds can build trust by disclosing processes and practices - Rüdiger Dorn, Microsoft
• Clarify responsibilities
• "35% of security breaches are due to stupid things" - like leaving memory sticks on a train or sending CDs by post... - Giorgio Aprile, AON
• Difficulty to inculcate good (security) practice in many end users
• "Opportunity to make big data available in cloud" - Robert Jenkins, CloudSigma
• Model assumes that end users pay for the ongoing use of data
• Democratise data
• Data protection
• Kuan Hon from QMUL instilled the fear of data protection in everyone that provides data storage. The new data protection stuff doesn't seem to take clouds into accounts - lots of scary implications. [Good thing we are not storing personal data on the grid...]
• Protection relies on legal frameworks - sign a contract saying you won't reveal the data - rather than technology (encrypt it to preventing your revealing the data)
• Joe Baguley from vmware talked about the abstractions: where RAID abstracted harddrives from storage, we now do lots more abstractions with hypervisors, containers, software-defined-X, etc.
• Layers can optimise, so can get excellent performance
• Stack can be hard to debug when something doesn't work so well...
• Generally more benefits than drawbacks, so a Good Thing™
• Overall, speed up data → analysis → app → data → analysis → app → ... cycle
• "What's hot in the cloud" - panel of John Higgings (DigitalEurope), Joe Baguley (vmware), David Bernstein (Cloud Strategy Partners), Monique Morrow (CISCO)
• Big data is also fast data (support for more Vs), lots of opportunities for in memory processing
• Data - use case for predictive analysis and pattern recognition (and in general machine learning)
• devops needed to break down barriers [as we know quite well from the grid where we have tb-support née dteam]
• Disruptive technological advances to, er, disrupt?
• Many end users are using clouds without knowing it -like people using facebook.

Hope I've done it some justice. As always, lots of very interesting things in CloudScape even for those of us who have been providing "cloud" services (in some sense) for a while. Also good of course to catch up with old friends and meeting new ones.

Storage accounting revisited?

One of the basic features of containers - a thing which can contain something - is that you can see how full it is. If your container happens to be a grid storage element, monitoring information is available in gstat and in our status dashboard. The BDII information system publishes data, and so does the SRM (the storage element control interface), and the larger experiments at least track how much they write.

So what happens if all these measures don't agree?  We had a ticket against RAL querying why the BDII published different values from what the experiment thought they had written. It turned out to be partly because someone was attempting to count used space by space token, which leads to quite the wrong results:

Leaving aside whether these should be the correct mappings for ATLAS, the space tokens on the left do not map one-to-one to the actual storage areas (SAs) in the middle (and in general there are SAs without space tokens pointing to them). Note also that the SAs split the accounting data of the disk pools (online storage) so that the sum of the values are the same -- to avoid double counting.

The other reason for the discrepancy was the treatment of read-only servers: these are published as used space by the SRM, but not by the BDII. This is because the BDII is required to be compliant with the installed capacity agreement from a working group from 2008. The document says on p.33,

TotalOnlineSize (in GB=10⁹) is the total online [..] size available at a given moment (it SHOULD not [sic] include broken disk servers, draining pools, etc.)

RAL uses read only disk pools essentially like draining disk pools (unlike tapes, where a read only tape is perfectly readable), so read only disk pools do not count in the total -- they do, however, count as "reserved" as specified in the same document (the GLUE schema probably intended reserved to be more like SRM's reserved, but the WLCG document interprets the field as "allocated somewhere."

Interestingly, RAL does not comply with the installed capacity document in publishing UseddOnlineSize for tape areas. The document specifies

UsedOnlineSize (in GB=10⁹ bytes) is the space occupied by available and accessible files that are not candidates for garbage collection.

It then kind of contradicts itself in the same paragraph, saying

For CASTOR, since all files in T1D0 are candidates for garbage collection, it has been agreed that in this case UsedOnlineSize is equal to [..] TotalOnlineSize.

If we published like this, the used online size would always equal the total size, and the free size would always be zero (because the document also requires that used and free sum to total -- which doesn't always make sense either, but that is a different story.)

OK, so what might we have learnt today about storage accounting?

1. Storage accounting is always tricky: there are all sorts of funny boundary cases, like candidates for deletion, temporary replicas, scratch space, etc.
2. Aggregating accounting data across sites only makes sense if they all publish in the same way: they use the same attributes for the same types of values, etc. However, the supported storage elements all vary somewhat in how they treat storage internally.
3. Before making use of the numbers, it is useful to have some sort of understanding of how they are generated (what do space tokens do? if numbers are the same for two SAs, is it because they are counting the same area twice, or because they split it 50/50? Implementers should document this and keep the documentation up to date!)
4. There should probably be a time to review these agreements - what is the use of publishing information if it does not tell people what they want to know?
5. Storage accounting is non-trivial... getting it right vs useful vs achievable is a bit of a balancing act.