Storage accounting in OSG and OGF

Groups like UR are getting around to discussing storage records. OSG already create storage records: they have XML-formatted records for both the transfer and the file history. (With thanks to Steve Timm from FNAL.)

<StorageElementRecord xmlns:urwg="http://www.gridforum.org/2003/ur-wg">

<RecordIdentity urwg:createTime="2011-07-17T21:18:07Z" urwg:recordId="head01.aglt2.org:544527.26"/>

<UniqueID>AGLT2_SE:Pool:umfs18_3</UniqueID>

<MeasurementType>raw</MeasurementType>

<StorageType>disk</StorageType>

<TotalSpace>25993562993750</TotalSpace>

<FreeSpace>6130300894785</FreeSpace>

<UsedSpace>19863262098965</UsedSpace>

<Timestamp>2011-07-17T21:18:02Z</Timestamp>

<ProbeName>dcache-storage:head01.aglt2.org</ProbeName>

<SiteName>AGLT2_SE</SiteName>

<Grid>OSG</Grid>

</StorageElementRecord>

Over in GLUE-land, the GLUE group insist that using the GLUE schema to publish accounting data - and indeed to use GLUE data for anything other than resource selection - "cannot be done." Unfortunately the chairs didn't make it to OGF, but next steps will include work on the XML rendering of GLUE 2.0, along with the implementations.

Meanwhile, back home in GridPP-land, we use GLUE 1.3 for dynamic data. The question is still mainly about the accuracy (and freshness) of the information published: e.g. temporary copies on disk, files being "deleted" from tape, etc, how these should affect the published dynamic data. As we now have "accurate" tape accounting, the information provider should be updated soon.

OSG storage group

I was in a meeting yesterday with the OSG data storage and management group. I had realised that there was some activity going on in the US regarding this, but didn't know just how many people are involved, or what precisely they were doing. Turns out that there is actually quite a few FTEs looking at dCache 1.8 testing, client tools, deployment scripts, bug reports and support. There is clearly quite an overlap with what we try to do in GridPP, although there are a couple of differences:

1. GridPP has only me (and part of Jens) dedicated to storage. The rest comes from the community of site admins that have signed up to the mailing list. This works well, but it would be good if there were others who could dedicate more time to looking at storage issues. There were previously a couple of positions at RAL for this stuff. Hopefully there will be again in GridPP3.

2. Unlike GridPP, OSG is not dedicated to particle physics, hence they support other user communities. It appears that USATLAS and USCMS take on the lions share of support for storage at US Tier-2 sites.

You can find more information here:

https://twiki.grid.iu.edu/twiki/bin/view/Storage/