Update on the classic SE

As reported at last weeks WLCG operations meeting, the classic SE is now completely frozen (not that it affects any GridPP sites since we are all SRM-ified). There are a few points to note about this:

Are there things not in the classic SE that people want added? Any such features should already be in DPM so this is an acceptable upgrade.

Are there features in the classic SE that are not available in DPM or dCache? The obvious answer is real POSIX mounting of the file system probably via NFS. Both DPM and dCache are looking at supporting NFSv4, there are comments from both of these that NFSv4 might be available by end of the year or sooner.

Will the classic SE be included in the upcoming gLite release? The answer is yes, it will appear in gLite 3.1. Once there is a DPM with NFSv4 support then this will be re-evaluated.

Storage security service challenge

WLCG are asking each ROC to run a security service challenge their sites. Someone (probably Alessandra) will submit a job to each site which will attempt to write a file to the local SE, read it back, copy it to a remote SE, delete the file... Once complete, the submitter will issue a GGUS ticket against the site, asking them to provide information on which operations were performed on the file. You can see an example of what is expected here:

https://gus.fzk.de/pages/ticket_details.php?ticket=22012

The aim of this testing is to determine if SEs record sufficient information for tracing user operations and also to check that site admins are able to gather that information. I am currently putting together some scripts that will perform the querying and parsing of DPM/dCache databases and log files in order to gather the information.

In addition to going through the SE files, it is likely that sites will have to parse the PBS logs on the CE to determine the UI that was originally used for the job submission.

DPM 1.6.4 and SL4

I have just come across a problem with running DPM v1.6.4 on SL4. Well, it's not actually a problem with DPM itself, but rather a problem with the BDII that it is now using as an information provider. SL4 comes with openldap v2.2 which appears to have stricter schema checking than openldap v2.0 (which comes with SL3). This causes problems like this:

$ ldapsearch -LLL -x -H ldap://wn4.epcc.ed.ac.uk:2170 -b mds-vo-name=resource,o=grid
Invalid DN syntax (34)
Additional information: invalid DN

Meaning that your SE can't publish anything about itself. This can be resolved by adding this block of code

attributetype ( 1.3.6.1.4.1.3536.2.6.1.4.0.1
NAME 'Mds-Vo-name'
DESC 'Locally unique VO name'
EQUALITY caseIgnoreMatch
ORDERING caseIgnoreOrderingMatch
SUBSTR caseIgnoreSubstringsMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.44
SINGLE-VALUE
)

to /opt/glue/schema/ldap/Glue-CORE.schema and then restarting the ldap and bdii processes. This is covered by this bug:

https://savannah.cern.ch/bugs/index.php?15532

DPM 1.6.4 released (with a few problems)

DPM v1.6.4 was released into production this week. First of all, there are a few points to be aware of:

1. This release requires an update of the v1.6.3 DB schema. **YAIM will take care of this for you**. It is not necessary to run the DB migration script by hand.

2. Two new YAIM variables, DPM_DB and DPNS_DB, are introduced.

3. After the reconfiguration, DPM will use the BDII as an information provider instead of Globus MDS. By default the BDII runs on port 2170 whereas globus-mds was on 2135. You need to change the site-info.def variable to this (so that the site BDII looks in the right place)

BDII_SE_URL="ldap://$DPM_HOST:2170/mds-vo-name=resource,o=grid"

4. YAIM does some tweaking of the /etc/sysctl.conf values. The old values are copied to /etc/sysctl.conf.orig if you want to reinstate them.

However, once the release was announced, a couple of problems soon reared their heads:

a) Sites were recommended not to upgrade due to problem left over from the build

http://glite.web.cern.ch/glite/packages/R3.0/updates.asp

For sites who had already upgraded, the fix was this:

   mkdir -p /home/glbuild/GLITE_3_0_3_RC1_DATA/stage/etc
   ln -s /opt/lcg/etc/lcgdm-mapfile \
 /home/glbuild/GLITE_3_0_3_RC1_DATA/stage/etc

b) With the latest update the info provider of the DPM machines has changed from MDS to BDII. However the YAIM ( -15) coming with the update does not configures edguser's certificate.

The fix was to perform these steps manually:

mkdir -p ~edguser/.globus
chown edguser:edguser ~edguser/.globus
cp /etc/grid-security/hostcert.pem ~edguser/.globus/usercert.pem
cp /etc/grid-security/hostkey.pem ~edguser/.globus/userkey.pem
chown edguser:edguser /home/edguser/.globus/user*
chmod 400 /home/edguser/.globus/userkey.pem

Obviously the certification testing isn't quite as water-tight as we would hope.

Manchester Tier2 dcache goes resilient II

Yesterday we completed the scheduled downtime, and now dcache02 is up and resilient, it's still chewing through the list of files and making copies of them, going by past experience it will probably finish somewhere around lunchtime tomorrow. It's so nice to know we're not in the dark ages of dcache-1.6.6 any more. Of course, there's still small niggles to iron out and we've yet to really throw a big load at it, but it's looking a lot, lot better.

DPM 1.6.4-3 on PPS

DPM v1.6.4-3 is now available on the PPS. I would imagine that it will move into production in the next couple of weeks. This version requires a schema change to the dpm_db (3.0.0 -> 3.1.0). YAIM will take of this for you, although a DB backup is recommended beforehand.

We have now moved to YAIM 3.0.1-15, so the installation and configuration steps now look like:

$ /opt/glite/bin/yaim -i -s /opt/glite/yaim/etc/site-info.def -n glite-SE_dpm_mysql

$ /opt/glite/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n SE_dpm_mysql

https://savannah.cern.ch/patch/index.php?1121

The information provider plugin is still the *old* one (which does not account for the used space properly). Therefore you will need to install Graeme's new one by hand (again).

http://www.gridpp.ac.uk/wiki/DPM_Information_Publishing#Beta_Release_Plugin

With this version of DPM there is a BDII process on port 2170 that is used to provide the information about the DPM. This replaces globus-mds as the information provider which ran on port 2135.

This version of YAIM includes the some /etc/sysctl.conf tweaks in the config_DPM_disk function. This is nice (since it could lead to some optimisations) but I think sites should be warned about this beforehand and be allowed to turn off these changes:

https://gus.fzk.de/pages/ticket_details.php?ticket=21713

Anyway, I upgraded from v1.6.3 to v1.6.4 today (on SL4 32bit). No problems so far,  but I will let you know if anything comes up.

ZFS performance on RAID

http://milek.blogspot.com/2007/04/hw-raid-vs-zfs-software-raid-part-iii.html

Manchester Tier2 dcache goes resilient

We're half way through the combined upgrade from dcache-1.6.6-vanilla to dcache1.7.0-with-replica-manager, so far only one of the two head-nodes has been upgraded, but so far so good, the other is scheduled for upgrade next week, and I appear to have scheduled the queue shutdown at 8am on bank-holiday Monday! Documentation will obviously follow including cfengine snippets for those people that love it.

Storage meeting: Wednesday 2nd May

Agenda is here:



http://indico.cern.ch/conferenceDisplay.py?confId=15668

Powered by ScribeFire.

Video of DPM and SRM Presentations at HEPiX

Starring Mr Steve Traylen, including Video download! See Steve's Blog for the links.

Storage talks at HEPiX

There are lots of storage related presentations at HEPiX today:

https://indico.desy.de/conferenceTimeTable.py?confId=257&showDate=25-April-2007&showSession=all&detailLevel=contribution&viewMode=parallel

Of particular relevance to WLCG are the presentations on dCache, DPM and SRM. There are also talks about data corruption and distributed filesystems (GPFS, Lustre,...).

Should make for some interesting reading.

dCache, DPM and SRM2.2

As most of you know, the LCG experiments are requiring that all storage be accessible via the SRM2.2 interface. The current version of dCache, v1.7.0, only provides the SRM1 interface (and an incomplete SRM2.2). Full SRM2.2 support will only be available in the v1.8.0 branch of dCache. Once v1.8.0 has been fully tested and moves into production, all GridPP dCache sites will have to upgrade.

As I understand the situation, no upgrade path between v1.7.0 and v1.8.0 is planned. Sites will first have to upgrade to v1.7.1 and then move onto v1.8.0. The plan is such that v1.7.1 will contain the same code as v1.8.0, minus the SRM2.2 stuff.

Obviously all dCache sites will want to ensure that there is a stable version of the system, particularly as all sites now have 10's of TBs of experiment data on disk. The SRM2.2 bits of v1.8.0 are currently being tested. Once v1.7.1 is released we can test out the upgrade path before giving the nod to sites. There will be some additional complexity when it comes to setting up the space reservation parts of SRM2.2 in your dCache. Documentation will be available when sites have to perform this additional configuration step. In fact, all of this configuration may go into YAIM.

The situation for DPM is slightly simpler. Full SRM2.2 support exists in v1.6.4 which is going through certification at the moment (1.6.3 is the current production version). Again, there will be some additional complexity in configuring the SRM2.2 spaces, but this will be documented.

Even once the SRM2.2 endpoints are available, it is likely that the SRM1 endpoint (running simultaneously on the same host) will continue to be used by the experiments until SRM2.2 becomes widely deployed and the client tools start using it as the default interface for file access.

PostgreSQL housekeeping

The Edinburgh dCache recently started to show increased CPU usage (a few days after an upgrade to 1.7.0-34) as shown in the top plot. The culprit was a postgres process:

$ ps aux|grep 4419
postgres 4419 24.8 0.6 20632 12564 ? R Mar27 6091:11 postgres: pnfsserver atlas [local] PARSE

After performing a VACUUM ANALYSE on the atlas database (in fact, on the all of the databases), the CPU usage dropped back to normal, as can be seen in the bottom. I had thought auto-vacuuming was enabled by default in v8.1 of postgres, but I was mistaken. This has now been enabled by modifying the relevant entries in postgresql.conf.

stats_start_collector = on
stats_row_level = on
autovacuum = on

I also changed these parameters after the VACUUM process sent out a notice:

max_fsm_pages = 300000 # min max_fsm_relations*16, 6 bytes each
max_fsm_relations = 2000 # min 100, ~70 bytes each

The server requires a restart after modifying the last two parameters.

Repository got lost

Recently the Edinburgh dCache has been failing. The usageInfo page was reporting

[99] Repository got lost

for all of the pools. This has been seen before, but only now do I understand why.

The dCache developers have added a process that runs in the background and periodically tries to touch a file on each of the pools. If this process fails, something is regarded as being wrong and the above message is generated. This could happen if there was a problem with the filesystem or a disk was slow to respond for some reason.

Edinburgh was being hit with this issue due to some of the disks pools being completely full, i.e., df was reporting 0 free space, while dCache still thought there was a small amount of space available. This mismatch seems to arise from the presence of the small control files on each dCache pool (these contain metadata information). Each file may take up an entire block on the disk without actually using up all of the space. I'm still trying to find out if dCache performs a stat() call on these files. It should also be noted that dCache has to read each of these control files at pool startup, so a full pool takes longer to come online than one that is empty.

There also appears to be a bug in this background process since all of the Edinburgh disk pools were reporting the error, even though some of them were empty. In the meantime, I have set the full pools to readonly and this appears to have prevented the problem reoccurring.

dCache v1.8.0 BETA released

dCache v1.8.0 beta is now available for download:

http://www.dcache.org/downloads/1.8.0/index.shtml

As the page states, this includes the required SRM v2.2 stuff for WLCG but it is *NOT FOR PRODUCTION*. Sites should only upgrade once sufficient testing has been completed. If you would like to try it out as a test then feel free. I'm sure the dCache team would appreciate any feedback.

YAIM does no yet support the configuration of SRM 2.2 spaces and space reservation, this has to be done by hand and requires the use of a new concept of link groups. You already had pools and pool groups, units and ugroups, well now you've got links and link groups. More information is here:

https://srm.fnal.gov/twiki/bin/view/SrmProject/SrmSpaceReservation

Again, no dCache in the UK should upgrade to this version yet.

dCache-NGDF workshop

Colin Morey and myself attended the dCache-NGDF workshop last week in Copenhagen (hence the little mermaid). The dCache developers presented lots of useful information. The presentations can all be found here.

NGDF are committing effort to dCache development as they plan to use dCache to create a distributed Tier-1 with the "head" nodes based in Copenhagen (where the network fibres come in) and gridftp doors and pool nodes spread across mulitple sites in Scandinavia. It looks to be quite an ambitious project, but they have already started making changes to the dCache GridFTP code in order to get round the problem of a file transfer first getting routed through a door node before ending up on the destination pool. This might be OK within a site, but it becomes more of a problem when the transfer is over the WAN. Another solution to this problem involves adopting the GridFTP v2 protocol (which has not yet been adopted by Globus). It appears that both approaches will be developed.


Another interesting bit of news regards new releases of dCache. All of the SRM 2.2 stuff that is required by WLCG will come with v1.8 (sometime this month). At the same time, v1.7.1 will be released which will contain all of the same code as v1.8
other than the SRM 2.2 stuff. It would appear that they want to retain a "stable" version of dCache at all times, in addition to a version that is required by sites supporting WLCG VOs. While sensible, it doesn't inspire huge confidence in the SRM 2.2 code. In summary, all of the GridPP sites will have to upgrade to v1.8.

One last thing about 1.8 is that although it will support all of the SRM 2.2 concepts such as space reservation and storage classes (T0D1), it will not come with a suitable generic information provider (GIP) to publish this information to the BDII. GridPP may have to lend some effort in order to get this fixed.

DPM srmPutDone Errors Understood

As usual the response of the DPM team to the report of srmPutDone errors was excellent.

There is a long term fix in the pipeline and a short term work around has been found: increase the maximum idle timeout in MySQL.

See the full posting over on the scotgrid blog for details.

dCache 1.7.0-33 released

Most of you will have seen this already, but v1.7.0-33 of the dCache server available which should fix the data corruption problem that was announced on the user-forum list. Sites are recommended to install as soon as possible. Get in contact if there are any problems.

Storage accounting project page

I've created a project page for the storage accounting system. This will
allow Dave Kant and myself to better keep track of the open issues.

http://savannah.cern.ch/projects/storage-account/

Could all sites continue to check the published numbers and report any
inconsistencies to me.

As usual, the storage accounting page can be found here:

http://goc02.grid-support.ac.uk/storage-accounting/view.php?queryType=storage

Suggestions for improvements are welcome.

DPM 1.6.4 Tagged

DPM version 1.6.4 has been tagged. The major change here is that support for secondary groups has been added. See the savannah patch for details.

Unfortunately there's another schema change in the offing. Details are in the twiki.