DPM admin tools

I have started to put together a set of scripts for performing common (and possibly not-so-common) administration tasks with DPM. This uses the python interface provided by the DPM-interfaces package. It's definitely a work in progress, but you can you can checkout the latest set of scripts from here (assuming you have a CERN account):

isscvs.cern.ch:/local/reps/lhcbedinburgh/User/gcowan/storage/dpm/admin-tools

If not, then I have put them in a tarball here.

They are very much a work in progress, so things will be changing over the next few weeks (i.e., keep a lookout for changes). As always, contributions are welcome. I should probably package this stuff up as an rpm, but haven't got round to that yet. I should also put them into the sys-admin repository and write some documentation (other than the -h option). Like I say, contributions are always welcome...

CCRC May'08 data transfers

The plots above shows the slow ramp up in data transfers being run by the experiments during the May phase of CCRC. Clearly CMS are dominating proceedings at the moment. It's not clear what has happened to ATLAS after an initial spurt of activity. Hopefully they get things going soon, otherwise the Common elememt of CCRC might not be achieved. What is good is that a wide variety of sites are involved. You can even see the new Edinburgh site in there (although it is coming up as unregistered in GridView).

I'll take this opportunity to remind everyone of the base version of the gLite middleware that sites participating in the CCRC exercise are expected to be running. Have a look at the list here.

Grid storage not working?

Well, going by what I heard last week at LHCb software week, I think the answer to this question is "No". The majority of the week focussed on all the cool new changes to the core LHCb software and improvements to the HLT, but there was an interesting session on Wednesday afternoon covering CCRC and more general LHCb computing operations. The point was made in 3 (yes, 3!) separate talks that LHCb continue to be plagued with storage problems which prevent their production and reconstruction jobs from successfully completing. The main issue is the instability of using local POSIX-like protocols to remotely open files on the grid SE from jobs running on the site WNs. From my understanding, this issue could broadly be separated into two categories:

1. Many of the servers being used have been configured in such a way that if a job held a file in an open state for longer than (say) 1 day, the connection was being dropped, causing the entire job to fail.

2. Sites have been running POSIX-like access serices on the same hosts that are providing the SRM. This isn't wrong, but is definitely not recommended due to the load on the system. Anyway, the real problem comes when the SRM has to be restarted for some reason (most likely an upgrade) and the site(s) appear to have just been restarting all services on the node which again resulted in any open file connections being dropped and jobs subsequently failing. I thought it was basic knowledge that everyone knew about, but apparently I was wrong.

LHCb seem to be particularly vulnerable as they have long running reconstruction jobs (>33 hours),resulting in low job efficiency when the above problems rear their ugly heads. I would be interested in comments from other experiments on these observations. Anyway, the upshot of this is that LHCb are now considering on copying data files locally prior to starting their reconstruction jobs. This won't be possible for user analysis jobs, which will be accessing events from a large number of files. Copying all of these locally isn't all that efficient, nor do you know a priori how much local space the WN has available.

xrootd was also proposed as an alternative solution. Certainly dCache, CASTOR and DPM all now provide an implementation of the xrootd protocol in addition to native dcap/rfio, so getting it deployed at sites would be relatively trivial (some places already have it available for ALICE). I don't know enough about xrootd to comment, but I'm sure if properly configured it would be able to deal with case 1 above. Case 2 is a different matter entirely... It should be noted (perhaps celebrated?) that none of the above problems have to do with SRM2.2.

Of course, LHCb only require disk at Tier-1s, so none of this applies to Tier-2 sites. Also, they reported that they saw no problems at RAL: well done guys!

In addition, the computing team have completed a large part of the stripping that the physics planning group have asked for (but this isn't really storage related).

Visualising FTS transfers

As always, monitoring is a hot topic. Sites, experiments and operations people all want to know what Grid services are doing and how this is impacting on their work. In particular, it is clear from todays GDB that monitoring of the FTS is important. The above graph shows the output of a script which I have put together over the past day which queries the RAL FTS service for information about its channels, their transfer rates and information about the number of jobs that are scheduled to be transferred. In fact, I don't query the FTS directly, but use some CGI that Matt Hodges at RAL set up (thanks Matt!). It's a prototype at the moment, but I think it could be one useful way of looking at the data and getting an overview of the state of transfers.

You can see the latest plots for the WLCG VOs here. They are updated hourly. I still need to play about with the colours to try and improve the visual effect. It would be great if the line thickness could be varied with transfer rate, but I don't think GraphViz/pydot can do that.

dCache configuration, graphviz style

I don't know about anyone else, but I'm fed up having to try and debug different site's PoolManager.conf files, especially with all this LinkGroup stuff going on. I find it too too hard to manually parse a file when it stretches to 100's of lines, making it virtually impossible to know if there are any mistakes.

In an effort to try and improve the situation, I put together a little python script last night that converts a PoolManager.conf into a .dot file. This can then be processed by GraphViz to produce a structured graph of the dCache configuration. You can see some examples of currently active dCache configurations here. The above plot shows the config at Edinburgh.

I have been creating both directional (dot) and undirectional (neato) graphs. At the moment, the most useful one is the dot plot. I'm still exploring what neato can be used for.

I think the fact that we even have to consider looking at things this way tells you two things:

1. dCache is a complex beast, with a multitude of different ways of setting things up (which has both pros and cons).
2. The basic configuration really has to be improved to save multiple man-hours that are spent across the Grid trying to debug basic problems.

At the moment, this system is only a prototype. It is intended as an aide to understanding dCache configuration and looking for potential bugs. As always, comments are welcome.

PS Thanks to Steve T for inspiring me to work on this following his graphing glue project.

The CASTOR beaver

It's not often in my work that I get to talk about beavers, but I figured this was appropriate for the storage blog. I discovered yesterday that Castor is the single remaining genus of the family Castoridae, of which the beaver is a member. I think this explains the logo (both new and old).

Upcoming Software

Not strictly GridPP but storage related. Discovered in a recent LWN Article some info about Coherent Remote File System (CRFS) as yet another possible NFS replacement.

CMS CCRC storage requirements

Since I've already talked about ATLAS tonight, I thought it might be useful to mention CMS. The official line from the CCRC meeting that I was at today is that Tier-2 sites supporting CMS should reserve space (probably wise to give them a couple of TBs). The space token should be assigned the CMS_DEFAULT space token *description*.

DPM sites should follow a similar procedure to that for ATLAS described below. dCache sites have a slightly tougher time of it. (Matt, Mona and Chris, don't worry, I hear your cries now...)

ATLAS CCRC storage requirements

For those of you reading the ScotGrid blog, you will have seen this already. Namely, that SRMv2 writing during CCRC (which started yesterday!) will happen using the atlas/Role=production VOMS role. Therefore sites should restrict access to the ATLASMCDISK and ATLASDATADISK space tokens to this role. To do this, release and then recreate the space reservation:

# dpm-releasespace --token_desc ATLASDATADISK
# dpm-reservespace --gspace 10T --lifetime Inf --group atlas/Role=production --token_desc ATLASDATADISK

Note that at Edinburgh I set up a separate DPM pool that was only writable by atlas, atlas/Role=production and atlas/Role=lcgadmin. When reserving the space, I then used the above command but also specified the option "--poolname EdAtlasPool" to restrict the space reservation to that pool (I also only gave them 2TB ;)

Also, for those interested, here's the python code (make sure you have DPM-interfaces installed and /opt/lcg/lib/python in yout PYTHONPATH).

#! /usr/bin/env python
#
# Greig A Cowan, 2008
#

import dpm

def print_mdinfo( mds):
    for md in mds:
        print 's_type\t', md.s_type
        print 's_token\t', md.s_token
        print 's_uid\t', md.s_uid
        print 's_gid\t', md.s_gid
        print 'ret_pol\t', md.ret_policy
        print 'ac_lat\t', md.ac_latency
        print 'u_token\t', md.u_token
        print 't_space\t', md.t_space
        print 'g_space\t', md.g_space
        print 'pool\t', md.poolname
        print 'a_life\t', md.a_lifetime
        print 'r_life\t', md.r_lifetime, '\n'

print '################'
print '# Space tokens #'
print '################'

tok_res, tokens = dpm.dpm_getspacetoken('')

if tok_res > -1:
    md_res, metadata = dpm.dpm_getspacemd( list(tokens) )
    if md_res > -1:
        print_mdinfo( metadata) 

gLite 3.1 DPM logrotate bug

It's in savannah (#30874) but this one may catch out people who have installed the DPM-httpd. There's a missing brace that breaks logrotate on the machine.

CASTOR dynamic information publishing

Folks, we now have dynamic data published for CASTOR. Yes, it's something we were supposed to do since the EGEE conference in Pisa(!), but only now has it percolated up through all the other tasks to become a high priority - otherwise the focus has been on CASTOR itself.
For details (and current limitations) of the implementation, please refer to: https://www.gridpp.ac.uk/wiki/RAL_Tier1_CASTOR_Accounting

CCRC confusion

So, which Tier-2s are involved in the February CCRC exercise? Does anyone know? What about the date that they will get involved? Do they need to have SRM2.2, or not? Some sources suggest they do, others suggest they don't. If anyone knows the answers to these questions, please email me. I think I'll start attending the daily meetings to find out what is going on.

Also, it turns out that although the 1.8.0-12 release of dCache was made sometime last week, it turns out that this is also the CCRC branch of dCache. Do you spot the difference? Good, then we can continue. This explains the weird naming convention that the dCache developers are using for all CCRC related releases, namely 1.8.0-12p1, 1.8.0-12p2 for patch versions 1 and 2 of the 1.8.0-12 branch. Hope that clears things up.

Banned by SAM

Looks like the SAM people weren't very happy with the monitoring that I was running to summarise the status of storage resources on the Grid. In fact, they were so unhappy that they decided to block the IP address that I was using and switch off access to the /sqldb/ query path on their server! As such, the monitoring hasn't been working for the past few days.

I'll admit that I was putting a fair bit of load on their database since I was using a multithreaded application to request information about 6 different tests for ~200 SEs over the past 30 days, but I think this is a bit harsh. Another part of the problem is that the script seemed to trip up when processing information on a couple of sites (and it was always the same sites), leading to it continually requesting the same information. I'd love to debug why this was, but this is somewhat difficult when I can't access the DB.

I'm currently jumping through some hoops to try and get access to the new JSP endpoint. Hopefully this will be granted soon and I can update the application to deal with the new XML schema.

Monitoring SRM2.2 deployment

Using a combination of the WLCG information system, srmPing from the StoRM SRM client, a multithreaded python script, GraphTool and Apache, I've set up some daily monitoring of all of the SRM v2.2 endpoints that are available on the Grid. This will help track deployment of the new software, allowing us to see which versions are running in production.

http://wn3.epcc.ed.ac.uk/srm/xml/

Using the cool GraphTool features, you can drill down into the data by typing dCache, CASTOR or DPM in the SRM_flavour box. You can also look at a particular countries endpoints by putting something like .uk in the endpoint box (for all UK sites). Hopefully poor old wn3.epcc can handle all this monitoring that it's doing!

One thing to note is that there is quite a variety of dCache 1.8.0-X out there. What are the developers playing at?

A slightly annoying feature is that srmPing requires a valid proxy, so I'll need to come up with a good way of creating a long-lived one.

Global dCache monitoring

It's been a while since my last post - been busy with some things that I will post about shortly. As you know, I've been running the GridPP SAM storage monitoring for the past few weeks. It looks like the dCache team got wind of this and have asked if I could set up something similar to summarise the SAM test results for all dCache sites in the WLCG information system. This is now done and the results can be seen above and at this new page in the dCache wiki:

http://trac.dcache.org/trac.cgi/wiki/MonitoringDcache

I don't seem to be able to get SAM results for the US Tier-2 sites, so I'll need to investigate this. The above link contains other useful information that I'll talk about shortly.

dCache PostgreSQL monitoring

Happy New Year everyone!

Following a recommendation from the dCache developers, I set up some PostGreSQL monitoring using pgFouine. It is very simple to install and configure. The plots are generated automatically by the pgfouine tool in addition to giving a full breakdown of what queries took the longest time. This should be useful to let you understand what your database (and hence dCache) is doing. You can also use it to analyse the output of the VACUUM (FULL) ANALYSE command which should give you an idea as to how large the FSM should be set to (which is an important configuration parameter).

PS Fouine is French for stone marten, which is something like a weasel.

Anyone for some dCache monitoring?

The above plots come from some new dCache monitoring that I have set up to study the behaviour of the Edinburgh production storage (srm.epcc.ed.ac.uk). This uses Brian Bockleman's GraphTool and some associated scripts to query the dCache billing database. You can find the full set of plots here (I know, it's a strange hostname for a monitoring, but it's all that was available):

http://wn3.epcc.ed.ac.uk/billing/xml/

GraphTool is written in python and uses matplotlib to generate the plots. Cherrypy is used for the web interface. The monitoring can't just be installed as an rpm: you need to have PostGreSQL 8.2 available; create a new view in the billing database; set up Apache mod_rewrite; ensure you have the correct compilers installed..., but these steps shouldn't be a problem for anyone.

I think you will agree that the monitoring presents some really useful views of what the dCache is actually doing. It's still a work in progress, but let me know when you want to set it up and I should be able to help.

It should be possible to do something similar for DPM in the coming weeks.

DPM on SL4

Time to break out the champagne, it looks like DPM will be officially released in production on SL4 next Wednesday.

"Based on the feedback from PPS sites, we think that the following
patches can go to production next Wednesday:


# available on the linked GT-PPS ticket(s)

#1349 glite-LFC_mysql metapackage for SLC4 - 3.1.0 PPS Update 10

#1350 glite-SE_dpm_disk metapackage for SLC4 - 3.1.0 PPS Update 10

#1352 glite-SE_dpm_mysql metapackage for SLC4 - 3.1.0 PPS Update 10

#1541 glite-LFC_oracle metapackage for SLC4 - 3.1.0 PPS Update 10

#1370 R3.1/SLC4/i386 DPM/LFC 1.6.7-1 - 3.1.0 PPS Update 10"

Of course, some sites have been running SL3 DPM on SL4 for over a year and others have been running the development SL4 DPM in production for months. One warning I would give would be to make sure the information publishing is working, I've had a few problems with that in the past (in fact today I was battling with an incompatible version of perl-LDAP from the DAG repository).

Storage as seen by SAM

We all need more monitoring, don't we? I knocked up these plots showing the storage SAM test results for the ops VO at GridPP sites over the past month. I am only looking at the SE and SRM tests here, where the result for each day is calculated as the number of successes over the total number of tests. The darker-green the square the higher the availability. I think it's clear which sites are having problems.

http://www.gridpp.ac.uk/wiki/GridPP_storage_available_monitoring

We always hear that storage is really unreliable for the experiments, so I was actually quite surprised at the amount of green on the first plot. However, I think since these results are only for the short duration ops tests, they do not truely reflect the view that the experiments have of storage when they are performing bulk data transfer across the WAN or a large amount of local access to the compute farm.

These plots were generated thanks to some great python scripts/tools from Brian Bockleman (and others, I think) from OSG. Brian's also got some interesting monitoring tools for dCache sites which I'm having a look at. It would be great if we could use something similar in GridPP.

dCache 1.8.0-X

A new patch to dCache 1.8.0 was released on Friday (1.8.0-6). In addition, there is now a 1.8.0 dcap client. All rpms can be found here:

http://www.dcache.org/downloads/1.8.0/index.shtml

Sites (apart from Lancaster!) should wait for all of the Tier-1s to upgrade first of all as there are still some bugs being worked out.

dCache admin scripts

Last week I finally got a chance to have another look at some of the dCache administration scripts that are in the sysadmin wiki [1]. There is a jython interface to the dCache admin interface, but I find it difficult to use. As an alternative, the guys at IN2P3 have written a python module that creates a dCache admin door object which you can then use in your own python scripts to talk to the dCache [2]. One thing that I did was use the rc cleaner script [3] to clean up all of the requests (there were 100's!) that were stuck in Suspended state. You can see how the load on the machine running postgres dropped after removing the entries. Highly recommended.

I also wrote a little script to get information from the LoginBroker in order to print out how many doors are active in the dCache. This is essential information for sites that have many doors (i.e. Manchester) but find the dCache 2288 webpage difficult to use. I'll put it in the SVN repository soon.

[1] http://www.sysadmin.hep.ac.uk/wiki/DCache
[2] http://www.sysadmin.hep.ac.uk/wiki/DCache_python_interface
[3] http://www.sysadmin.hep.ac.uk/wiki/DCache_rc_cleaner

gridpp-storage turns 1(00)

Just thought I would let everyone know that we have now reached the 100th posting on this little blog and it also happens to fall exactly on the 1 year anniversary of it's creation. Strange how these two milestones coincide like that. Doing the maths, it means that there are about 2 postings per week. What I don't know, however, is whether or not this implies that there is too much or too little to talk about when it comes to Grid storage? Maybe it means I should be doing more work.

As an aside, when I said "everyone" above, I don't actually know how large "everyone" is. As a guess, I would say at most 7 people. Maybe a few comments below would prove me wrong...

Also, I should say that Lancaster are planning on moving to dCache 1.8.0 in production next week. This is great news as it measn they will be all set for the CCRC'08 (the last C means Challenge, that's the important bit) that is due to start 1Q08. Everyone else should have similar plans in their minds.

SRM2.2 deployment workshop - day 2 summary

OK, time to finish off my summary of the workshop; I'll try not to take as long this time.

After a great conference dinner (and amazing speech!) in the George Hotel, I made sure I was in NeSC early on the Wednesday morning to check everything was in working order after the power cut from the previous night. Unfortunately, there were a few gremlins in the system. First of all, a circuit breaker had gone, meaning that only half of the tutorial machines had power, and of those, only half of them had networking. At that point, the hands-on session was basically dead in the water. Fortunately, a University spark (that means electrician) appeared out of nowhere and flicked the switch. A quick reboot and all of the machines came back up - phew! I then had to spend a bit of time reconfiguring a few nodes which the developers had been set loose on; using virtual machines here would definitely have helped, but when I tried this prior to the workshop, the machines at NeSC were really struggling (I've already recommended that they upgrade their hardware). Anyway, the tutorials started and people were able to log on, configure some SRM2.2 spaces and run some client commands as a check. It helped to have the tutorial hand-out that I had prepared along with the dCache and DPM developers (thanks Timur + Sophie).

So far, so good, but disaster wasn't far away as the NeSC firewall had been reset overnight meaning that the tutorial machines could no longer speak to the top-level BDII in the physics department. This was discovered after a quick debug session from Grid guru Maarten Litmaath and the firewall rules were fixed. Unfortunately, this broke wireless connectivity for almost everyone in the building! By far, this was the worst thing that could happen at a Grid meeting - everyone was starting to foam at the mouth by lunch. To be honest, this was the best thing to happen as it meant there were no distractions from the tutorial - something to remember for next time I think ;)

The afternoon kicked off with a discussion about the the rigourous testing that the different SRM2.2 implementations have gone through to ensure they conform to the spec and are able to inter-operate. Things look good so far - fingers crossed in production it reamins the same. We then had a short sessiona about support as this is the latest big thing to talk about with regards to storage. The message that I would give to people is that make sure you stay involved in the community that we have built up; contribute to the mailing list (dpm-user-forum@cern.ch was announced!) and add to the wikis and related material that is on the web. It's really important that we help one another and not constantly pester the developers (although there help will still be needed!). The final session of the workshop talked about the SRM client tools and changes that have been made to them to support SRM2.2. Clearly, sites should have a good working relationship with these tools as they will be one of the mains weapons to check that SRM2.2 configuration is working as expected.

So that's it, the end of the SRM2.2 deployment workshop. I think everyone (well, most) enjoyed their time in Edinburgh and learnt a lot from each other. The hands-on proved a success and this should be noted for future events. Real data is (hopefully!) coming in 2008 so we had better make sure that the storage is prepared for it so that the elusive Higgs can be found! Remember, acting as a community is essential, so make sure that you stay involved!

See you next time.

SRM2.2 deployment workshop - day 1 summary

So that's the workshop over - it really flew by after all of the preparation that was required to set things up! Thanks to everyone at NeSC for their help with the organisation and operations on the day. Thanks also to everyone who gave presentations and took tutorials at the workshop, I think these really allowed people to learn about SRM2.2 and all of the details required to configure DPM, dCache or StoRM storage at their site.

Day 1 started with parallel dCache Tier-1 and StoRM sessions. It was particularly good for the StoRM developers to get together with interested sites as they haven't had much of a chance to do this before. For dCache (once the main developer arrived!), FZK and NDGF were able to pass on information to the developers and other Tier-1s about performing the upgrade to v1.8.0. It was noted that NDGF are currently just using 1.8.0 as a drop in replacement for 1.7.0 - space management has yet to be turned on. FZK had experienced a few more problems - primarily due to a high load on PNFS. However, the eventual cause was traced to a change in pool names without a subsequent re-registration - this is unrelated to 1.8.0.

I had arranged for lunch for the people who turned up in the morning. Of course, I should have known that a few extras would have turned up - but I didn't expect quite as many. Luckily we had enough food, just. I'll know better for next time!

The main workshop started in the afternoon where the concepts of SRM2.2 were presented in order to educate sites and give them the language that would be used during the rest of the workshop. Thanks to Flavia Donno and Maarten Litmaath describing the current status. We then moved onto short presentations from each of the storage developers, outlining the system and how SRM2.2 was implemented. Again, this helped to explain concepts to sites and enabled them to see the differences with the software that is currently deployed.

Stephen Burke gave a detailed presentation about v1.3 of the GLUE schema. This is important as it was introduced to allow for SRM2.2 specific information to be published in the information system. Sites need to be aware of what their SEs should be publishing - there should be a SAM test that will check this out.

The final session of the day was titled "What do experiments want from your storage?". I was hoping that we could get a real idea of how ATLAS, CMS and LHCb would want to use SRM2.2 at the sites, particularly Tier-2s. LHCb appear to have the clearest plan of what they want to do (although they don't want to use Tier-2 disk) as they were presenting exactly the data types and space tokens that they would like set up. CMS presented gave a variety of ideas for how they could use SRM2.2 to help with the data management. While these are not finalised I think they should form the basis for further discussion between all interested parties. For ATLAS, Graeme Stewart gave another good talk about their computing model and data management. Unfortunately, it was clear that ATLAS don't really have a plan for SRM2.2 at Tier-1s or 2s. He talked about ATLAS_TAPE and ATLAS_DISK space tokens, which is a start, but what is the difference between this and just having separate paths (atlas/disk and atlas/tape) in the SRM namespace? What was clear, however, was that ATLAS (and the other experiments) want storage to be available, accessible and reliable. This is essential for both WAN data transfers and local access for compute jobs and is really what the storage community have to focus on prior to the start of data taking - we are here for the physics after all!

So, Day 1 went well, that is until we had a power cut just after the final talk! To be honest though, it turned out to be a good thing as it meant people stopped working on their laptops and got out to the pub. This was another reason for having the workshop - allowing people to mix and match faces to the names that they have seen on the mailing lists. It all helps to foster that community spirit that we need to support each other,

OK, enough for now. I'll talk about Day 2 later.

SRM2.2 deployment workshop - picture gallery

If people have any more photos that they would like to share, then feel free to send them to me.

Storage workshops - the solution to (most of) our problems

While reviewing the outcomes of the workshop, I have come to the conclusion that we should have storage workshops more often as they clearly improve the reliability and availability of Grid storage. I took this snapshot of SAM test results for UK sites the day after the Edinburgh meeting and I think it's obvious that there is a lot of green about, although one site shall remain nameless...

Unfortunately, normal service appears to have resumed. So who is planning the next meeting?

Dark storage?

The lights went out on the SRM2.2 deployment workshop yesterday evening, but it was timed to perfection as the final talk had just finished! Clearly my plan to get everyone out of the building worked really well ;)

Dinner in the George Hotel was excellent (as was the beer afterwards), I will post some photos from the restaurant and day-1 at some point today.

SRM2.2 deployment workshop - bulletin 5

View Larger Map

The workshop is almost upon us, I can't believe it's come round this fast! It was only a few weeks ago that I was proposing the idea of running this event. Preparations have been continuing apace and I now have the computer lab set up with 20 dCache and DPM test instances. Interested participants will use their laptops to connect to these machines during the tutorials and will configure the SRM2.2 service. There will be contributions from many sources during the workshop; the developers, experiment representatives and members of the WLCG GSSD are all putting together material. Thanks to everyone for all of their efforts!

The map above pinpoints some of the main locations that you will need to know about during the workshop. Hopefully you'll get the chance to see some more of Edinburgh's attractions while you are here.

Check Indico for the latest agenda:

http://indico.cern.ch/conferenceDisplay.py?confId=21405

Note that we have added a discussion session about the future of storage support. This is important given that (fingers crossed) we will be receiving physics data next year; we need to ensure the storage services have high availability and reliability.

See you next week.

SRM2.2 deployment workshop - bulletin 4

We are now up to 55 registered participants! This is great - clearly a lot of people are interested in learning about SRM and the workshop presents a good opportunity for everyone to meet the middleware developers; find out about basics (and the complexities) of SRM2.2 and exchange hints/tips with site admins from all over WLCG.

I would ask everyone to come prepared for the workshop - have a think about your current storage setup and how this will be changing over the coming year. I would imagine everyone will be buying more disk so how do you want this set up? If you support many VOs, do you want to give dedicated disk to some of them while the others share?

Also, I've worked out some more of the technicalities of the hands on session. There are 20 desktop machines at NeSC which we can use for the tutorial and I have started setting up dCache/DPM images which will be rolled out across the cluster. I'll ask everyone to use their laptops to ssh onto the machines, but everyone will have to work in pairs; unfortunately there aren't hosts for one each. We'll be using a temporary CA for host and user certificates.

Finally, just to clarify that the workshop starts at 1pm on Tuesday 13th Nov. Make sure you pick up some lunch beforehand! You should be able to find plenty of eateries near to NeSC. Looking forward to see you all soon.

http://indico.cern.ch/conferenceDisplay.py?confId=21405

SRM2.2 deployment workshop - bulletin 3

Just thought I would give a quick update as to the organisation of the workshop. First of all, thanks to those who have already registered - we are fast approaching our limit and the registration page will be closed soon (so apply now if you still would like to come).

http://www.nesc.ac.uk/esi/events/827/

Secondly, I have changed the format of the Indico agenda page to use the "conference" format. Hopefully this makes more sense since we are having a couple of parallel sessions during the workshop which was difficult to see from the default layout.

Thirdly, I am working with people at NeSC to try and setup a hands on session where delegates get a chance to "play" with the SRM2.2 configuration on a dCache or DPM. At the moment there are still some technical "challenges" to overcome. I'll keep you posted.

As always, get in touch if there are any questions.

PS Can anyone name the tartan?

SRM2.2 deployment workshop - bulletin 2

The registration page for the workshop is now available.

http://www.nesc.ac.uk/esi/events/827/

Could everyone who has pre-registered their interest to me please complete the registration form. If you require accommodation you should select the relevant nights and this will be booked for you by NeSC (they will not be paying for it though!). I have a list of names of those who have pre-registered, so I'll be chasing up anyone who does not fill out the form.

The agenda can be found here:

http://indico.cern.ch/conferenceDisplay.py?confId=21405

If you have any SRM or storage issues that you would like raised at the workshop, please let me know and I will try to ensure that it is covered at some stage.

Looking forward to seeing you in Edinburgh!

OSG storage group

I was in a meeting yesterday with the OSG data storage and management group. I had realised that there was some activity going on in the US regarding this, but didn't know just how many people are involved, or what precisely they were doing. Turns out that there is actually quite a few FTEs looking at dCache 1.8 testing, client tools, deployment scripts, bug reports and support. There is clearly quite an overlap with what we try to do in GridPP, although there are a couple of differences:

1. GridPP has only me (and part of Jens) dedicated to storage. The rest comes from the community of site admins that have signed up to the mailing list. This works well, but it would be good if there were others who could dedicate more time to looking at storage issues. There were previously a couple of positions at RAL for this stuff. Hopefully there will be again in GridPP3.

2. Unlike GridPP, OSG is not dedicated to particle physics, hence they support other user communities. It appears that USATLAS and USCMS take on the lions share of support for storage at US Tier-2 sites.

You can find more information here:

https://twiki.grid.iu.edu/twiki/bin/view/Storage/

Interview with ZFS techies - mentions LHC

These guys (Jeff Bonwick and Bill Moore) are techies so not a sales pitch at all. Running time about 48 mins but you can do something else and just listen to it (like in a meeting :-)

http://www.podtech.net/scobleshow/technology/1619/talking-storage-systems-with-suns-zfs-team

Suitable for everyone, covers a lot of software engineering ("software is only as good as its test suite") but also obviously filesystems and "zee eff ess" and storage.

Also mentions CERN's LHC, and the LHC Atlas detector in particular.
And work done by CERN to examine silent data corruption (at around 33 mins into the interview).

SRM2.2 deployment workshop - bulletin 1

This is the first bulletin for the "SRM2.2 Deployment Workshop" which will take place on Tuesday the 13th and Wednesday the 14th of November 2007. The workshop is being organised by the UK's National eScience Centre (NeSC), GridPP and WLCG. It will take place at NeSC, Edinburgh, Scotland.

The goal of the workshop is to bring WLCG site administrators with experience of operating dCache and DPM storage middleware together with the core developers and other Grid storage experts. This will present a forum for information exchange regarding the deployment and operation of
SRM2.2 services on the WLCG. By the end of the workshop, site administrators will be fully aware of the technology that must be deployed in order to provide a service that fully meets the needs of the LHC physics programme.

Particular attention will be paid to the large number of sites who contribute small amounts of computing and storage resource (Tier-2s), as compared with national laboratories (Tier-1s). Configuration of SRM2.2 spaces and storage information publishing will be the main topics of
interest during the workshop. In addition, there will be a dedicated session for the Tier-1 sites running dCache.

As SRM2.2 has been proposed as an Open Grid Forum (OGF) standard, it is likely that the workshop will be of wider interest than only WLCG. However, it should be noted that the intention of the meeting is not to continue discussions of the current specification or of any future
version. The meeting will focus on the deployment of a production system.
It is intended that the meeting will be by invitation only. Anyone who thinks that they should receive an invitation should contact Greig Cowan to register their interest.

A detailed agenda and general information about the event are currently being prepared. A second announcement will be made once these are in place.

DPM Dies

For those who don't read the ScotGRID blog:

http://scotgrid.blogspot.com/2007/09/dpm-dies.html

Filesystems turn readonly

Some of the RAID'ed filesystems at Edinburgh recently decided that they would become readonly. This caused the dCache pool processes that depended on them to fall over (Repository got lost). Some of the affected filesystems were completely full, but not all of them. An unmount/mount cycle seems to have fixed things. Anyone seen this sort of thing before?

SRM2.2 spaces at Tier-2s

One thing that is really worrying me is regarding the current deployment plan for SRM2.2. The idea is to get this stuff rolled out into production at all Tier-2s by the end of January. This is a difficult task when you think of the number of Tier-2s, all with their different configurations and experiments that need supporting. Oh, and it would also be good if we actually knew what the experiments want from SRM2.2 at Tier-2s. There needs to be a good bit more dialogue between them and the GSSD group to find out what spaces should be setup and how the disk should be separated. Or maybe they just don't care and want all the disk in a large block with a single space reservation made against it. One way or the other, it would be good to know.

dCache SRM2.2

There was a dedicated dCache session during CHEP for discussion between site admins and the developers in order to discover the latest developments and get help in the server configuration, which is *difficult*. Link groups and space reservations were flying about all over the place. More documentation is required, but this seems to be difficult when things are changing so fast (new options magically appear, or don't appear, in the dCacheSetup file). A training workshop would also be useful...

Shameless self promotion

OK, so obviously I've got nothing to do tonight other than talk about my contributions to CHEP. The presentation was on the possibility of using DPM in the distributed Tier-2 environment that we have within GridPP. We (Graeme Stewart and myself) used a custom RFIO client running on multiple nodes of the Glasgow CPU farm to read data from a DPM that was sitting in Edinburgh. It was surprisingly easy to do actually. You can find the slides in Indico.
Futuer investigations will use a dedicated, low-latency ligthpath rather than the production network.

We also had a poster at CHEP which was looking at the scalability of DPM when using RFIO access across the LAN. It used an identical method to the WAN paper, but in this case we were interested in really stressing the system and seeing how DPM scales are you add in more hardware. Summary: it performs very well and can easily support at least 100TB of disk. Check out Indico for details.

DPM browser

Something I picked up from a DPM poster was the imminent release of an http/https browser for the DPM namespace. There weren't many details and the poster isn't online, but I think it claims to be using apache.

Full chain testing

The experiments want to test out the complete data management chain, from experiment pit->Tier-0->Tier-1->Tier-2. This exercise has been given the snappy title of Common Computing Readiness Challenge (CCRC) and is expected to run in a couple of phases in 2008: February, then May. This will be quite a big deal for everyone, and we need to make sure the storage is ready to cope with the demands. SRM2.2 is coming "real soon now" and should be fully deployed by the time of these tests (well it *had* better be...), which will make it the first large scale test of the new interface.

Storage is BIG!

It quickly became clear during CHEP that sites are rapidly scaling up their disk and tape systems in order to be ready for LHC turn on next year (first data in October...maybe). For instance, CNAF will soon have 3PB of disk, 1.5PB of tape along with 7.5MSpecInt. That is basically an order of magnitude larger than what they have currently. It was the same story elsewhere.

Kors was right, storage is hot. Fingers crossed that the storage middleware scales up to these levels.

dCache update

I was speaking to Tigran from dCache during CHEP and got some new information about dCache and Chimera.

First off, ACLs are coming, but these are not tied to Chimera. They are implementing NFS4 ACLs, which are then mapped to POSIX, which (according to Tigran) makes them more like NT ACLs. Need to look into this further.

Secondly, the dCache guys are really pushing the NFS v4.1 definition as they see it as the answer to their local data access problems. 4.1 clients are being implemented in both Linux and Solaris (no more need to dcap libraries!). According to Tigran, NFS4.1 uses transactional operations. The spec doesn't detail the methods and return codes exactly. Rather, it defines a set of operations that can be combined into a larger operation. This sounds quite poweful, but how will the extra complexity lead to client-server interoperation?

Finally, one thing which I had realised about Chimera is that it allows you to modify the filesystem without actually mounting it. There is an API which can be used.

"Storage is HOT!"*

* Kors Bos, CHEP'07 summary talk, slide 22.

Now that CHEP is finished (and I'm back from my holiday!) I thought it would be good to use the blog to reflect on what had been said, particularly when it comes to storage. In fact, between the WLCG workshop and CHEP itself, it is clear that storage is the hot topic on the Grid. This was made quite explicit by Kors during his summary talk. Over the next few days I'll post some tidbits that I picked up (once I've finished my conference papers, of course ;).

Improved DPM GIP Plugin Now In Production

The improved DPM plugin, where the SQL query works far better for large DPMs and a number of minor bugs have been fixed, is now in production (should make the next gLite release).

See https://savannah.cern.ch/patch/?1254 for details.

Possibly this will be my last hurrah! with this fine script, before passing off to Greig to do the Glue 1.3 version, now with added space tokens....

DPM vulnerability

Another security hole in the DPM gridftp server has been found and subsequently patched.

All details of the update can be found here.

The security advisory issued by the GSVG can be found here.

All DPM sites should use YAIM (or your method of choice) to upgrade to the latest version (DPM-gridftp-server-1.6.5-6) ASAP. Depending on how regularly you have been updating, there may also be new rpms available for other components of the DPM (all of these are on 1.6.5-5).

sgm and prod pool accounts

I've been a bit confused of late regarding what the best course of action is regarding how to deal with sgm and prod pool accounts on the SEs, in particular, dCache. As an example, Lancaster have run into the problem where a user with an atlassgm proxy has copied files into the dCache and has correspondingly been mapped to atlassgm:atlas (not atlassgm001 etc, just plain old sgm). Non-sgm users have then tried to remove these files from the dCache and have been denied since they are simple atlas001:atlas users. The default dCache file permissions do not allow group write access. This raises a few issues:

1. Why is atlassgm being used to write files into the dCache in the first place?

2. Why are non-sgm users trying to remove files that were placed into the dCache by a (presumably privileged) sgm user?

3. When will dCache have ACLs on the namespace to allow different groups of users access to a bunch of files?

The answer to the 3rd point is that ACLs will be available some time next year when we (finally) get the Chimera namespace replacement to PNFS. ACLs come as a plugin to Chimera.

The interim solution appears to be just to map all atlas users to atlas001:atlas, but this obviously doesn't help the security and traceability aspect that pool accounts are partially trying to solve. Since DPM supports namespace ACLs, we should be OK with supporting sgm and prod pool accounts. Of course, this requires that everyone has the appropriately configured ACLs, which isn't necessarily the case, as we've experienced before.

Comments welcome below.

Storage accounting - new and improved

We have made some improvements to the storage "accounting" portal (many thanks go to Dave Kant) during the past week or so. The new features are:

1. "Used storage per site" graphs are now generated (see image). This shows the breakdown of resources per site, which is good when looking at the ROC or Tier-2 views.

2. "Available storage per VO" graphs are generated in addition to the "Used" plots that we've always had. This comes with the usual caveats of available storage being shared among multiple VOs.

3. There is a Tier-2 hierarchical tree, so that you can easily pick out the Tier-2s of interest.

4. A few minor tweaks and bug fixes.

Current issues are in savannah.

The page is occasionally slow to load up as the server is also used by the GOC to provide RB monitoring of the production grid. Alternatives to improve speed are being looked at.

CE-sft-lcg-rm-free released!

A new SAM test is now in production. It does a BDII lookup to check that there is sufficient space on the SE before attempting to run the standard replica management tests. This is good news for sites whose SEs fill up with important experiment data. If the tests finds that there is no free space, then the RM tests don't run. Of course this requires that the information being published into the BDII is correct in the first place. I'll need to check if this system could be abused by sites who publish 0 free space by default, thereby by-passing the RM tests and therefore any failures that could occur. I suppose that GStat already reports sites as being in a WARNING status when they have no free space.

See the related post here.

CLOSE_WAIT strikes again

Multiple DPM sites are reporting instabilities in the DPM service. The symptoms are massive resource usage by multiple dpm.ftpd processes on the disk servers (running the v1.6.5 of DPM). These have been forked by the main gridftp server process to deal with client requests. Digging a little further we find that the processes are responsible for many CLOSE_WAIT TCP connections between the DPM and the RAL FTS server. It also happens that all of the dpm.ftpd processes are owned by the atlassgm user, but I think this is only because ATLAS are the main (only?) VO using FTS to transfer data at the moment.

CLOSE_WAIT means that the local end of the connection has received a FIN from the other end, but the OS is waiting for the program at the local end to actually close its connection.

Durham, Cambridge, Brunel and Glasgow have all seen this effect. The problem is so bad at Durham that they have written a cron job that kills off the offending dpm.ftpd processes at regular intervals. Glasgow haven't been hit to badly, but then they do have 8GB of RAM on each of their 9 disk servers!

The DPM and FTS developers have been informed. From emails I have seen it appears that the DPM side is at fault, although the root cause is still not understood. This situation is very reminiscent of the CLOSE_WAIT issues that we were seeing with dCache at the end of last year.

Also see here.

DPM and xrootd

Following on from dCache, DPM is also developing an xrootd interface to the namespace. xrootd is the protocol (developed by SLAC) that provides POSIX access to their Scalla storage system, who's other component is the oldb clustering server.

DPM now has a usable xrootd interface. This will sit alongside the rfiod and gridftp servers. Currently, the server has some limitations (provided by A Peters at CERN):

* xrootd server runs as a single 'DPM' identity, all file reads+writes are done on behalf of this identity. However, it can be restricted to read-only mode.

* there is no support of certificate/proxy mapping

* every file open induces a delay of 1s as the interface is implemented as an asynchronous olbd Xmi plugin with polling.

On a short time scale the certificate support in xrootd will be fixed and VOMS roles added (currently certificate authentication is broken for certain CAs) . After that, the DPM interface can be simplified to use certificates/VOMs proxies & run as a simple xrootd OFS plugin without need for an olbd setup.

So it seems that xrootd is soon going to be available across the Grid. I'm sure that ALICE (and maybe some others...) will be very interested.

dcache on SL4

As part of our planned upgrade to SL4 at Manchester, we've been looking at getting dcache running.
The biggest stumbling block is a lack of glite-SE_dcache* profile, luckily it seems that all of the needed components apart from dcache-server are in the glite-WN profile. Even the GSIFtp Door appears to work.

SRMv2.2 directory creation

Just discovered that automatic directory creation doesn't happen with SRMv2.2. Directories are created when using SRMv1.

Annoyed

I re-ran YAIM yesterday on the test DPM I've got at Edinburgh as it turned out we were not publishing the correct site name. Annoyingly, this completely broke information publishing as the BDII couldn't find the correct schema files (again). I had to re-create the symbolic link from /opt/glue/schemas/ldap to /opt/glue/schemas/openldap2.0 and then double check that all was well with the /opt/bdii/etc/schemas files. A restart of the BDII then sorted things out.

It's not really fair to blame YAIM here since I'm running the SL3 build of DPM on SL4, which isn't really supported. Well, I'm hoping that's the source of the trouble.

Non-improvement of SAM tests

For a while I have been pushing for the creation of a SAM test that only probes the SRM and does not depend on any higher level services (like the LFC or BDII). This would be good as it would prevent sites being marked as unavailable when in fact their SRM is up and running.

Unfortunately, the SAM people have decided to postpone the creation of a pure-SRM test. I don't really understand their concerns. I thought using srmcp with a static (but nightly updated) list of SRM endpoints would have been sufficient. I guess they have some reservations about using the FNAL srmcp client, since it isn't lcg-utils/GFAL, which are the official storage access methods.

https://savannah.cern.ch/bugs/?25249

Improvement to SAM replica management tests

The SAM people have added a new test that checks if the default SE has any free space left. This
test will not be critical by default, however its failure will cause the actual replica management tests not to execute at all. This is good news, as a full SE (in my opinion) is not really a site problem and not a reason that the site should fail a SAM test.

The full bug can be found here:

http://savannah.cern.ch/bugs/?26046

Greig

SAM failures, again

Looks like something changed inside SAM (again) yesterday, causing a large number of sites to fail the CE replica management tests with a "permission denied" error.

Further investigation shows that the failed tests were being run by someone with a DN from Cyfronet.
By default, this is mapped to the ops group in the grid map file, not opssgm like Judit Novak and Piotr Nyczyk. It is clear from the Glasgow DPM logs that this DN does not belong to ops/Role=lcgadmin. This then leads to failures in DPMs due to the fact that the dpm/domain/home/ops/generated/ directories have ACLs on them which only grant write permissions to people in ops/Role=lcgadmin.

Looks like things have been rectified now.

Why do we keep on getting hit by things like this?

SRM and SRB interoperability - at last!

People have been talking for years about getting SRM and SRB "interoperable", mostly involving building complicated interfaces from SRX to SRY in various ways.

Now it turns out SRB has a GridFTP interface, developed by Argonne. So here's the idea: why don't we pretend the SRB is a Classic SE?

So we can now transfer files with gridftp (i.e. globus-url-copy) from dCache to SRB and vice versa, although the disadvantage is that you have to know the name of a pool node with a GridFTP door. Incidentally, if you try that, don't forget -nodcau or it won't work (for GridFTP 3rd party copying).

But here's the brilliant thing: it also works with FTS, since FTS still supports Classic SEs. So we have successfully transferred data between dCache, the SRM, and SRB as a, well, GridFTP server, and back again.

Cool, eh?

Next step is to set up a Classic SE-shaped information system for SRB and see if it works with lcg-utils and GFAL (because FTS does not depend on the SE having a GRIS).

This is work with Matt Hodges at Tier 1 who set up FTS, and with Roger Downing and Adil Hasan from STFC for the SRB.

--jens

Slightly Modified DPM GIP Plugin

The last version of the DPM GIP plugin had a few minor bugs:

1. The "--si" flag had been lost somewhere.
2. DNS style VOs were not handled properly (e.g., supernemo.vo.eu-egee.org).

There's a new version which corrects these little problems: http://www.physics.gla.ac.uk/~graeme/scripts/packages/lcg-info-dynamic-dpm-2.2-2.noarch.rpm.

Now submitted as a patch in Savannah.

Optimised DPM GIP Plugin

Lana Abadie noticed that my DPM GIP plugin was rather inefficient (table joins are expensive!), and sent a couple of options for improving the SQL query. I implemented them in the plugin and it speeded up by a factor of 10.

I have produced a new RPM with the optimised query, which is available here: http://www.physics.gla.ac.uk/~graeme/scripts/packages/lcg-info-dynamic-dpm-2.2-1.noarch.rpm.

I am running this already at Glasgow and I would recommend it for anyone with a large DPM.

N.B. It is compatible with DPM 1.6.3 and 1.6.5, but remember to modify the /opt/lcg/var/gip/plugin/lcg-info-dynamic-se wrapper to run /opt/lcg/libexec/lcg-info-dynamic-dpm instead of /opt/lcg/libexec/lcg-info-dynamic-dpm-beta.

SRM2.2 storage workshop

There was a storage workshop held at CERN on the 2nd and 3rd of July. The focus of discussions was on the SRM2.2 developments and testing of the endpoints. The majority of the endpoints are being published in the PPS, the intention being that the experiments will be able to use them in a ~production environment and allow some real stress tests to be run against them. The experiments see SRM2.2 as being an essential service for them, so hopefully they have sufficient manpower to run the tests...

Getting the software installed on the machines isn't a problem, but getting it configured can be tricky. The main point that I tried to highlight on a number of occasions was the necessity for sites to have really good documentation from both the developers (how the SRM2.2 spaces can be configued) and the experiments (how the SRM2.2 spaces should be configured for their needs). I will make sure that I provide instructions for everyone to ensure that the deployment goes (relatively) smoothly. It shouldn't be too much of a problem for DPM sites, dCache sites will need to start playing around with link groups ;-)

From mid-October, sites should be thinking of having these SRM2.2 spaces configured. The plan is that by January 2008, everyone will have this functionality available, and SRM2.2 will become the default interface.

DPM gridftp security

Apologies for not posting for a while, it's been a busy few weeks. First thing that should be mentioned is the gaping security hole that existed in the DPM gridftp server. Users using the uberftp (or some other suitable) client could log into the server and change permissions on anyones files, move files to different areas of the DPM namespace or even move files outside of the namespace altogether. Thanks to Kostas and Olivier at Imperial for spotting this. Unfortunately, it took a couple of weeks, 3 patch releases and a lot of testing within GridPP before we finally plugged the hole.

Initially only patched version of the 1.6.5 server was produced. I asked for the fix to be back-ported to 1.5.10 as there were a few sites still running this version, unable to upgrade to the latest release (due to the upgrade problems) due to ongoing experiment tests and wanting to be as secure as they could be. This was done, so thanks to the DPM team.

All sites should upgrade to the latest version of DPM and ensure that they are running patch -4 of the gridftp server.

DPM 1.6.5-1 in PPS

v1.6.5-1 of DPM is now in pre-production. The relevant savannah page is here:

https://savannah.cern.ch/patch/index.php?1179

This release involves various bug fixes. What is interesting is that it will now be possible to set ACLs on DPM pools, rather than just limiting a pool to either a single VO or all VOs. This should make sites happy. The previous posting on this version of DPM mentioned that gridftpv2 would be used, but the release notes don't mention this, so we will have to wait and see.

Also out in PPS is the use of v1.3 of the GLUE schema. This is really good news since GLUE 1.3 will allow SEs to properly publish information about SRM2.2 storage spaces (i.e. Edinburgh has 3TB of ATLAS_AOD space).

https://savannah.cern.ch/patch/index.php?980

DPM 1.5.10 -> 1.6.4 upgrade path broken in YAIM

As reported at yesterdays storage meeting, the upgrade path from DPM 1.5.10 to 1.6.4 is broken in YAIM 3.0.1-15. The different versions of DPM require database schema upgrades in order to be able to handle all of the SRM2.2 stuff (space reservation etc). YAIM should contain appropriate scripts to perform these upgrades, but it appears that they appropriate code has been removed, meaning that it is no longer possible to move from schema versions 2.[12].0 in v1.5.10 of DPM to schemas 3.[01].0 in v1.6.4. We stumbled upon this bug when I asked Cambridge to upgrade to the latest DPM in an attempt to resolve the intermittent SAM failures that they were experiencing. A fairly detailed report of what was required to solve the problem can be found in this ticket:

https://gus.fzk.de/pages/ticket_details.php?ticket=23569

It should be noted that for some reason (a bug in a YAIM script?) the Cambridge DPM was missing two tables from the dpm_db database. These were dpm_fs and dpm_getfilereq (I think). This severely hindered the upgrade since we were trying to upgrade the schema, which was successful, but then the DPM wouldn't start. A restore of the database backup, then an upgrade to DPM 1.6.3 then onto DPM (I'm keeping a close eye on the SAM tests...). Sites should be aware that they may need to follow the steps detailed in this link while performing the database upgrade.

https://twiki.cern.ch/twiki/bin/view/LCG/DpmSrmv2Support

After the installation, the srmv2.2 daemon was running and the SRM2.2 information was being published by the BDII. This is all good. If you end up using yaim 3.0.1-16, it should not be necessary to manually install the host certificates for the edguser.

In summary, the 1.5.10 to 1.6.4 upgrade was a lot of work. Thanks to Santanu for giving me access to the machine. This problem raises issues about sites keeping up to date with the latest releases of middleware. Although there were problems with the configuration of 1.6.4, v1.6.3 has been stable in production for a while now. I'm not really sure why some sites hadn't upgraded to that. It would be great if every site could publish the version of the middleware that they are using. In fact, such a feature may be coming very soon. Just watch this space.

Anyone for a DPM filesystem?

Looks like someone at CERN is developing a mechanism to enable DPM servers to be mounted. This DPMfs could be used as a simple DPM browser, presenting the namespace in a more user-friendly form than the DM command line utilities. The DPM fs is implemented using the FUSE kernel module interface. The file system calls are forwarded to the daemon which communicates with the DPM servers using the rfio and dpns API and sends back the answer to the kernel.

It's in development and not officially supported:

https://twiki.cern.ch/twiki/bin/view/LCG/DPMfs

DPM 1.6.5 Coming...

It's not here yet, but DPM 1.6.5 has been tagged for release as part of gLite 3.1. A list of goodies with this release are:

- remove expired spaces
- avoid crash in dpm_errmsg/Cns_errmsg when supplied
 buffer is too small (GGUS ticket 21767)
- correct processing of rfio_access on DPM TURLs
 (Atlas)
- return DPM version in otherInfo field of srmPing
 response
- dpm-shutdown: take "server" into account
- add methods ping and getifcevers in LFC/DPM
- fixed bug #25830: add ACLs on disk pools
- dpm-qryconf: add option --group to display
 groupnames instead of gids
- dpm-qryconf: add option --proto to display
 supported protocols
- fixed bug #25810: dpm-qryconf: add option --si
 to display sizes in power of 10
- implement recursive srmLs and srmRmdir
- DPM-DSI plug-in for the GT4 gridftp2 server

The gridftp v2 server looks to be rather an interesting development.

The patch has all the details.

SAM test failures explained

Here's the story: The past couple of weeks have been pretty bad for SAM. There have been at least 3 big problems with the service due to backend database issues, moving to new hardware, etc. In amongst all of this, the certificate of the user who runs the SAM test ran out (I don't know what happened to the CA warning a month before). It was decided to implement a quick fix by using a different users certificate to submit the test. This was OK for a while, until the ops replica management tests then tried to create a new ops/generated/YYYY-MM-DD directory early on Saturday morning. This was fine for dCache sites, but DPM sites suffered due to the DPM not mapping the new certificate DN + VOMs attributes to a virtual gid that would give permission to create these generated directories. This was the source of the "permission denied" errors that were being reported by lcg-cr. Once sites updated the ACLs on the ops/generated directories, the new certificate DN + VOMs attributes had authorisation to write a new directory and the tests started to pass again.

As an aside, the initially errors pointed to a permissions problem on the LFC, but this was a red herring. This is another example of the poor error messages that are reported by grid middleware.

DPM sites failing SAM due to change in ops VOMs role

The SAM people changed the VOMs role of the certificate being used to run the ops SAM tests. This led to the majority of DPM sites on the grid failing the replica management tests on over the weekend. Why they made this change (with no announcement) on a Friday is unknown. Graeme's got some information here:

http://scotgrid.blogspot.com/2007/06/sam-tests-changed-voms-role-without.html

All UK DPM sites were failing with the exception of RHUL and Brunel (well done Duncan). All of these sites should run the script that was posted to LCG-ROLLOUT as this will alter the ACLs on the generated directories appropriately.

The other annoying thing is that this wouldn't have happened if all sites were running DPM 1.6.4 (which supports secondary groups). The problem is that this release is broken (due to 2 different problems) meaning that no one is running it!

Update on the classic SE

As reported at last weeks WLCG operations meeting, the classic SE is now completely frozen (not that it affects any GridPP sites since we are all SRM-ified). There are a few points to note about this:

Are there things not in the classic SE that people want added? Any such features should already be in DPM so this is an acceptable upgrade.

Are there features in the classic SE that are not available in DPM or dCache? The obvious answer is real POSIX mounting of the file system probably via NFS. Both DPM and dCache are looking at supporting NFSv4, there are comments from both of these that NFSv4 might be available by end of the year or sooner.

Will the classic SE be included in the upcoming gLite release? The answer is yes, it will appear in gLite 3.1. Once there is a DPM with NFSv4 support then this will be re-evaluated.

Storage security service challenge

WLCG are asking each ROC to run a security service challenge their sites. Someone (probably Alessandra) will submit a job to each site which will attempt to write a file to the local SE, read it back, copy it to a remote SE, delete the file... Once complete, the submitter will issue a GGUS ticket against the site, asking them to provide information on which operations were performed on the file. You can see an example of what is expected here:

https://gus.fzk.de/pages/ticket_details.php?ticket=22012

The aim of this testing is to determine if SEs record sufficient information for tracing user operations and also to check that site admins are able to gather that information. I am currently putting together some scripts that will perform the querying and parsing of DPM/dCache databases and log files in order to gather the information.

In addition to going through the SE files, it is likely that sites will have to parse the PBS logs on the CE to determine the UI that was originally used for the job submission.

DPM 1.6.4 and SL4

I have just come across a problem with running DPM v1.6.4 on SL4. Well, it's not actually a problem with DPM itself, but rather a problem with the BDII that it is now using as an information provider. SL4 comes with openldap v2.2 which appears to have stricter schema checking than openldap v2.0 (which comes with SL3). This causes problems like this:

$ ldapsearch -LLL -x -H ldap://wn4.epcc.ed.ac.uk:2170 -b mds-vo-name=resource,o=grid
Invalid DN syntax (34)
Additional information: invalid DN

Meaning that your SE can't publish anything about itself. This can be resolved by adding this block of code

attributetype ( 1.3.6.1.4.1.3536.2.6.1.4.0.1
NAME 'Mds-Vo-name'
DESC 'Locally unique VO name'
EQUALITY caseIgnoreMatch
ORDERING caseIgnoreOrderingMatch
SUBSTR caseIgnoreSubstringsMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.44
SINGLE-VALUE
)

to /opt/glue/schema/ldap/Glue-CORE.schema and then restarting the ldap and bdii processes. This is covered by this bug:

https://savannah.cern.ch/bugs/index.php?15532

DPM 1.6.4 released (with a few problems)

DPM v1.6.4 was released into production this week. First of all, there are a few points to be aware of:

1. This release requires an update of the v1.6.3 DB schema. **YAIM will take care of this for you**. It is not necessary to run the DB migration script by hand.

2. Two new YAIM variables, DPM_DB and DPNS_DB, are introduced.

3. After the reconfiguration, DPM will use the BDII as an information provider instead of Globus MDS. By default the BDII runs on port 2170 whereas globus-mds was on 2135. You need to change the site-info.def variable to this (so that the site BDII looks in the right place)

BDII_SE_URL="ldap://$DPM_HOST:2170/mds-vo-name=resource,o=grid"

4. YAIM does some tweaking of the /etc/sysctl.conf values. The old values are copied to /etc/sysctl.conf.orig if you want to reinstate them.

However, once the release was announced, a couple of problems soon reared their heads:

a) Sites were recommended not to upgrade due to problem left over from the build

http://glite.web.cern.ch/glite/packages/R3.0/updates.asp

For sites who had already upgraded, the fix was this:

   mkdir -p /home/glbuild/GLITE_3_0_3_RC1_DATA/stage/etc
   ln -s /opt/lcg/etc/lcgdm-mapfile \
 /home/glbuild/GLITE_3_0_3_RC1_DATA/stage/etc

b) With the latest update the info provider of the DPM machines has changed from MDS to BDII. However the YAIM ( -15) coming with the update does not configures edguser's certificate.

The fix was to perform these steps manually:

mkdir -p ~edguser/.globus
chown edguser:edguser ~edguser/.globus
cp /etc/grid-security/hostcert.pem ~edguser/.globus/usercert.pem
cp /etc/grid-security/hostkey.pem ~edguser/.globus/userkey.pem
chown edguser:edguser /home/edguser/.globus/user*
chmod 400 /home/edguser/.globus/userkey.pem

Obviously the certification testing isn't quite as water-tight as we would hope.

Manchester Tier2 dcache goes resilient II

Yesterday we completed the scheduled downtime, and now dcache02 is up and resilient, it's still chewing through the list of files and making copies of them, going by past experience it will probably finish somewhere around lunchtime tomorrow. It's so nice to know we're not in the dark ages of dcache-1.6.6 any more. Of course, there's still small niggles to iron out and we've yet to really throw a big load at it, but it's looking a lot, lot better.

DPM 1.6.4-3 on PPS

DPM v1.6.4-3 is now available on the PPS. I would imagine that it will move into production in the next couple of weeks. This version requires a schema change to the dpm_db (3.0.0 -> 3.1.0). YAIM will take of this for you, although a DB backup is recommended beforehand.

We have now moved to YAIM 3.0.1-15, so the installation and configuration steps now look like:

$ /opt/glite/bin/yaim -i -s /opt/glite/yaim/etc/site-info.def -n glite-SE_dpm_mysql

$ /opt/glite/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n SE_dpm_mysql

https://savannah.cern.ch/patch/index.php?1121

The information provider plugin is still the *old* one (which does not account for the used space properly). Therefore you will need to install Graeme's new one by hand (again).

http://www.gridpp.ac.uk/wiki/DPM_Information_Publishing#Beta_Release_Plugin

With this version of DPM there is a BDII process on port 2170 that is used to provide the information about the DPM. This replaces globus-mds as the information provider which ran on port 2135.

This version of YAIM includes the some /etc/sysctl.conf tweaks in the config_DPM_disk function. This is nice (since it could lead to some optimisations) but I think sites should be warned about this beforehand and be allowed to turn off these changes:

https://gus.fzk.de/pages/ticket_details.php?ticket=21713

Anyway, I upgraded from v1.6.3 to v1.6.4 today (on SL4 32bit). No problems so far,  but I will let you know if anything comes up.

ZFS performance on RAID

http://milek.blogspot.com/2007/04/hw-raid-vs-zfs-software-raid-part-iii.html

Manchester Tier2 dcache goes resilient

We're half way through the combined upgrade from dcache-1.6.6-vanilla to dcache1.7.0-with-replica-manager, so far only one of the two head-nodes has been upgraded, but so far so good, the other is scheduled for upgrade next week, and I appear to have scheduled the queue shutdown at 8am on bank-holiday Monday! Documentation will obviously follow including cfengine snippets for those people that love it.

Storage meeting: Wednesday 2nd May

Agenda is here:



http://indico.cern.ch/conferenceDisplay.py?confId=15668

Powered by ScribeFire.

Video of DPM and SRM Presentations at HEPiX

Starring Mr Steve Traylen, including Video download! See Steve's Blog for the links.