DPM 1.5.10 -> 1.6.4 upgrade path broken in YAIM

As reported at yesterdays storage meeting, the upgrade path from DPM 1.5.10 to 1.6.4 is broken in YAIM 3.0.1-15. The different versions of DPM require database schema upgrades in order to be able to handle all of the SRM2.2 stuff (space reservation etc). YAIM should contain appropriate scripts to perform these upgrades, but it appears that they appropriate code has been removed, meaning that it is no longer possible to move from schema versions 2.[12].0 in v1.5.10 of DPM to schemas 3.[01].0 in v1.6.4. We stumbled upon this bug when I asked Cambridge to upgrade to the latest DPM in an attempt to resolve the intermittent SAM failures that they were experiencing. A fairly detailed report of what was required to solve the problem can be found in this ticket:

https://gus.fzk.de/pages/ticket_details.php?ticket=23569

It should be noted that for some reason (a bug in a YAIM script?) the Cambridge DPM was missing two tables from the dpm_db database. These were dpm_fs and dpm_getfilereq (I think). This severely hindered the upgrade since we were trying to upgrade the schema, which was successful, but then the DPM wouldn't start. A restore of the database backup, then an upgrade to DPM 1.6.3 then onto DPM (I'm keeping a close eye on the SAM tests...). Sites should be aware that they may need to follow the steps detailed in this link while performing the database upgrade.

https://twiki.cern.ch/twiki/bin/view/LCG/DpmSrmv2Support

After the installation, the srmv2.2 daemon was running and the SRM2.2 information was being published by the BDII. This is all good. If you end up using yaim 3.0.1-16, it should not be necessary to manually install the host certificates for the edguser.

In summary, the 1.5.10 to 1.6.4 upgrade was a lot of work. Thanks to Santanu for giving me access to the machine. This problem raises issues about sites keeping up to date with the latest releases of middleware. Although there were problems with the configuration of 1.6.4, v1.6.3 has been stable in production for a while now. I'm not really sure why some sites hadn't upgraded to that. It would be great if every site could publish the version of the middleware that they are using. In fact, such a feature may be coming very soon. Just watch this space.

Glasgow Upgrade to DPM 1.6.3

I have upgraded Glasgow's DPM to 1.6.3 this morning. Executive summary is that everything went well, and we're now happily running the new DPM with a shiny SRM v2.2 daemon.

The key thing about this upgrade is the need to update the database schema for the new SRM v2.2 services. This update requires the DPM and DPNS daemons to be stopped. The easiest way to achieve this is to do the update through YAIM - this will stop the daemons, update the schema and then restart them. Have a look at the config_DPM_upgrade function (in YAIM 3.0.0-36) - the important upgrade is upgradeC.

As a piece of extra insurance at Glasgow I shutdown the DPM and took an extra database dump before I ran YAIM, just in case anything went wrong.

1. Enter downtime in the GOC - your DPM will be down from 10-30 minutes, probably, depending on how big it is. This downtime only affects your SE though (probably safest to put in an hour, then come out early once things are working).
   
2. Stop your DPM daemons. The right order is:
   

   
   service dpm-gsiftp stop
   service srmv2 stop
   service srmv1 stop
   service dpm stop
   service rfiod stop
   service dpnsdaemon stop
   

   
   
3. Dump your database (instuctions on the wiki).
   
4. Update your RPMs. If you use APT then YAIM's install_node works. At Glasgow we just do yum update.
   
5. Rerun YAIM:
   

   
   /opt/glite/yaim/scripts/configure_node /opt/glite/yaim/etc/site-info.def SE_dpm_mysql | tee /tmp/upgrade
   

   
   
6. Now, when the script gets to the database schema upgrade beware that this takes a considerable length of time. If you look at the ganglia plot above you'll see that on Glasgow's DPM, which is a fast machine, it took 12 minutes (we have about 100 000 files in our DPM).
   
7. As advised, we got the harmless duplicate key warning:
   

   
   Configuring config_DPM_upgrade
   INFO Checking for database schema version... 
   INFO Database version used: 2.2.0
   INFO Upgrading database schema from 2.2.0 to 3.0.0 !
   INFO:  Stopping DPM services.
   [...]
   failed to query and/or update the DPNS/DPM databases : DBD::mysql::db do failed: Duplicate key name 'G_PFN_IDX' at UpdateDpmDatabase.pm line 264.
   Issuing rollback() for database handle being DESTROY'd without explicit disconnect().
   Issuing rollback() for database handle being DESTROY'd without explicit disconnect().
   Mon Mar 12 12:01:03 2007 : Starting to update the DPNS/DPM database.
   Please wait...
   INFO Schema version upgrade: Now, may you see the "Duplicate key name 'G_PFN_IDX' at UpdateDpmDatabase.pm" error message, it is harmless. The indexes are already created.
   INFO Starting DPM services
   

   
   
8. However, check this area of the upgrade very carefully for errors - if there is a problem with the update script running from YAIM you'll have to investgate and fix it, possibly running the dpm_support_srmv2.2 script by hand.
   
9. Check all is well, e.g., lcg-cr a file into your DPM and then lcg-cp it back out.
   
10. If you're really nervous (or extra careful) you could login to MySQL and check your schema version is 3.0.0 (select * from schema_version; on dpm_db and cns_db).
    
11. Come out of downtime and pat yourself on the back.
    

It would be possible to do this update by hand as well. A rough outline would be:

1. Downtime as above.
   
2. Stop your DPM as above.
   
3. Run the update script (/opt/lcg/share/DPM/dpm-support-srmv2.2/dpm_support_srmv2.2 --db-vendor MySQL --db localhost --user $DPM_DB_USER --pwd-file $tmpfile --dpns-db cns_db --dpm-db dpm_db --verbose) by hand.
   
4. Check the db schema is updated.
   
5. Restart DPM.
   
6. Test file transfers.
   
7. Come out of downtime.
   

I haven't tried this so at the very least double check these instructions against how YAIM might do it.

Finally, DPM disk servers and WN/UI clients can be upgraded very simple. Do it though YAIM or just simply do a yum update and on the disk servers restart rfiod and dpm-gsiftp.