12 March 2007

Glasgow Upgrade to DPM 1.6.3

I have upgraded Glasgow's DPM to 1.6.3 this morning. Executive summary is that everything went well, and we're now happily running the new DPM with a shiny SRM v2.2 daemon.

The key thing about this upgrade is the need to update the database schema for the new SRM v2.2 services. This update requires the DPM and DPNS daemons to be stopped. The easiest way to achieve this is to do the update through YAIM - this will stop the daemons, update the schema and then restart them. Have a look at the config_DPM_upgrade function (in YAIM 3.0.0-36) - the important upgrade is upgradeC.

As a piece of extra insurance at Glasgow I shutdown the DPM and took an extra database dump before I ran YAIM, just in case anything went wrong.

  1. Enter downtime in the GOC - your DPM will be down from 10-30 minutes, probably, depending on how big it is. This downtime only affects your SE though (probably safest to put in an hour, then come out early once things are working).
  2. Stop your DPM daemons. The right order is:

    service dpm-gsiftp stop
    service srmv2 stop
    service srmv1 stop
    service dpm stop
    service rfiod stop
    service dpnsdaemon stop

  3. Dump your database (instuctions on the wiki).
  4. Update your RPMs. If you use APT then YAIM's install_node works. At Glasgow we just do yum update.
  5. Rerun YAIM:

    /opt/glite/yaim/scripts/configure_node /opt/glite/yaim/etc/site-info.def SE_dpm_mysql | tee /tmp/upgrade

  6. Now, when the script gets to the database schema upgrade beware that this takes a considerable length of time. If you look at the ganglia plot above you'll see that on Glasgow's DPM, which is a fast machine, it took 12 minutes (we have about 100 000 files in our DPM).
  7. As advised, we got the harmless duplicate key warning:

    Configuring config_DPM_upgrade
    INFO Checking for database schema version...
    INFO Database version used: 2.2.0
    INFO Upgrading database schema from 2.2.0 to 3.0.0 !
    INFO: Stopping DPM services.
    failed to query and/or update the DPNS/DPM databases : DBD::mysql::db do failed: Duplicate key name 'G_PFN_IDX' at UpdateDpmDatabase.pm line 264.
    Issuing rollback() for database handle being DESTROY'd without explicit disconnect().
    Issuing rollback() for database handle being DESTROY'd without explicit disconnect().
    Mon Mar 12 12:01:03 2007 : Starting to update the DPNS/DPM database.
    Please wait...
    INFO Schema version upgrade: Now, may you see the "Duplicate key name 'G_PFN_IDX' at UpdateDpmDatabase.pm" error message, it is harmless. The indexes are already created.
    INFO Starting DPM services

  8. However, check this area of the upgrade very carefully for errors - if there is a problem with the update script running from YAIM you'll have to investgate and fix it, possibly running the dpm_support_srmv2.2 script by hand.
  9. Check all is well, e.g., lcg-cr a file into your DPM and then lcg-cp it back out.
  10. If you're really nervous (or extra careful) you could login to MySQL and check your schema version is 3.0.0 (select * from schema_version; on dpm_db and cns_db).
  11. Come out of downtime and pat yourself on the back.

It would be possible to do this update by hand as well. A rough outline would be:

  1. Downtime as above.
  2. Stop your DPM as above.
  3. Run the update script (/opt/lcg/share/DPM/dpm-support-srmv2.2/dpm_support_srmv2.2 --db-vendor MySQL --db localhost --user $DPM_DB_USER --pwd-file $tmpfile --dpns-db cns_db --dpm-db dpm_db --verbose) by hand.
  4. Check the db schema is updated.
  5. Restart DPM.
  6. Test file transfers.
  7. Come out of downtime.

I haven't tried this so at the very least double check these instructions against how YAIM might do it.

Finally, DPM disk servers and WN/UI clients can be upgraded very simple. Do it though YAIM or just simply do a yum update and on the disk servers restart rfiod and dpm-gsiftp.

No comments: