Draining dCache pools

I did a bit of housekeeping on the Edinburgh dCache yesterday. dteam was sharing a bunch of pools with atlas so I wanted to drain the dteam files to their own dedicated set of disks. Once you have been through the process a few times, it is relatively painless, but it's certainly not as easy as just submitting a single command.

First, you need to use the copy module to copy files from a source pool to a destination pool or pool group. The module gives you some tools that allow you to limit which files that have to be copied, i.e.

maintenance> load pool pool1_01

loads the source pool.

maintenance> exclude atlas:GENERATED

allows you to exlude all files in the atlas:GENERATED storage class (not SRM v2.2 classes BTW) from the list of files to be copied. You then just start the transfer by running,

maintenance> copyto pools pool1_20 pool1_21

Once the copy has finished, I found that there were a number of errors returned ( ls stat ) for certain PNFSids. Running pathfinder on these showed that these PNFSids were orphaned files in that although they appeared in the pool and physically on the disk, they were no longer in the namespace. This has been observed a number of times by many people. These files that have been left behind after the copy process can be clearly identified in the companion database by following the query in the wiki here. The wiki also contains some examples of scripts that can be used to interface with the dCache ssh admin shell in order to allow you to check the status of orphaned files and remove them if necessary.

Once these orphaned files were identified and removed I then ran the same query on the companion database to ensure that each of the (non-orphaned) source dteam files resided on 2 pools (source and destination). Once this was confirmed, I could then issue the relevant command (rep rm -force ) in each of the pool cells to force the removal of the dteam PNFSids from the source pools. This is best done by scripting the admin interface. You should follow the examples in the links given above for doing this.

In summary, it would be great if there was simply a drain-pool command that did all of this for you. In saying that, the situation has improved with the new functionality available in the copy module. Maybe I will try and combine the above scripts into a single (or at least a smaller number) of steps.