28 February 2017

ZFS snapshots on Linux

One great feature of ZFS is its possibility to have instant snapshots due to the general use of Copy-on-Write.
Such snapshots are a read-only view of the file system at the time the snapshots were created. Also, at the time of the snapshot creation, a snapshot does not take any disk space. Over time, only blocks that were changed on the original file system will take up space for the snapshot. This makes snapshots an ideal candidate for on-disk backups to recover from accidental file changes.
One of the best use cases for it is when ZFS is used for home directories where every user gets an own ZFS (directory). However, it can also be useful in other cases, like for software repositories stored on ZFS.

To create a snapshot manually, the following command can be used:

     zfs snapshot  home/user1@Tuesday

or if one wants to create it recursive for all child file systems:
     zfs snapshot -r home@Tuesday

Snapshots itself can be accessed then through the hidden directory .zfs/snapshot in the main ZFS directory for which the snapshot was created. In the above example, user1 could access the own snapshot through

     cd $HOME/.zfs/snapshot/Tuesday

This access will also work through NFS if the sharenfs feature of ZFS is used.

Since there is no automated snapshot possibility built into Linux/ZFSonLinux, one needs to implement an own solution for that. I wrote a script for such use case a while ago to be used on Solaris, but it can easily be used on Linux too. I will use that script and the example of home directories to show how snapshots work and can be used.

Let's assume a zpool "home" will be created and the user home directories for user1 and user2:

   zpool create home raidz2  sdb sdc sdd sde sdf spare sdg
    zfs set compression=on home
    zfs set xattr=sa home
    zfs create home/user1
    zfs create home/user2
    chown user1:users /home/user1
    chown user2:users /home/user2

In addition, one wants to have a snapshot every 15min for the last hour, an hourly snapshot for the last day, and a daily snapshot for the last month. Then a cronjob can be created like this:

    */15 * * * * /usr/bin/backup-zfs.sh home

If one doesn't want to have frequent snapshots every 15min, then substitute "*/15" with "00" which will only create the hourly/daily snapshots.
If a user goes then to $HOME/.zfs/snapshot, all snapshots are listed there and one can choose which one to use to restore a accidentally changed/deleted file. To restore a file, one can simply copy the file from a snapshot over to the live file system.

The script will do the following:

  • create a frequent snapshot as specified by the frequency of the cronjob, here every 15min
    • there will be 4 snapshots for the last hour: frequent-15, frequent-30, frequent-45, and frequent-00
  • at every full hour, the frequent-00 snapshot from the last hour will be renamed to an hourly snapshot
  • all other frequent snapshots will be overwritten during the next hour
  • at midnight the hourly snapshot from last midnight will be renamed to a daily snapshot
  • all other hourly snapshots will be overwritten during the next day
  • all daily snapshots will be overwritten during the next month
On the Solaris machine it was developed for, the script runs without problems and any manually interaction needed for about 500 user directories since about 5 years. 
The frequency specified in the cronjob should make sure that it is executed at the full hour, or the script needs to be adapted.

The permissions to access a snapshot are the same like for the original file system since it lives within the live file system.

Keep in mind that this is not a real backup, since it is lost as soon as the original file system is lost, for example due to broken disks or machine!
But it is a good way to let users restore their own files instead of having the sysadmin doing this manually and responding to service request emails for restoring files from a central backup.