ZFS snapshots on Linux

One great feature of ZFS is its possibility to have instant snapshots due to the general use of Copy-on-Write.
Such snapshots are a read-only view of the file system at the time the snapshots were created. Also, at the time of the snapshot creation, a snapshot does not take any disk space. Over time, only blocks that were changed on the original file system will take up space for the snapshot. This makes snapshots an ideal candidate for on-disk backups to recover from accidental file changes.
One of the best use cases for it is when ZFS is used for home directories where every user gets an own ZFS (directory). However, it can also be useful in other cases, like for software repositories stored on ZFS.

To create a snapshot manually, the following command can be used:

     zfs snapshot  home/user1@Tuesday

or if one wants to create it recursive for all child file systems:
 
     zfs snapshot -r home@Tuesday

Snapshots itself can be accessed then through the hidden directory .zfs/snapshot in the main ZFS directory for which the snapshot was created. In the above example, user1 could access the own snapshot through

     cd $HOME/.zfs/snapshot/Tuesday

This access will also work through NFS if the sharenfs feature of ZFS is used.


Since there is no automated snapshot possibility built into Linux/ZFSonLinux, one needs to implement an own solution for that. I wrote a script for such use case a while ago to be used on Solaris, but it can easily be used on Linux too. I will use that script and the example of home directories to show how snapshots work and can be used.

Let's assume a zpool "home" will be created and the user home directories for user1 and user2:

   zpool create home raidz2  sdb sdc sdd sde sdf spare sdg
    zfs set compression=on home
    zfs set xattr=sa home
    zfs create home/user1
    zfs create home/user2
    chown user1:users /home/user1
    chown user2:users /home/user2


In addition, one wants to have a snapshot every 15min for the last hour, an hourly snapshot for the last day, and a daily snapshot for the last month. Then a cronjob can be created like this:

    */15 * * * * /usr/bin/backup-zfs.sh home

If one doesn't want to have frequent snapshots every 15min, then substitute "*/15" with "00" which will only create the hourly/daily snapshots.
If a user goes then to $HOME/.zfs/snapshot, all snapshots are listed there and one can choose which one to use to restore a accidentally changed/deleted file. To restore a file, one can simply copy the file from a snapshot over to the live file system.

The script will do the following:

• create a frequent snapshot as specified by the frequency of the cronjob, here every 15min
• there will be 4 snapshots for the last hour: frequent-15, frequent-30, frequent-45, and frequent-00
• at every full hour, the frequent-00 snapshot from the last hour will be renamed to an hourly snapshot
• all other frequent snapshots will be overwritten during the next hour
• at midnight the hourly snapshot from last midnight will be renamed to a daily snapshot
• all other hourly snapshots will be overwritten during the next day
• all daily snapshots will be overwritten during the next month

On the Solaris machine it was developed for, the script runs without problems and any manually interaction needed for about 500 user directories since about 5 years. 

The frequency specified in the cronjob should make sure that it is executed at the full hour, or the script needs to be adapted.

The permissions to access a snapshot are the same like for the original file system since it lives within the live file system.

Keep in mind that this is not a real backup, since it is lost as soon as the original file system is lost, for example due to broken disks or machine!
But it is a good way to let users restore their own files instead of having the sysadmin doing this manually and responding to service request emails for restoring files from a central backup.

Yet another exercise in data recovery?

Just before the Christmas break, my main drive on my main PC - at home - seemed to start to fail (the kernel put it in read-only mode). Good thing we have backups, eh? They are all on portable hard drives, usually encrypted, and maintained with unison. No, they are not "in the cloud."

Surprisingly much of my data is WORM so what if there are differences between the backups? Was it due to those USB3 errors (caused a kernel panic, it did), hardware fault, or that fsck which seemed to discover a problem, or has the file actually changed? (And a big "boo, hiss" to applications that modify files just by opening them - yes, you know who you are.) In my case, I would prefer to re-checksum them all and compare against at least four of the backups. So I need a tool.

My Christmas programming challenge for this year (one should always have one) is then to create a new program to compare my backups. Probably there is one floating around out there, but my scheme - the naming scheme, when I do level zeros, increments, masters, replicas - is probably odd enough that it is useful having a bespoke tool.

On the grid we tend to checksum files as they are transferred. Preservation tools can be asked to "wake up" data every so often and re-check them. Ideally the backup check should quietly validate the checksums in the background as long as the backup drive is mounted.

Lesions learned?

I have a file which is wobbly. It is a good file, but it wobbled, and the letters got in the wrong places.

It is a file I was reading at home, on my home PC, from my home hard drive (no cloud stuff) and suddenly discovered had a lot of NUL characters in it, 3062 to be precise. Ewan suggested in the storage meeting that this might have been caused by an XFS crash - the PC is on a UPS but occasionally slightly experimental X clients lock it up and on a few occasions "security" features I foolishly left running had prevented me from logging in from my laptop and shutting it down cleanly.

Anyway, this file is part of a directory I synchronise to work, and surenuf, the file on my work PC was equally corrupted. And the corrupted file was backed up. Backups are not so helpful when the file you back up is corrupted...(the timestamp said December 2001). The file had been migrated from another older hard drive.

Now it so happens that this file was not precious, it was an RFC, so I could just get a fresh one. Comparing them, there seemed to be more changes...(and the NULs confused some diff tools) - the file seemed to have changed mysteriously in about eight places; in one 749 bytes were missing.

Back in the days, there were computer virus which would alter files on your hard drive... I don't believe that's what happened here - I invoke Hanlon's razor. Or a variant of it. ``Never attribute to malice that which can be adequately explained by 'stuff happens'''

It may be worth investigating my backups from back then - I was still using optical writeable CDs for my home PC backups back then, and I still have them - whether I can read them is another question.  But it seems I need to write a utility to background checksum my files for me.

For my precious data at home that I don't copy to work machines (when it's not work related), I have a stronger system, based on unison and multiple mutually independent backups, but the question is still whether I would have noticed the one change that should not propagate to the backup? Worth a test...