CentOS8.1 (teaching an old server new tricks)

At Edinburgh we recently retired some old(er) storage servers from our Tier2.

These storage servers consist of a Dell Poweredge R610 with 2 RAID controllers.

A PERC H710 internal for managing the boot disks and a PERC H800 external which connects to 3 MD1200 DAS.

We still maintain very similar hardware on our Tier2 as part of our grid Storage Element and we've been considering skipping CentOS7 and jumping straight to CentOS8 on this hardware.

As an experiment I recently attempted an update from CentOS7 to CentOS8 on a VM recently. Whilst this  can be done in a Saturday I wouldn't recommend it as the resulting OS isn't really production stable. With that in mind a clean install is the way to go.

This would be simple if it wasn't for 1 gotcha. The RAID controllers on these servers are out of support in CentOS8: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8-beta/html/8.0_beta_release_notes/removed_functionality

There is, however a solution: https://wiki.centos.org/Manuals/ReleaseNotes/CentOS8.1905#Known_Issues

This solution is to make use of the driver update images compiled by the nice people behind CentOS. These images allow the installer to pick up the 3rd party module and install it into your system as-if it were fully supported.

However following this proved to be problematic.

For CentOS8.0 the installer failed to identify the disks behind either controller, and on more than 1 occasion lead to either the kernel or the installer crashing.

(although admittedly we had also mis-configured this card in a minor way during this testing).

CentOS8.1 on the other hand worked out of the box with the following images:

> sha256sum dd-megaraid_sas-07.707.51.00-1.el8_1.elrepo.iso CentOS-8.1.1911-x86_64-dvd1.iso 

31a169d5eab1371893347c4d8482896e0fcc9b0a813b9210b1c0e77f68b09702 dd-megaraid_sas-07.707.51.00-1.el8_1.elrepo.iso 

3ee3f4ea1538e026fff763e2b284a6f20b259d91d1ad5688f5783a67d279423b CentOS-8.1.1911-x86_64-dvd1.iso

To get the installer to work you will need 2 drives to be connected to the system you want to deploy on.,

NB: Don't do the 'obvious thing' of burning the dud driver image to a disk. It will not work if you do this.

The following worked for me:

1. Burn your CentOS8.1 install image
2. Format a 2nd drive fat(msdos) and copy the dud driver dd-megaraid....iso to it
3. Startup the install image
4. Before you boot into the installer add the following to the boot command to boot the installer kernel:
   inst.dd 
5. Now wait for the installer to ask for the dud driver. Navigate to the disk containing the iso, select it and continue to boot.
6. Now perform your install as usual.

This isn't quite the full story as there was a minor gotcha which seemed to hit occasionally.

To fix the system not booting after a kernel update:

echo 'force_drivers+="megaraid_sas"' >>  /etc/dracut.conf.d/force_drivers.conf

The CentOS8 installer has changes in subtle but important ways compared to CentOS7 but nothing was scary other than on my first attempt being greeted by a GUI desktop thinking it was in New York because I was more focussed on my storage network configurations.

There are some unresolved problems with using non standard kernels such as kernel-ml and kernel-plus with 3rd party dkms drivers such as ZFS. However the out of the box kernel is 4.18 which is relatively recent and performs quite well form initial testing.

Next to test the performance of this as a mock storage node before we consider deploying these across the rest of our Tier2.

Help, ZFS ate my storage server (kernel segfaults on SL6)

At Edinburgh our storage test server (sl6) just updated it's kernel and had to reboot. Unfortunately it did not come back and suffered a kernel segfault during the reboot.

This was spotted to be during the filesystem mounting stage in the init scripts and specifically was caused by modprobe-ing the zfs module which had just been built by dkms.

The newer sl6 redhat kernels (2.6.32-754....) appear to have broken part of the abi used by the ZFS modules built by dkms.

The solution to fix this was found to be:

1. Reboot into the old kernel (anything with a version 2.6.32-696... or older)
2. check dkms for builds of the zfs/spl modules:   dkms status
3. run:   dkms uninstall zfs/0.7.9; dkms uninstall spl/0.7.9
4. make sure dkms removed this for ALL kernel versions (if needed run dkms uninstall zfs/0.7.9 -k 2.6.32-754) to remove it for a specific kernel
5. remove all traces of these modules:
    for i in /lib/modules/*; do
  for j in extra weak-updates; do
   for k in avl icp nvpair spl splat unicode zcommon zfs zpios ; do
     rm -r ${i}/${j}/${k};
   done;
  done;
 done
6. reboot back into the new kernel and reinstall zfs:
   dkms install zfs/0.7.9; dkms install spl/0.7.9
7. Check that you've saved everything important.
8. Now load the new modules: modprobe zfs
9. re-import your pools: zpool import -a

Alternatively: Remove all of the zfs modules (steps 3 and 5) before you reboot your system after installing the new kernel and then dkms will re-install everything on the next reboot.

For more info: https://github.com/zfsonlinux/zfs/issues/7704

TL;DR  When building new kernel modules dkms doesn't always rebuild external modules safely, make sure you remove these dependencies when you perform a kernel update so that everything is rebuilt safely

ZFS 0.7.6 release

ZFS on Linux 0.7.6 has now landed.

https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.7.6

For everyone running the 0.7.0-0.7.5 builds I would encourage people to look into updating as there are a few performance fixes associated with this build.
Large storage servers tend to have ample hardware, however if you're running this on systems with a small amount of RAM then the fixes may have a dramatic performance improvement.
Anecdotally I've also seen some improvements on a system which hosts a large number of smaller files which could be due to some fixes around the ZFS cache.


What if an update goes wrong?

I'm linking a draft of a flowchart I'm still working on to help debug what to do if a ZFS filesystem has disappeared after rebooting a machine:
https://drive.google.com/file/d/1hqY_qTfdpo-g_qApcP9nSknIm8X3wMwo/view?usp=sharing(Download and view offline for best results, there's a few things to check for!)

Hosting a large web-forum on ZFS (a case study)

Hosting a large web-forum on ZFS (a case study)

Over the course of last weekend I worked with a friend on deploying zfs across their infrastructure.

Their infrastructure in this case is a popular website written in php and administering to some 20,000+ users. They, like many gridpp sysadmins use CentOS for their back-end infrastructure. However due to being a regularly high profile target for attacks they have opted to run their systems using the latest kernel installed from the elrepo.
The infrastructure for this website is heavily docker orientated due to the (re)deployment advantages that this offers.
Due to problems with the complex workflow selinux has been set to permissive.

Data for the site was stored within a /data directory which stored both the main database for the site and files which are hosted by the site.
Prior to the use of zfs the storage used for this site was xfs.

The hardware used to run this site is a dedicated 8 intel cores, 32Gb RAM, 2 * 2Tb disks managed by soft-raid(mirror) and partitioned using lvm.

Installing zfs

Initially setting up ZFS couldn't have been easier. Install the correct rpm repo, update, install zfs and reboot:

yum update
yum install 
yum update
yum install zfs
reboot

Fixing zfs-dkms

As they are using the latest stable kernel they opted to install zfs using dkms which has pros/cons to the kmod install.

This unfortunately didn't work as it should have done (possibly due to a pending kernel update on reboot). After rebooting the following commands were needed to install the zfs driver:

dkms build spl/0.6.5.10
dkms build zfs/0.6.5.10
dkms install spl/0.6.5.10
dkms install zfs/0.6.5.10

This step triggered the rebuild and installation of the spl (solaris porting layer) and the zfs modules.
(Adding this to the initrd shouldn't be required but can probably be done as per usual once this has been build)

Migrating data to ZFS

The initial step was to migrate the storage backend and main database for the site. This storage is approximately 0.5Tb of data which was constructed of numerous files with an average file size close to 1Mb. The SQL database is approximately 50Gb in size containing most of the site data.

mv /data/webroot /data/webroot-bak
mv /data/sqlroot /data/sqlroot-bak
zfs create webrootzfs vgs/webrootzfs
zfs create sqlrootzfs vgs/sqlrootzfs
zfs set mountpoint=/data/webroot webrootzfs
zfs set mountpoint=/data/sqlroot sqlrootzfs
zfs set compression=lz4 webrootzfs
zfs set compression=lz4 sqlrootzfs
zfs set primarycache=metadata sqlrootzfs
zfs set secondarycache=none webrootzfs
zfs set secondarycache=none sqlrootzfs
zfs set recordsize=16k sqlrootzfs # Matches the db block size
rsync -avP /data/webroot-bak/* /data/webroot/
rsync -avP /data/sqlroot-bak/* /data/sqlroot/

After migrating these the site was then brought back up for approximately 24hr and there were no performance problems observed.

The webroot data which contained mainly user submitted files reached a compression level of about 1.1.
The sql database reached a compression level of about 2.4.

Given the increased performance of the site due to this migration it was decided 24hr later to investigate migrating the main website itself rather than just the backend.

Setting up systemd

The following systemd services and targets were enabled but rebooting the system has not (yet) been tested.

systemctl enable zfs.target
systemctl enable zfs-mount
systemctl start zfs-mount
systemctl enable zfs-import-cache
systemctl start zfs-import-cache 
systemctl enable zfs-share 
systemctl start zfs-share

Impact of using ZFS

A nice solution for this was found to already exist quite well. This is the zfs storage driver for docker.

https://docs.docker.com/engine/userguide/storagedriver/zfs-driver/

After this was setup the site was brought back online and the performance was notable.

Page load time for the site dropped from about 600ms to 300ms. That is a 50% drop in page load time entirely due to replacing the backend storage with zfs.
This was with the ARC cache running with a 95% hit rate.

Problems Encountered

Unfortunately about 30min of running after of migrating the docker service to use ZFS the site fell over.
(page load times increased to multiple seconds and the backend server load spiked.)

Upon initial inspection it was discovered that the zfs arc cache had dropped to 32M (almost absolute minimum) and the arc-reclaim process was consuming 100% of 1 CPU.

The ZFS arc cache maximum was increased to 10Gb but the cache refused to increase.

echo 10737418240 > /sys/module/zfs/parameters/zfs_arc_max

Increasing the minimum forced the arc cache to increase however the arc-reclaim process still was consuming 1 CPU core.

Fixing the Problems

A better workaround was found to be to disable the transparent_hugepage using:

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

This stopped the arc-reclaim process from consuming 100% CPU as well as triggering the arc cache to start regrowing.
(For the interested this has been reported: https://github.com/zfsonlinux/zfs/issues/4869)

Summary of Tweaks made

A summary of some of the optimizations applied to these pools are:

# ZFS settings
zfs set compression=lz4 webrootzfs # Enable best compression
zfs set compression=lz4 sqlrootzfs # Enable best compression 
zfs set primarycache=all # This is default
zfs set primarycache=metadata sqlrootzfs # Don't store DB in cache
zfs set secondarycache=none webrootzfs # Not using l2arc
zfs set secondarycache=none sqlrootzfs # Not using l2arc
zfs set recordsize=16k sqlrootzfs # Matches the db block size

# Settings changed through /sys
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo 10737418240 > /sys/module/zfs/parameters/zfs_arc_max # 10Gb max
echo 4294967296 > /sys/module/zfs/parameters/zfs_arc_min # 4Gb min

# repeat the following for /sys/block/sda and /sys/block/sdb
echo 4096 > /sys/block/sda/queue/nr_requests
echo 0 > /sys/block/sda/queue/iosched/front_merges
echo noop > /sys/block/sda/queue/scheduler
echo 150 > /sys/block/sda/queue/iosched/read_expire
echo 1500 > /sys/block/sda/queue/iosched/write_expire
echo 4096 > /sys/block/sda/queue/nr_requests
echo 4096 > /sys/block/sda/queue/read_ahead_kb
echo 1 > /sys/block/sda/queue/iosched/fifo_batch
echo 16384 > /sys/block/sda/queue/max_sectors_kb


Additionally for the docker-zfs pool:

zfs set primarycache=all zpool-docker
zfs set secondarycache=none zpool-docker
zfs set compression=lz4 zpool-docker

All docker containers built using this engine inherit these properties from the base pool zpool-docker however, a remove/rebuild will be needed to take advantage of settings such as compression.

Current storage scope within GridPP

With a new project year upon us, I decided to review which site's storage support is used used by the WLCG VOs, what SRMs are used and the file systems used. On that last part , we now don't just have filesystems but also object stores with the usage of CEPH. Other filesystems are XFS, ZFS, HDFS, Spectrum Scale (or the artist formerly known as GPFS), and Lustre.

In terms of Storage elements/systems we have DPM, dCache, Castor, classic SE, stand alone xrootd, and stand alone gsiftp services. When it comes to the regional T2s and who they are used by, the following helps.

I thought about embedding SE system into the font used for each site, but thought that was too much overlay of information.

ZFS snapshots on Linux

One great feature of ZFS is its possibility to have instant snapshots due to the general use of Copy-on-Write.
Such snapshots are a read-only view of the file system at the time the snapshots were created. Also, at the time of the snapshot creation, a snapshot does not take any disk space. Over time, only blocks that were changed on the original file system will take up space for the snapshot. This makes snapshots an ideal candidate for on-disk backups to recover from accidental file changes.
One of the best use cases for it is when ZFS is used for home directories where every user gets an own ZFS (directory). However, it can also be useful in other cases, like for software repositories stored on ZFS.

To create a snapshot manually, the following command can be used:

     zfs snapshot  home/user1@Tuesday

or if one wants to create it recursive for all child file systems:
 
     zfs snapshot -r home@Tuesday

Snapshots itself can be accessed then through the hidden directory .zfs/snapshot in the main ZFS directory for which the snapshot was created. In the above example, user1 could access the own snapshot through

     cd $HOME/.zfs/snapshot/Tuesday

This access will also work through NFS if the sharenfs feature of ZFS is used.


Since there is no automated snapshot possibility built into Linux/ZFSonLinux, one needs to implement an own solution for that. I wrote a script for such use case a while ago to be used on Solaris, but it can easily be used on Linux too. I will use that script and the example of home directories to show how snapshots work and can be used.

Let's assume a zpool "home" will be created and the user home directories for user1 and user2:

   zpool create home raidz2  sdb sdc sdd sde sdf spare sdg
    zfs set compression=on home
    zfs set xattr=sa home
    zfs create home/user1
    zfs create home/user2
    chown user1:users /home/user1
    chown user2:users /home/user2


In addition, one wants to have a snapshot every 15min for the last hour, an hourly snapshot for the last day, and a daily snapshot for the last month. Then a cronjob can be created like this:

    */15 * * * * /usr/bin/backup-zfs.sh home

If one doesn't want to have frequent snapshots every 15min, then substitute "*/15" with "00" which will only create the hourly/daily snapshots.
If a user goes then to $HOME/.zfs/snapshot, all snapshots are listed there and one can choose which one to use to restore a accidentally changed/deleted file. To restore a file, one can simply copy the file from a snapshot over to the live file system.

The script will do the following:

• create a frequent snapshot as specified by the frequency of the cronjob, here every 15min
• there will be 4 snapshots for the last hour: frequent-15, frequent-30, frequent-45, and frequent-00
• at every full hour, the frequent-00 snapshot from the last hour will be renamed to an hourly snapshot
• all other frequent snapshots will be overwritten during the next hour
• at midnight the hourly snapshot from last midnight will be renamed to a daily snapshot
• all other hourly snapshots will be overwritten during the next day
• all daily snapshots will be overwritten during the next month

On the Solaris machine it was developed for, the script runs without problems and any manually interaction needed for about 500 user directories since about 5 years. 

The frequency specified in the cronjob should make sure that it is executed at the full hour, or the script needs to be adapted.

The permissions to access a snapshot are the same like for the original file system since it lives within the live file system.

Keep in mind that this is not a real backup, since it is lost as soon as the original file system is lost, for example due to broken disks or machine!
But it is a good way to let users restore their own files instead of having the sysadmin doing this manually and responding to service request emails for restoring files from a central backup.

File system tests

Since there is interest in filesystem test, I put the script I used for the ZFS/Ext4/XFS tests on a web server. If you test file systems for Grid storage purpose, feel free to give it a try.
It can of course also be used by anyone else, but this test is not doing any random read/writes.

In general, it would be good to run this test (or any other test) under 3 different scenarios when using raid systems:

1. normal working raid system
2. degraded raid system
3. rebuild of raid system

The script needs 2 areas:

1. one where you have files that are read during the read tests, and 
2. one where you want write files to. This write area should have no compression since the writes come from /dev/zero.                                                                                                                                                                                                                                                                                          

By default it is doing reads over all specified files, writes to large files, and writes to small files.
For the reads, first it is doing a sequential read for all files, and in a second pass it reads files in parallel for the same set of files.
For the writes, it is doing a sequential write first and in a second pass it is writing in parallel do different files.   That is the same for the writing of large files and of small files.

After each read/write pass there is a cache flush and also each single write issues a file sync after each file is written to make sure that the time is measured to really write a file to disk.


The script needs 3 parameters:

1. location of a text files that contains the file name including absolute path for all files that you want to include in the read tests
2. a name used as description for your test, it can be used to distinguish between different tests (e.g. ZFS-raid2-12 disks or ZFS-raidz3-12disks)
3. absolute path to an area where the write test can write it files too; this area should have no compression enabled

The parameters inside the script, like number of parallel read/writes and  file sizes, can easily be configured. By default about 5TB space are needed for the write tests.

The script itself can be downloaded here. 

ZFS auto mount for CentOS7

When testing ZFS installs on servers running on CentOS7.3, it can happen that ZFS is not available after a restart. After some testing this seems to be related to systemd and probably affects other systemd Linux distributions too.

What I used were ZFS installs using different versions of ZFS on Linux. After looking into the system setup, I noticed that by default ZFS is just disabled. Doing the following solved the problem on the machines I tested:

systemctl enable zfs.target

systemctl start zfs.target

systemctl enable zfs-import-cache.service

systemctl enable zfs-mount.service

systemctl enable zfs-share.service 

This solved all auto mount issues for me on the CentOS systems.

 Note: At least when using the latest version 0.6.5.8, one can also use the following command as explained on the ZFSonLinux web page:

systemctl preset zfs-import-cache zfs-import-scan zfs-mount zfs-share zfs-zed zfs.target


Everyone who is upgrading to the latest version should also have a look to the ZFS on Linux web page since the repository address has changed. While it should have updated it automatically, if you haven't run any updates since some month, then it can't get the new repository automatically.

ZFS compression for LHC experiments data

An interesting feature of ZFS is that it supports transparent compression. Different to typical file compression, ZFS compression works on the record size/block size that it writes (which is variable in ZFS depending on the data and file size itself). Since it is important to have a fast compression/decompression algorithm to reduce the overhead compared to file access without compression, it can not be expected to get compression results similar to for example bzip in its highest compression level.  Also, the data files of the LHC experiments are ROOT files which already store data in a compressed format.

Therefore, I was not expecting any benefit of enabling compression on our servers, but since the newly implemented algorithm LZ4 has nearly no overhead even for non-compressible data, it shouldn't hurt to enable it.  Especially since our storage servers have Dual-CPUs with 12 cores each, running most of the time idle.

After enabling the default lz4 compression on 4 machines that were already migrated to ZFS and copying data on it, the first compression result looks like this:


NAME       SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank-2TB  32.5T  8.73T  23.8T         -    15%    26%  1.00x  ONLINE  -
tank-8TB   116T  24.0T  92.0T         -    10%    20%  1.00x  ONLINE  -

NAME                    PROPERTY       VALUE  SOURCE
tank-2TB                compressratio  1.03x  -
tank-2TB/gridstorage01  compressratio  1.03x  -
tank-2TB/gridstorage02  compressratio  1.03x  -
tank-2TB/gridstorage03  compressratio  1.03x  -
tank-2TB/gridstorage04  compressratio  1.03x  -
tank-8TB                compressratio  1.03x  -
tank-8TB/gridstorage01  compressratio  1.03x  -
tank-8TB/gridstorage02  compressratio  1.03x  -
tank-8TB/gridstorage03  compressratio  1.03x  -
tank-8TB/gridstorage04  compressratio  1.03x  -
tank-8TB/gridstorage05  compressratio  1.04x  -
tank-8TB/gridstorage06  compressratio  1.03x  -
tank-8TB/gridstorage07  compressratio  1.03x  -
tank-8TB/gridstorage08  compressratio  1.03x  -
tank-8TB/gridstorage09  compressratio  1.03x  -
tank-8TB/gridstorage10  compressratio  1.03x  -
tank-8TB/gridstorage11  compressratio  1.03x  -

NAME       SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT

tank-2TB  32.5T  8.45T  24.0T         -    11%    26%  1.00x  ONLINE  -

tank-8TB   116T  24.1T  91.9T         -     7%    20%  1.00x  ONLINE  -

NAME                    PROPERTY       VALUE  SOURCE

tank-2TB                compressratio  1.03x  -

tank-2TB/gridstorage01  compressratio  1.03x  -

tank-2TB/gridstorage02  compressratio  1.03x  -

tank-2TB/gridstorage03  compressratio  1.03x  -

tank-2TB/gridstorage04  compressratio  1.04x  -

tank-8TB                compressratio  1.03x  -

tank-8TB/gridstorage01  compressratio  1.03x  -

tank-8TB/gridstorage02  compressratio  1.03x  -

tank-8TB/gridstorage03  compressratio  1.03x  -

tank-8TB/gridstorage04  compressratio  1.03x  -

tank-8TB/gridstorage05  compressratio  1.03x  -

tank-8TB/gridstorage06  compressratio  1.03x  -

tank-8TB/gridstorage07  compressratio  1.03x  -

tank-8TB/gridstorage08  compressratio  1.03x  -

tank-8TB/gridstorage09  compressratio  1.03x  -

tank-8TB/gridstorage10  compressratio  1.03x  -

tank-8TB/gridstorage11  compressratio  1.03x  -

NAME       SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT

tank-4TB   127T  9.05T   118T         -     3%     7%  1.00x  ONLINE  -

NAME                    PROPERTY       VALUE  SOURCE

tank-4TB                compressratio  1.03x  -

tank-4TB/gridstorage01  compressratio  1.03x  -

tank-4TB/gridstorage02  compressratio  1.03x  -

tank-4TB/gridstorage03  compressratio  1.04x  -

tank-4TB/gridstorage04  compressratio  1.02x  -

tank-4TB/gridstorage05  compressratio  1.03x  -

tank-4TB/gridstorage06  compressratio  1.03x  -

tank-4TB/gridstorage07  compressratio  1.03x  -

tank-4TB/gridstorage08  compressratio  1.03x  -

tank-4TB/gridstorage09  compressratio  1.03x  -

tank-4TB/gridstorage10  compressratio  1.04x  -

tank-4TB/gridstorage11  compressratio  1.03x  -

tank-4TB/gridstorage12  compressratio  1.02x  -

tank-4TB/gridstorage13  compressratio  1.03x  -

tank-4TB/gridstorage14  compressratio  1.03x  -

NAME       SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT

tank-2TB  63.5T  15.4T  48.1T         -    11%    24%  1.00x  ONLINE  -

NAME                    PROPERTY       VALUE  SOURCE

tank-2TB                compressratio  1.03x  -

tank-2TB/gridstorage01  compressratio  1.03x  -

tank-2TB/gridstorage02  compressratio  1.04x  -

tank-2TB/gridstorage03  compressratio  1.03x  -

tank-2TB/gridstorage04  compressratio  1.03x  -

tank-2TB/gridstorage05  compressratio  1.03x  -

tank-2TB/gridstorage06  compressratio  1.03x  -

tank-2TB/gridstorage07  compressratio  1.03x  -

Although there is not much data stored so far on each of the machines, this means we can still reduce the used disk space by some percent, 2-4% here depending on the file system and the data on it.
We have a bit more than 1PB disk storage in total on our site and the servers with 2TB disks provide about 50TB usable storage each. If we can get 4% compression for all the data, that would mean we could get nearly the space provided by one of the 2TB-disk servers additionally for free, without the cost of a new machine, power, extra disks,.... ! And that's just with the default compression while the compression level could also be tuned in ZFS...
This saving could be even bigger if we consider that in the future sites will also store more non-LHC data, like for LSST, which use a different and maybe uncompressed file format.
Another positive aspect of compression is that it reduces disk I/O since it needs to read less data blocks from disk.

It will be interesting to see how the compression rate will be after all our servers have been switch over  to ZFS.

Setting up of a ZFS based storage server

As it was previously found that ZFS has a good performance in our use case which is even better than the hardware raid performance, new storage servers on our site to be used within GridPP will use ZFS as storage file system in the future.
In this post, I will show how a server for that purpose can easily be setup.  The previous posts which also mention details about the used hardware can be found here, here, and here.

The purpose of this storage server is to be used for LHC data storage which is mostly consistent of GB sized files. At the time of using these data files as input for user jobs, typically the whole  file is copied over to the local node where the user job runs. That means that the configuration needs to deal with large sequential read and writes, but not with small random block access.

The typical hardware configuration of  the storage servers we have is:

• Server with PERC H700 and/or H800 hardware raid controller
• 36 disk slots available
• on some server available through 3 external PowerVault MD-devices (3x12 disks)
• on some servers available through 2 external PowerVault MD-devices (2x12disks) and 12 internal storage disks
• 10Gbps network interface
• Dual-CPU (8 or 12 physical cores on each)
• between 12GB and 64GB of RAM 

In this blog post, as I did before, I will describe the ZFS setup based on a machine with 12 internal disks (2TB disks on H700) and 24 external disks (17x8TB + 7x2TB  on H800). The machine is already setup with SL6 and has the typical GridPP software (DPM clients, xrootd, httpd,...) installed.

Preparing the disks

Since both raid controllers don't support JBOD, first single raid0 devices have to be created. To find out which disks are available and can be used, omreport can be used:

[root@pool7 ~]# omreport storage pdisk controller=0|grep -E "^ID|Capacity"

ID                              : 0:0:0

Capacity                        : 1,862.50 GB (1999844147200 bytes)

ID                              : 0:0:1

Capacity                        : 1,862.50 GB (1999844147200 bytes)

ID                              : 0:0:2

Capacity                        : 1,862.50 GB (1999844147200 bytes)

ID                              : 0:0:3

Capacity                        : 1,862.50 GB (1999844147200 bytes)

ID                              : 0:0:4

Capacity                        : 1,862.50 GB (1999844147200 bytes)

ID                              : 0:0:5

Capacity                        : 1,862.50 GB (1999844147200 bytes)

ID                              : 0:0:6

Capacity                        : 1,862.50 GB (1999844147200 bytes)

ID                              : 0:0:7

Capacity                        : 1,862.50 GB (1999844147200 bytes)

ID                              : 0:0:8

Capacity                        : 1,862.50 GB (1999844147200 bytes)

ID                              : 0:0:9

Capacity                        : 1,862.50 GB (1999844147200 bytes)

ID                              : 0:0:10

Capacity                        : 1,862.50 GB (1999844147200 bytes)

ID                              : 0:0:11

Capacity                        : 1,862.50 GB (1999844147200 bytes)

ID                              : 0:0:12

Capacity                        : 278.88 GB (299439751168 bytes)

ID                              : 0:0:13

Capacity                        : 278.88 GB (299439751168 bytes)

The disks 0:0:12 and 0:0:13 are the system disks in a mirrored configuration and shouldn't be touched. The disks 0:0:0 to 0:0:11 can be converted to single raid0 using omconfig:

for i in $(seq 0 11); 

do 

  omconfig storage controller controller=0 action=createvdisk raid=r0 size=max pdisk=0:0:$i; 

done

The same procedure has to be repeated for the second controller.

After that, the disks are available to the system and to find out which are the 2TB and which are the 8TB disks, lsblk can be used:

[root@pool7 ~]# lsblk |grep disk

sda      8:0    0 278.9G  0 disk 

sdb      8:16   0   1.8T  0 disk 

sdc      8:32   0   1.8T  0 disk 

sdd      8:48   0   1.8T  0 disk 

sde      8:64   0   1.8T  0 disk 

sdf      8:80   0   1.8T  0 disk 

sdg      8:96   0   1.8T  0 disk 

sdh      8:112  0   1.8T  0 disk 

sdi      8:128  0   1.8T  0 disk 

sdj      8:144  0   1.8T  0 disk 

sdk      8:160  0   1.8T  0 disk 

sdl      8:176  0   1.8T  0 disk 

sdm      8:192  0   1.8T  0 disk 

sdn      8:208  0   7.3T  0 disk 

sdo      8:224  0   7.3T  0 disk 

sdp      8:240  0   7.3T  0 disk 

sdq     65:0    0   7.3T  0 disk 

sdr     65:16   0   7.3T  0 disk 

sds     65:32   0   7.3T  0 disk 

sdt     65:48   0   7.3T  0 disk 

sdu     65:64   0   7.3T  0 disk 

sdv     65:80   0   7.3T  0 disk 

sdw     65:96   0   7.3T  0 disk 

sdx     65:112  0   7.3T  0 disk 

sdy     65:128  0   7.3T  0 disk 

sdz     65:144  0   7.3T  0 disk 

sdaa    65:160  0   7.3T  0 disk 

sdab    65:176  0   1.8T  0 disk 

sdac    65:192  0   7.3T  0 disk 

sdad    65:208  0   1.8T  0 disk 

sdae    65:224  0   1.8T  0 disk 

sdaf    65:240  0   7.3T  0 disk 

sdag    66:0    0   1.8T  0 disk 

sdah    66:16   0   1.8T  0 disk 

sdai    66:32   0   7.3T  0 disk 

sdaj    66:48   0   1.8T  0 disk 

sdak    66:64   0   1.8T  0 disk 

/dev/sda is the system disk and shouldn't be touch, but all the other disks can be used for the storage setup.

ZFS installation

The current version of ZFS can be downloaded from the ZFS on Linux web page. Depending on the used distribution, there are also instructions on how to install ZFS through the package management. In the worst case, one can download the source code and compile on the own system.   

Since we use SL which is RH based, we can follow the instructions provided on the page .

After the installation of ZFS through yum, the module needs to be loaded using modprobe zfs to continue without a reboot.   

To have file systems based on zfs automounted at system start, unfortunately also selinux options need to be changed. In the selinux config file, in our case at /etc/sysconfig/selinux, we need to change "SELINUX=enforcing" to at least "SELINUX=permissive"

This will probably be needed as long as zfs is not part of the RH distribution and zfs will not be recognized by selinux as a valid file system. More about this issue can be found here.

ZFS storage setup

Since we have the ZFS driver installed and the disks prepared now, we can continue to setup the storage pool.  

In this example, we create 2 different storage pools - one for the 2TB disks and one for the 8TB disks - as a good compromise between possible IOPS and available space. For ZFS, it doesn't matter if the disks within one storage pool are connected through the same controller or through different ones, like in our case for the 2TB disks. For the configuration, it's decided to use raidz2 which has 2 redundancy disks similar to raid6. Also, one disk of each kind will be used as hot spare.

To do so, we need to find all disks of a given kind in the system and creating a storage pool for these disks using zpool create :

[root@pool7 ~]# lsblk |grep 1.8T

sdb      8:16   0   1.8T  0 disk 

sdc      8:32   0   1.8T  0 disk 

sdd      8:48   0   1.8T  0 disk 

sde      8:64   0   1.8T  0 disk 

sdf      8:80   0   1.8T  0 disk 

sdg      8:96   0   1.8T  0 disk 

sdh      8:112  0   1.8T  0 disk 

sdi      8:128  0   1.8T  0 disk 

sdj      8:144  0   1.8T  0 disk 

sdk      8:160  0   1.8T  0 disk 

sdl      8:176  0   1.8T  0 disk 

sdm      8:192  0   1.8T  0 disk 

sdab    65:176  0   1.8T  0 disk 

sdad    65:208  0   1.8T  0 disk 

sdae    65:224  0   1.8T  0 disk 

sdag    66:0    0   1.8T  0 disk 

sdah    66:16   0   1.8T  0 disk 

sdaj    66:48   0   1.8T  0 disk 

sdak    66:64   0   1.8T  0 disk 

[root@pool7 ~]# zpool create -f tank-2TB raidz2 sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdab sdad sdae sdag sdah sdaj spare sdak

[root@pool7 ~]# lsblk |grep 7.3T

sdn       8:208  0   7.3T  0 disk 

sdo       8:224  0   7.3T  0 disk 

sdp       8:240  0   7.3T  0 disk 

sdq      65:0    0   7.3T  0 disk 

sdr      65:16   0   7.3T  0 disk 

sds      65:32   0   7.3T  0 disk 

sdt      65:48   0   7.3T  0 disk 

sdu      65:64   0   7.3T  0 disk 

sdv      65:80   0   7.3T  0 disk 

sdw      65:96   0   7.3T  0 disk 

sdx      65:112  0   7.3T  0 disk 

sdy      65:128  0   7.3T  0 disk 

sdz      65:144  0   7.3T  0 disk 

sdaa     65:160  0   7.3T  0 disk 

sdac     65:192  0   7.3T  0 disk 

sdaf     65:240  0   7.3T  0 disk 

sdai     66:32   0   7.3T  0 disk 

[root@pool7 ~]# zpool create -f tank-8TB raidz2 sdn sdo sdp sdq sdr sds sdt sdu sdv sdw sdx sdy sdz sdaa sdac sdaf spare sdai

After the zpool create commands, the storage is setup in a raid configuration, a file system created on top of it, and mounted under /tank-2TB and /tank-8TB. There are no additional commands needed and all is available within seconds.   

At this point the system looks like:

[root@pool7 ~]# mount|grep zfs

tank-2TB on /tank-2TB type zfs (rw)

tank-8TB on /tank-8TB type zfs (rw)

[root@pool7 ~]# zpool status

  pool: tank-2TB

 state: ONLINE

  scan: none requested

config:

        NAME        STATE     READ WRITE CKSUM

        tank-2TB    ONLINE       0     0     0

          raidz2-0  ONLINE       0     0     0

            sdb     ONLINE       0     0     0

            sdc     ONLINE       0     0     0

            sdd     ONLINE       0     0     0

            sde     ONLINE       0     0     0

            sdf     ONLINE       0     0     0

            sdg     ONLINE       0     0     0

            sdh     ONLINE       0     0     0

            sdi     ONLINE       0     0     0

            sdj     ONLINE       0     0     0

            sdk     ONLINE       0     0     0

            sdl     ONLINE       0     0     0

            sdm     ONLINE       0     0     0

            sdab    ONLINE       0     0     0

            sdad    ONLINE       0     0     0

            sdae    ONLINE       0     0     0

            sdag    ONLINE       0     0     0

            sdah    ONLINE       0     0     0

            sdaj    ONLINE       0     0     0

        spares

          sdak      AVAIL   

errors: No known data errors

  pool: tank-8TB

 state: ONLINE

  scan: none requested

config:

        NAME        STATE     READ WRITE CKSUM

        tank-8TB    ONLINE       0     0     0

          raidz2-0  ONLINE       0     0     0

            sdn     ONLINE       0     0     0

            sdo     ONLINE       0     0     0

            sdp     ONLINE       0     0     0

            sdq     ONLINE       0     0     0

            sdr     ONLINE       0     0     0

            sds     ONLINE       0     0     0

            sdt     ONLINE       0     0     0

            sdu     ONLINE       0     0     0

            sdv     ONLINE       0     0     0

            sdw     ONLINE       0     0     0

            sdx     ONLINE       0     0     0

            sdy     ONLINE       0     0     0

            sdz     ONLINE       0     0     0

            sdaa    ONLINE       0     0     0

            sdac    ONLINE       0     0     0

            sdaf    ONLINE       0     0     0

        spares

          sdai      AVAIL   

errors: No known data errors

[root@pool7 ~]# zpool list

NAME       SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT

tank-2TB  32.5T   153K  32.5T         -     0%     0%  1.00x  ONLINE  -

tank-8TB   116T   153K   116T         -     0%     0%  1.00x  ONLINE  -

[root@pool7 ~]# 

[root@pool7 ~]# zfs list

NAME       USED  AVAIL  REFER  MOUNTPOINT

tank-2TB   120K  28.0T  40.0K  /tank-2TB

tank-8TB   117K  97.7T  39.1K  /tank-8TB

[root@pool7 ~]# 

[root@pool7 ~]# df -h|grep tank

tank-2TB         28T     0   28T   0% /tank-2TB

tank-8TB         98T     0   98T   0% /tank-8TB

Setting additional filesystem properties

Since there is lz4 available as compression algorithm which has a very small impact on performance, we can enable compression on our storage. This has probably not a large impact on the storage of LHC data, but could lower the storage space for non-LHC experiments that will be supported in the near future.
In addition, the storage of xattr will also be changed to a similar behaviour like in ext4.
Also since we have a spare configured in our pools, we need to activate auto replacement in failure cases making it a hot spare. An interesting feature of ZFS is also to grow the pool size if the disks are replaced by new disks with a larger capacity. This needs to be done for all disks within one vdev to have an effect, but can be done one by one over time.

[root@pool7 ~]# zfs set compression=lz4 tank-2TB
[root@pool7 ~]# zfs set compression=lz4 tank-8TB

[root@pool7 ~]# zpool set autoreplace=on tank-2TB
[root@pool7 ~]# zpool set autoreplace=on tank-8TB

[root@pool7 ~]# zpool set autoexpand=on tank-2TB
[root@pool7 ~]# zpool set autoexpand=on tank-8TB

[root@pool7 ~]# zfs set relatime=on tank-2TB
[root@pool7 ~]# zfs set relatime=on tank-8TB

[root@pool7 ~]# zfs set xattr=sa tank-2TB
[root@pool7 ~]# zfs set xattr=sa tank-8TB

Changing disk identification

Using the disk identification by letters, like sdb or sdc, is easy to handle and good to setup a pool. However, the order how disks are identified could be changed on a reboot and also will change if the disks need to be rearranged on the server, for example after replacing one of the external MD devices.

While in such cases ZFS should still be able to identify the disks belonging to the same pool and import the pool, it is better to use the disk IDs to identify disks. To change this behaviour, we only need to export the pool and import using the disk IDs:

[root@pool7 ~]# zpool export -a

[root@pool7 ~]# zpool import -d /dev/disk/by-id tank-8TB

[root@pool7 ~]# zpool import -d /dev/disk/by-id tank-2TB

Making the space available to DPM

Traditionally, the available space on a large vdev was divided into smaller parts by creating partitions in fdisk. In ZFS however this can be done directly on top of the just created pool. All properties set for the top level ZFS will be distributed to the new zfs too. There will be no need to set the compression property or other properties again, except if one wants to have different properties than before.

A new file system is created by using zfs create :

[root@pool7 ~]# zfs create -o refreservation=9T tank-8TB/gridstorage01

[root@pool7 ~]# zfs create -o refreservation=9T tank-8TB/gridstorage02

[root@pool7 ~]# zfs create -o refreservation=9T tank-8TB/gridstorage03

[root@pool7 ~]# zfs create -o refreservation=9T tank-8TB/gridstorage04

[root@pool7 ~]# zfs create -o refreservation=9T tank-8TB/gridstorage05

[root@pool7 ~]# zfs create -o refreservation=9T tank-8TB/gridstorage06

[root@pool7 ~]# zfs create -o refreservation=9T tank-8TB/gridstorage07

[root@pool7 ~]# zfs create -o refreservation=9T tank-8TB/gridstorage08

[root@pool7 ~]# zfs create -o refreservation=9T tank-8TB/gridstorage09

[root@pool7 ~]# zfs create -o refreservation=9T tank-8TB/gridstorage10

[root@pool7 ~]# zfs list

NAME                     USED  AVAIL  REFER  MOUNTPOINT

tank-2TB                 144K  28.0T  40.0K  /tank-2TB

tank-8TB                90.0T  7.67T  41.7K  /tank-8TB

tank-8TB/gridstorage01     9T  16.7T  39.1K  /tank-8TB/gridstorage01

tank-8TB/gridstorage02     9T  16.7T  39.1K  /tank-8TB/gridstorage02

tank-8TB/gridstorage03     9T  16.7T  39.1K  /tank-8TB/gridstorage03

tank-8TB/gridstorage04     9T  16.7T  39.1K  /tank-8TB/gridstorage04

tank-8TB/gridstorage05     9T  16.7T  39.1K  /tank-8TB/gridstorage05

tank-8TB/gridstorage06     9T  16.7T  39.1K  /tank-8TB/gridstorage06

tank-8TB/gridstorage07     9T  16.7T  39.1K  /tank-8TB/gridstorage07

tank-8TB/gridstorage08     9T  16.7T  39.1K  /tank-8TB/gridstorage08

tank-8TB/gridstorage09     9T  16.7T  39.1K  /tank-8TB/gridstorage09

tank-8TB/gridstorage10     9T  16.7T  39.1K  /tank-8TB/gridstorage10

[root@pool7 ~]# zfs create -o refreservation=7.66T tank-8TB/gridstorage11

[root@pool7 ~]# zfs list

NAME                     USED  AVAIL  REFER  MOUNTPOINT

tank-2TB                 144K  28.0T  40.0K  /tank-2TB

tank-8TB                97.7T  9.91G  41.7K  /tank-8TB

tank-8TB/gridstorage01     9T  9.01T  39.1K  /tank-8TB/gridstorage01

tank-8TB/gridstorage02     9T  9.01T  39.1K  /tank-8TB/gridstorage02

tank-8TB/gridstorage03     9T  9.01T  39.1K  /tank-8TB/gridstorage03

tank-8TB/gridstorage04     9T  9.01T  39.1K  /tank-8TB/gridstorage04

tank-8TB/gridstorage05     9T  9.01T  39.1K  /tank-8TB/gridstorage05

tank-8TB/gridstorage06     9T  9.01T  39.1K  /tank-8TB/gridstorage06

tank-8TB/gridstorage07     9T  9.01T  39.1K  /tank-8TB/gridstorage07

tank-8TB/gridstorage08     9T  9.01T  39.1K  /tank-8TB/gridstorage08

tank-8TB/gridstorage09     9T  9.01T  39.1K  /tank-8TB/gridstorage09

tank-8TB/gridstorage10     9T  9.01T  39.1K  /tank-8TB/gridstorage10

tank-8TB/gridstorage11  7.66T  7.67T  39.1K  /tank-8TB/gridstorage11

Here a new property is set for each file of the new file systems - refreservation - which reserves the specified space for this particular file system, making sure this space is guaranteed. This is different to setting a quota which limits the space only to an upper limit.  However, to make sure the specified space is not exceeded in our case, also a quota of the same size should be specified. For the last file system in each pool, a larger amount could be specified which makes sure that all the space that can't be used by the other file systems due to quota limitations, will be used here.
[root@pool7 ~]# zfs set refquota=7T tank-2TB/gridstorage01
[root@pool7 ~]# zfs set refquota=7T tank-2TB/gridstorage02
[root@pool7 ~]# zfs set refquota=7T tank-2TB/gridstorage03
[root@pool7 ~]# zfs set refquota=7T tank-2TB/gridstorage04
[root@pool7 ~]# zfs set refquota=9T tank-8TB/gridstorage01
[root@pool7 ~]# zfs set refquota=9T tank-8TB/gridstorage02
[root@pool7 ~]# zfs set refquota=9T tank-8TB/gridstorage03
[root@pool7 ~]# zfs set refquota=9T tank-8TB/gridstorage04
[root@pool7 ~]# zfs set refquota=9T tank-8TB/gridstorage05
[root@pool7 ~]# zfs set refquota=9T tank-8TB/gridstorage06
[root@pool7 ~]# zfs set refquota=9T tank-8TB/gridstorage07
[root@pool7 ~]# zfs set refquota=9T tank-8TB/gridstorage08
[root@pool7 ~]# zfs set refquota=9T tank-8TB/gridstorage09
[root@pool7 ~]# zfs set refquota=9T tank-8TB/gridstorage10
[root@pool7 ~]# zfs set refquota=9T tank-8TB/gridstorage11

After that, the storage setup is finished and the just created file systems are already mounted and should be made available to the DPM user:
[root@pool7 ~]# chown -R dpmmgr:users /tank-2TB
[root@pool7 ~]# chown -R dpmmgr:users /tank-8TB

That was the last step needed on the storage server and the new file systems can be added to the DPM head node like any other file system before.

ZFS configuration options

To customize the ZFS behaviour, 2 main config files are available  - /etc/sysconfig/zfs and /etc/zfs/zed.d/zed.rc. I will not go into detail here about these 2 files, but if you want to setup your own ZFS based storage then have a look here. The options within are mainly self explaining, for example you can specify where to send email for disk problems and under which circumstances. 

As final section, I want to mention 2 very useful commands - zpool history and zpool iostat.

With the first command, one can display all commands that were run against a zpool since its creation together with a time stamp. This can be very useful for error analyses and also to repeat a configuration on another server.

[root@pool6 ~]# zpool history tank-2TB

History for 'tank-2TB':

2016-04-05.11:27:43 zpool create -f tank-2TB raidz2 sdd sdf sdg sdi sdj sdl sdm sdz sdaa sdab sdac sdad sdae sdaf sdag sdah sdai sdaj spare sdak

2016-04-05.11:40:32 zfs create -o refreservation=7TB tank-2TB/gridstorage01

2016-04-05.11:40:34 zfs create -o refreservation=7TB tank-2TB/gridstorage02

2016-04-05.11:40:39 zfs create -o refreservation=7TB tank-2TB/gridstorage03

2016-04-05.11:41:57 zfs create -o refreservation=6.97T tank-2TB/gridstorage04

2016-04-05.12:02:12 zpool set autoreplace=on tank-2TB

2016-04-05.12:02:17 zpool set autoexpand=on tank-2TB

2016-04-05.12:38:11 zpool export tank-2TB

2016-04-05.12:38:33 zpool import -d /dev/disk/by-id tank-2TB

2016-04-06.13:41:37 zfs set compression=lz4 tank-2TB

2016-04-07.14:36:37 zfs set relatime=on tank-2TB

2016-04-07.14:36:42 zfs set xattr=sa tank-2TB

2016-04-11.11:28:08 zpool scrub tank-2TB

2016-04-11.14:12:41 zfs set refquota=7T tank-2TB/gridstorage01

2016-04-11.14:12:43 zfs set refquota=7T tank-2TB/gridstorage02

2016-04-11.14:12:48 zfs set refquota=7T tank-2TB/gridstorage03

2016-04-11.14:12:59 zfs set refquota=7T tank-2TB/gridstorage04

The second command, zpool iostat, displays the current I/O on the pool, separately for read/write operations and for bandwidth. Information can be displayed for a given pool but also for each disk within the pool. 

The following example is taken from a server that was configured with 3 raidz2 while it was drained using the dpm-drain command on the head node with threads=1 and one drain per file system, resulting in 5 parallel drain commands running:

[root@pool5 ~]# zpool iostat 1

               capacity     operations    bandwidth

pool        alloc   free   read  write   read  write

----------  -----  -----  -----  -----  -----  -----

tank        6.03T  54.0T    110     24  13.5M  1.33M

tank        6.03T  54.0T  4.74K      0   602M      0

tank        6.03T  54.0T  4.63K      0   589M      0

tank        6.03T  54.0T  4.62K      0   587M      0

tank        6.03T  54.0T  4.42K      0   561M      0

tank        6.03T  54.0T  5.27K      0   669M      0

tank        6.03T  54.0T  4.51K      0   573M      0

tank        6.03T  54.0T  4.47K      0   568M      0

tank        6.03T  54.0T  4.47K      0   568M      0

tank        6.03T  54.0T  4.26K      0   542M      0

tank        6.03T  54.0T  4.56K      0   579M      0

tank        6.03T  54.0T  4.82K      0   613M      0

tank        6.03T  54.0T  4.60K      0   585M      0

tank        6.03T  54.0T  4.73K      0   601M      0

tank        6.03T  54.0T  4.20K      0   533M      0

tank        6.03T  54.0T  4.52K      0   574M      0

tank        6.03T  54.0T  3.72K      0   473M      0

tank        6.03T  54.0T  3.80K      0   484M      0

tank        6.03T  54.0T  4.46K      0   567M      0

tank        6.03T  54.0T  5.16K      0   655M      0

tank        6.03T  54.0T  5.25K      0   667M      0

[root@pool5 ~]# zpool iostat -v 1

                                               capacity     operations    bandwidth

pool                                        alloc   free   read  write   read  write

------------------------------------------  -----  -----  -----  -----  -----  -----

tank                                        5.87T  54.1T  4.77K      0   606M      0

  raidz2                                    1.96T  18.0T  1.60K      0   202M      0

    scsi-36a4badb044e936001e55b2111ca79173      -      -    329      0  21.2M      0

    scsi-36a4badb044e936001e55b2461fc3cb38      -      -    354      0  20.8M      0

    scsi-36a4badb044e936001e55b25520b22e10      -      -    337      0  21.3M      0

    scsi-36a4badb044e936001e55b2622171ff0b      -      -    333      0  21.3M      0

    scsi-36a4badb044e936001e55b26e2232640f      -      -    334      0  21.0M      0

    scsi-36a4badb044e936001e55b27d230ce0f6      -      -    333      0  21.2M      0

    scsi-36a4badb044e936001e55b293245ae11b      -      -    335      0  21.1M      0

    scsi-36a4badb044e936001e55b2b426603fbe      -      -    335      0  21.3M      0

    scsi-36a4badb044e936001e55b2c4274ec795      -      -    338      0  20.9M      0

    scsi-36a4badb044e936001e55b2d128122551      -      -    318      0  20.8M      0

    scsi-36a4badb044e936001e55b2f42a2e3006      -      -    342      0  21.3M      0

  raidz2                                    1.96T  18.0T  1.59K      0   203M      0

    scsi-36a4badb044e936001e830de6afdb1d8f      -      -    332      0  21.6M      0

    scsi-36a4badb044e936001e55b3082b59c1da      -      -    310      0  21.2M      0

    scsi-36a4badb044e936001e55b3142c0ac749      -      -    311      0  21.5M      0

    scsi-36a4badb044e936001e55b31f2cbeb648      -      -    319      0  21.8M      0

    scsi-36a4badb044e936001e55b44e3ecc77ea      -      -    313      0  21.7M      0

    scsi-36a4badb044e936001e55b33b2e6172a4      -      -    213      0  21.8M      0

    scsi-36a4badb044e936001e55b34c2f70184c      -      -    307      0  21.4M      0

    scsi-36a4badb044e936001e55b358301ee6a2      -      -    319      0  21.8M      0

    scsi-36a4badb044e936001e55b36530e4cb2a      -      -    331      0  21.8M      0

    scsi-36a4badb044e936001e55b3793218970b      -      -    325      0  21.8M      0

    scsi-36a4badb044e936001e55b38532cf8c68      -      -    324      0  21.8M      0

  raidz2                                    1.96T  18.0T  1.58K      0   201M      0

    scsi-36a4badb044e936001e55b39033741ccf      -      -    342      0  21.8M      0

    scsi-36a4badb044e936001e55b39b3421403b      -      -    323      0  21.4M      0

    scsi-36a4badb044e936001e55b3de3822569a      -      -    335      0  21.8M      0

    scsi-36a4badb044e936001e55b3eb38df509e      -      -    328      0  21.7M      0

    scsi-36a4badb044e936001e55b3f839a79c83      -      -    301      0  21.5M      0

    scsi-36a4badb044e936001e55b4023a46ae2a      -      -    325      0  21.7M      0

    scsi-36a4badb044e936001e55b40f3b0100cb      -      -    314      0  21.5M      0

    scsi-36a4badb044e936001e55b41d3bdd5a86      -      -    335      0  21.5M      0

    scsi-36a4badb044e936001e55b42b3cb239cd      -      -    324      0  21.8M      0

    scsi-36a4badb044e936001e55b4363d55d784      -      -    322      0  21.7M      0

    scsi-36a4badb044e936001e55b4413e09a9f1      -      -    331      0  21.8M      0

------------------------------------------  -----  -----  -----  -----  -----  -----

One important point to keep in mind is that all the properties set with zfs or zpool are stored within the file system and not within the OS config files! That means, if the OS gets upgraded then one can do a zpool import and all properties - like mount point, quota, reservations, compression, history - will instantly be available again. There is no need to touch manually any system config file, like /etc/fstab, to make the storage available.  This is also true for other properties, like nfs sharing, but since it's not needed in our case I haven't described that. To get an idea what properties are available and what else one can do with ZFS, zfs get all and zpool get all are useful.