CentOS8.1 (teaching an old server new tricks)

At Edinburgh we recently retired some old(er) storage servers from our Tier2.

These storage servers consist of a Dell Poweredge R610 with 2 RAID controllers.

A PERC H710 internal for managing the boot disks and a PERC H800 external which connects to 3 MD1200 DAS.

We still maintain very similar hardware on our Tier2 as part of our grid Storage Element and we've been considering skipping CentOS7 and jumping straight to CentOS8 on this hardware.

As an experiment I recently attempted an update from CentOS7 to CentOS8 on a VM recently. Whilst this  can be done in a Saturday I wouldn't recommend it as the resulting OS isn't really production stable. With that in mind a clean install is the way to go.

This would be simple if it wasn't for 1 gotcha. The RAID controllers on these servers are out of support in CentOS8: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8-beta/html/8.0_beta_release_notes/removed_functionality

There is, however a solution: https://wiki.centos.org/Manuals/ReleaseNotes/CentOS8.1905#Known_Issues

This solution is to make use of the driver update images compiled by the nice people behind CentOS. These images allow the installer to pick up the 3rd party module and install it into your system as-if it were fully supported.

However following this proved to be problematic.

For CentOS8.0 the installer failed to identify the disks behind either controller, and on more than 1 occasion lead to either the kernel or the installer crashing.

(although admittedly we had also mis-configured this card in a minor way during this testing).

CentOS8.1 on the other hand worked out of the box with the following images:

> sha256sum dd-megaraid_sas-07.707.51.00-1.el8_1.elrepo.iso CentOS-8.1.1911-x86_64-dvd1.iso 

31a169d5eab1371893347c4d8482896e0fcc9b0a813b9210b1c0e77f68b09702 dd-megaraid_sas-07.707.51.00-1.el8_1.elrepo.iso 

3ee3f4ea1538e026fff763e2b284a6f20b259d91d1ad5688f5783a67d279423b CentOS-8.1.1911-x86_64-dvd1.iso

To get the installer to work you will need 2 drives to be connected to the system you want to deploy on.,

NB: Don't do the 'obvious thing' of burning the dud driver image to a disk. It will not work if you do this.

The following worked for me:

1. Burn your CentOS8.1 install image
2. Format a 2nd drive fat(msdos) and copy the dud driver dd-megaraid....iso to it
3. Startup the install image
4. Before you boot into the installer add the following to the boot command to boot the installer kernel:
   inst.dd 
5. Now wait for the installer to ask for the dud driver. Navigate to the disk containing the iso, select it and continue to boot.
6. Now perform your install as usual.

This isn't quite the full story as there was a minor gotcha which seemed to hit occasionally.

To fix the system not booting after a kernel update:

echo 'force_drivers+="megaraid_sas"' >>  /etc/dracut.conf.d/force_drivers.conf

The CentOS8 installer has changes in subtle but important ways compared to CentOS7 but nothing was scary other than on my first attempt being greeted by a GUI desktop thinking it was in New York because I was more focussed on my storage network configurations.

There are some unresolved problems with using non standard kernels such as kernel-ml and kernel-plus with 3rd party dkms drivers such as ZFS. However the out of the box kernel is 4.18 which is relatively recent and performs quite well form initial testing.

Next to test the performance of this as a mock storage node before we consider deploying these across the rest of our Tier2.