OpenSUSE Linux Rants

How To Install SUSE 10.1 on a machine with a Hewlett-Packard NetRAID 1M/2M:

This is an experience that I had recently at work with an HP Netserver LP 2000r machine. My boss asked me to install SUSE 10.1 on it.

It might be worth it to note that any time I blow away a machine’s installation, I get some information from it before I do so. I’m glad I did in this case. I doubt I could have done this without it. The magic I use to conjure up this information is as follows:

# dmesg > dmesg.txt
# yast2 hwinfo
[save out the information into a text file]
# zcat /proc/config.gz > kernel_configuration.txt
# lsmod > lsmod.txt
# lspci -v > lspci-v.txt
# cp /boot/grub/menu.lst ./menu.lst
# cp /var/log/messages ./kernel-messages.txt

I just make a bzipped tarball of all these files and move them to another machine. That way, I know what modules were loaded, what kernel configurations were, and huge loads of other stuff, especially with the file created by exporting the hwinfo information. It is very fortunate for me that in this case, I did all of this.

I began by booting from the SUSE 10.1 CD 1 install disc. I got about 2 screens into the installer when the machine completely stopped responding. During the installation, you can press CTRL+ALT+F4 to see the debugging output and messages from the kernel. Reviewing the output, it was trying to access the Hard disc, which was now completely unavailable because the kernel module had crashed. I tried turning the machine off and doing it again, but the same thing happened.

A bit of investigation revealed that the Hewlett-Packard NetRAID 1M RAID controller that we are using in that box is supposed to use the megaraid_mbox kernel module as its driver. This kernel module is what was crashing. That’s when I hit Google to see what other information I could find.

After a little research, I found that the newer kernels have lost support for legacy megaraid controllers (which the NetRAID 1M certainly is). Extensive searching led me to a page that offered me a glimmer of hope. It contained a patch that would make even the new megaraid module support the legacy RAID controllers.

This patch is as follows:

--- kernel-source-2.6.11/drivers/scsi/megaraid.h	2005-03-01
23:38:09.000000000 -0800
+++ kernel-source-2.6.11.works/drivers/scsi/megaraid.h	2005-07-05
10:05:44.000000000 -0700
@@ -84,6 +84,10 @@
 #define LSI_SUBSYS_VID			0x1000
 #define INTEL_SUBSYS_VID		0x8086

+/* Sub-System Device IDs */
+#define HP_NETRAID1M_SUBSYS_DID		0x60E7
+#define HP_NETRAID2M_SUBSYS_DID		0x60E8
+
 #define HBA_SIGNATURE	      		0x3344
 #define HBA_SIGNATURE_471	  	0xCCCC
 #define HBA_SIGNATURE_64BIT		0x029

--- kernel-source-2.6.11/drivers/scsi/megaraid.c	2005-06-16
08:06:21.000000000 -0700
+++ kernel-source-2.6.11.works/drivers/scsi/megaraid.c	2005-07-05
10:06:39.000000000 -0700
@@ -5037,6 +5037,10 @@
 		PCI_ANY_ID, PCI_ANY_ID, 0, 0, 0},
 	{PCI_VENDOR_ID_AMI, PCI_DEVICE_ID_AMI_MEGARAID2,
 		PCI_ANY_ID, PCI_ANY_ID, 0, 0, 0},
+	{PCI_VENDOR_ID_AMI, PCI_DEVICE_ID_AMI_MEGARAID3,
+		HP_SUBSYS_VID, HP_NETRAID1M_SUBSYS_DID, 0, 0, 0},
+	{PCI_VENDOR_ID_AMI, PCI_DEVICE_ID_AMI_MEGARAID3,
+		HP_SUBSYS_VID, HP_NETRAID2M_SUBSYS_DID, 0, 0, 0},
 	{0,}
 };
 MODULE_DEVICE_TABLE(pci, megaraid_pci_tbl);

My next task was to get the module source patched, compiled and loaded into the kernel that the install CD uses.

The problem is that if you compile your own module, you have to compile it against the exact version of the source code that was used to create the kernel into which you want to load the module. You also have to compile it with the very same version of gcc that was used to compile that target kernel. I needed to find out the gcc version and the version of kernel source code used to make the kernel that the CD boots from.

I booted a different machine off the install CD. When I got to the install screen, I went to one of the virtual terminals with CTRL+ALT+F2. One command gave me all the information I needed:

# cat /proc/version

It said that we were using the 2.6.16.13-4-default kernel, and that this kernel was compiled with gcc 4.1.0.

Next, I set up a machine running that 2.6.16.13-4-default kernel, installed gcc 4.1.0, and installed the kernel-source RPM for that same kernel. On that machine I went into /usr/src/linux/drivers/scsi and manually edited the megaraid source, applying the changes specified in the patch. To play it safe, I also wanted to use the .config file that was used to build the 2.6.16.13-4-default kernel. Fortunately, this is located in the /boot directory. You can quickly get this file where it needs to be with this command:

# cp /boot/config-2.6.16.14-4-default /usr/src/linux/.config

I was all set up. I then recompiled the whole kernel and modules using “make && make modules”.

When that finished 93 years later, I copied the patched megaraid files (megaraid.c and megaraid.h) and the new megaraid.ko kernel module onto my USB stick.

I then went back over to the Netserver machine with the NetRAID controller. I booted off the CD, again. To my dismay, I saw that the megaraid_mbox module had loaded, disallowing me to load the megaraid module that I had just created.

Again, I hit the research path. After awhile, I found a post somewhere that pointed me to a file right on my own hard drive. After reading /usr/share/doc/packages/hwinfo/README, I saw that it was indeed possible to disable probing of the hardware during boot time. I was going to have to turn off probing of my NetRAID 1M with the kernel parameter “hwprobe=-0104:101e:1960″. These numbers were fished out of all that info I gathered out of the box before I wiped it.

I found I also would need to use “noapic” or other things would not work properly.

Into the drive went CD 1 and I started up the machine. When grub came up, I typed in “noapic hwprobe=-0104:101e:1960″ as my kernel parameters and away I went.

When I got to the SELECT LANGUAGE screen of the installer, I again pressed CTRL+ALT+F2 to get into a virtual terminal. I mounted my USB stick and copied my megaraid.ko module off it into the /modules directory. It went something like this:

# mkdir /usb
# cat /proc/partitions
(looking for the USB partition)
# mount /dev/sda1 /usb
(where /dev/sda1 is my USB partition)
# cp /usb/megaraid.ko /modules/

I took out the USB stick here so that the hard disk would be at /dev/sda. I also did this so that the installer would not try and mount my USB stick as some kind of Windows partition. At this point, I held my breath and ran the command that would solve my woes:

# modprobe megaraid

To my utter amazement, the module actually loaded right in without a problem. During a few previous attempts, I got an error like “-1 : Invalid module format”, which I hoped not to get again. As it turned out, everything went fine.

With ALT+F7, I was back at the installer screen. I went through the installer and rebooted the machine.

Bad news. The kernel that was installed was a bigsmp kernel, and could not use the megaraid module that I had compiled.

I started the installation over. This time, during the installation, I changed things a bit. I deselected the bigsmp kernel and instead selected the default one for installation. As it reached the point where it was going to reboot, I stopped it. There was one last thing that I needed to do before I restarted the machine.

When YAST installs a new machine, it puts the kernel into /boot along with an initial ram disk. This initial ram disk is used to get a minimalistic environment set up so that it can load kernel modules without having to have the filesystem available.

Knowing this, I mounted my root partition and copied the megaraid.ko module from /modules (I was still in the CD environment) over to the modules directory of the kernel installed on the hard disk. It went something like this:

# cat /proc/partitions
(I then found my root partition, made a mount point for it, and mounted it)
# mkdir /hdd
# mount /dev/sda1 /hdd
(where /dev/sda1 is my root partition)

I could then copy my megaraid.ko module to the appropriate place on the hard drive:

# cp /modules/megaraid.ko /hdd/libs/modules/2.6.16.13-4-default/kernel/drivers/scsi/

I copied the /etc/mtab file to the hard disk so I could chroot into it:

# cp /etc/mtab /hdd/etc/mtab
# chroot /hdd

I had to make sure that SUSE would put the megaraid module into the initial ram disk. This is done in the /etc/sysconfig/kernel text file:

# vim /etc/sysconfig/kernel

All I had to do was to make sure the INITRD_MODULES had “megaraid” in it:

INITRD_MODULES="megaraid serverworks sym53c8xx processor thermal fan reiserfs"

After saving and exiting, I then went about rebuilding the initial ram disk (hereafter referred to as ‘initrd’):

# rm /boot/initrd-2.6.16.13-4-default
# mkinitrd -k vmlinuz-2.6.16.13-4-default -i initrd-2.6.16.13-4-default

I then went into /boot/grub/menu.lst to clean it up a bit. I basically had to take out the hwprobe kernel option. Evidently, it put in the “noapic hwprobe=-0104:101e:1960″ options I had first entered when I first booted into the install disc.

I took a deep breath, took out the install disc and then rebooted the machine. To my utter thrill and excitement, the machine came right up, loading my patched megaraid module. After having gotten the “Invalid module format” error so many times previous to this, I was quite anxious. But it worked this time, so I was in great shape.

There was only one problem, though. I was now running a uniprocessor kernel on a machine with multiple processors, and the megaraid module that I had compiled was being rejected by that bigsmp kernel. I decided to have a go at making the patched megaraid module work with that bigsmp kernel.

The first thing I did was to install kernel-bigsmp-2.6.16.13-4 and kernel-source-2.6.16.13-4. That got me the right kernel and the right kernel source code. Now, I needed to install gcc and a few other things:

# yast2 -i gcc nfs-utils ncurses-devel

Fortunately, YAST installed the 4.1.0 version of gcc by default without much direction from me.

Now, I needed the .config file that was used to create the bigsmp kernel that I had just installed. This was accompished with the same method as I had previously done:

# cp /boot/config-2.6.16.13-4-bigsmp /usr/src/linux/.config

I now had the correct bigsmp kernel installed, I had the right kernel source package installed, and I had the right version of gcc installed. I had copied over the .config file. One thing left to do: copy the patched module source into the kernel source tree.

I mounted my USB stick and copied the patched versions of the megaraid.c and megaraid.h files over into the kernel source tree, putting them into the /usr/src/linux/drivers/scsi/ directory.

All was ready. I changed into the kernel source root and went for it:

# cd /usr/src/linux
# make && make modules

38,283 hours later, the compile finished. I now had a version of the module that was compiled specifically for the version of the kernel that was now installed. I had to copy it to the right place:

# cp /usr/src/linux/drivers/scsi/megaraid.ko /lib/modules/2.6.16.13-4-bigsmp/kernel/drivers/scsi

Again, I wanted to make sure that the /etc/sysconfig/kernel file was still going to include my megaraid module in the new initrd when I made one:

# vim /etc/sysconfig/kernel

Make sure the INITRD_MODULES has “megaraid” in it:

INITRD_MODULES="megaraid serverworks sym53c8xx processor thermal fan reiserfs"

I saved and quit, and made myself a new initrd:

# rm /boot/initrd-2.6.16.13-4-bigsmp
# mkinitrd -k vmlinuz-2.6.16.13-4-bigsmp -i  initrd-2.6.16.13-4-bigsmp

Next, I checked /boot to make sure that vmlinuz was pointing to vmlinuz-2.6.16.13-4-bigsmp and not something else. I also checked to make sure that initrd was pointing to initrd-2.6.16.13-4-bigsmp and not something else.

# ll /boot

I checked really quickly to make sure that /boot/grub/menu.lst had an entry in it that would be using vmlinuz as the kernel and initrd as the ramdisk and that grub was configured to use that entry by default:

# cat /boot/grub/menu.lst

Everything checked out, so I rebooted the machine:

# shutdown -r now

To my joy and amazement, the megaraid module built for the bigsmp kernel worked like a charm. As a last measure, I went into /boot/grub/menu.lst and took out the “noapic” kernel option and rebooted. It came right backup without a hitch.

I now had a completely usable system where the kernel was the one provided by the kernel-bigsmp-2.6.16.13-4 package, and the initrd was changed a bit to give it my patched megaraid module. Heh, I just have to be extremely careful with the updates I do on that box.

If you know anyone that has a NetRAID 1M/2M RAID controller and wants to put SUSE 10.1 on it, this is how you make it happen. As a matter of fact, I even have available the compiled binaries for both these kernel versions and the patched source code so you can build it for other kernels as necessary.

Good times.