CentOS kernel not loaded after reboot

wrong kernel gets loaded after install and reboot

I recently came across an issue where the CentOS kernel would not load after it was installed by yum and the machine was rebooted. So running

# uname -a

resulted in:

Linux hostname.example.com 2.6.18-194.32.1.el5.centos.plus #1 SMP Tue Apr 12 20:34:33 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

And not the expected:

Linux hostname.example.com 2.6.18-238.9.1.el5.centos.plus #1 SMP Tue Apr 12 20:34:33 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

I double checked what was installed:

# rpm -qa kernel
kernel-2.6.18-238.9.1.el5.centos.plus
kernel-2.6.18-238.5.1.el5.centos.plus
kernel-2.6.18-164.15.1.el5.centos.plus
kernel-2.6.18-194.32.1.el5.centos.plus

Next up was checking the /etc/grub.conf which is linked to /etc/grub.conf -> ../boot/grub/grub.conf but I find the first easier to find for some reason. But even that was showing me what I expected. The newest kernel in front of the rest and the default was set to 0.

# nano /etc/grub.conf


# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/md2
# initrd /initrd-version.img
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-238.9.1.el5.centos.plus)
root (hd0,0)
kernel /vmlinuz-2.6.18-238.9.1.el5.centos.plus ro root=/dev/md2 nodmraid
initrd /initrd-2.6.18-238.9.1.el5.centos.plus.img
title CentOS (2.6.18-238.5.1.el5.centos.plus)
root (hd0,0)
kernel /vmlinuz-2.6.18-238.5.1.el5.centos.plus ro root=/dev/md2 nodmraid
initrd /initrd-2.6.18-238.5.1.el5.centos.plus.img
title CentOS (2.6.18-194.32.1.el5.centos.plus)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.32.1.el5.centos.plus ro root=/dev/md2 nodmraid
initrd /initrd-2.6.18-194.32.1.el5.centos.plus.img
title CentOS (2.6.18-164.15.1.el5.centos.plus)
root (hd0,0)
kernel /vmlinuz-2.6.18-164.15.1.el5.centos.plus ro root=/dev/md2 nodmraid
initrd /initrd-2.6.18-164.15.1.el5.centos.plus.img

So what is going on here?

It turned out that my softraid partition was running in a clean but degraded mode and that my new boot info was residing only on sdb1 as it was not getting synced to sda1. For some reason the system still comes across sda1 first. I will probably have to set  "boot=/dev/md0" in my grub so this doesn't happen again.

/dev/md0 101018 22240 73562 24% /boot

# mdadm --detail /dev/md0

/dev/md0:
Version : 0.90
Creation Time : Thu May 27 12:40:33 2010
Raid Level : raid1
Array Size : 104320 (101.89 MiB 106.82 MB)
Used Dev Size : 104320 (101.89 MiB 106.82 MB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Sun May 1 04:22:01 2011
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

UUID : 6a433eb3:432630bb:ecca9597:015f1bc6
Events : 0.498

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 1 1 active sync /dev/sdb1

Luckily for me my RAID was not degraded because of a disk failure but something else. So for me it was an easy fix.

#mdadm /dev/md0 -a /dev/sda1

It actually turned out that yum had updated all info correctly in grub.conf but that is was not getting synced to the first partition that the system actually booted from.

So when the softraid was rebuild and running again I rebooted the system. This time the new kernel was loaded correctly.


The Naxion weblog..