Aug 252010

My VPS host rolls their own Xen custom kernel but I wanted to run the Debian distribution supplied kernel simply so that it gets updated by the Debian security updates.

MammonthVPS has a nice tutorial here but I ran into this bug. Where grub throws: grub-probe: error: Cannot find a GRUB drive for /dev/xvda2. This is due to the missing root device /dev/xvda.

The cleanest solution is here but requires the host to change their Xen setup which may not be possible.
So I used a combination of the suggestions by adding a check near the top of /usr/sbin/update-grub:

vps01 #nano `which update-grub`
host_os=`uname -s | tr '[A-Z]' '[a-z]'`
# Added mknod check as work around to grub bug, GK.
if ! test -a /dev/xvda ; then
        mknod /dev/xvda b 202 0
# Find and modify this section for Debian 6.0, as per Nathan's comment.
find_device ()
    if ! test -e ${device_map} ; then
        echo quit | grub –batch –no-floppy –device-map=${device_map} > /dev/null
    #Comment out grub-probe and return hard coded device.
    #grub-probe –device-map=${device_map} -t device $1 2> /dev/null
    echo /dev/xvda

This will ensure that the root device exists when grub-probe is called by update-grub, otherwise the device will disappear after reboot when udev is called and a future kernel upgrade may fail. This is a satisfactory solution for me because grub is rarely updated. The surprise here was that I expected to loose some memory (which is of course at a premium on a VPS) but actually appear to have freed up some depending on the state of the caches and kernel modules loaded.

Host supplied kernel (just after reboot and after all procs started):

vps01 #free -m
             total       used       free     shared    buffers     cached
Mem:           498        478         19          0          9        185
-/+ buffers/cache:        283        214
Swap:         1023          0       1023

Distribution kernel (Debian 2.6.26-2-xen-686, just after reboot and after all procs started):

vps01 #free -m
             total       used       free     shared    buffers     cached
Mem:           512        434         77          0          9        139
-/+ buffers/cache:        285        226
Swap:         1023          0       1023

Note that you have to use a Xen enabled kernel (linux-image-2.6-xen-686 or linux-image-xen-686) and there is a libc6-xen package that you may be interested in installing but this appears to only be required to protect 32bit hosts info here. I had to use the host control panel restart to switch between supplied kernels and PvGrub but after that a command line reboot is all that is required after kernel updates.

Debian 6.0 update.

I have successfully upgraded from Lenny (Debian 5) to Squeeze (Debian 6.0.1) on this same VPS. Even booted with Grub2 but to follow the instructions here it is required to return to (or remain with) grub-legacy.

Install grub-legacy with aptitude or apt-get so that:

i    grub-common    1.98...
i    grub-legacy    0.97...

And grub-pc (grub2) is purged:

p    grub-pc        1.98...

Check /boot/grub/

vps01 #cat /boot/grub/ 
(hd1) /dev/xvda
If required:
vps01 #echo '(hd1) /dev/xvda' > /boot/grub/
vps01 #update-grub 0
vps01 #sed -i 's/xvda/xvda2/' /boot/grub/menu.lst
vps01 #cat /boot/grub/menu.lst |grep xvda
# kopt=root=/dev/xvda2 ro
kernel          /boot/vmlinuz-2.6.32-5-xen-686 root=/dev/xvda2 ro 
kernel          /boot/vmlinuz-2.6.32-5-xen-686 root=/dev/xvda2 ro single
kernel          /boot/vmlinuz-2.6.26-2-xen-686 root=/dev/xvda2 ro 
kernel          /boot/vmlinuz-2.6.26-2-xen-686 root=/dev/xvda2 ro single

Check for files in /boot/grub/:

vps01 #ls /boot/grub/
default  menu.lst

No UUID’s in /etc/fstab:

vps01 #cat /etc/fstab 
# /etc/fstab: static file system information.
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
devpts          /dev/pts        devpts  rw,noexec,nosuid,gid=5,mode=620 0  0
/dev/xvda1 none swap sw 0 0
/dev/xvda2 / ext3 noatime,nodiratime,errors=remount-ro 0 1

Gavin Kromhout:

Thank you for visiting.
Do look around.
Do leave a comment.

  8 Responses to “Running a custom or distribution kernel on a VPS”

  1. It seems to be even worse on Squeeze (Debian 6.0) – grub_probe is not sufficiently tricked by the presense of /dev/xvda , as it actually tries to read from the drive it seems. Here’s the steps I needed, in addition to yours:

    echo ‘(hd1) /dev/xvda’ > /boot/grub/
    vim `which update-grub` # Find the following section, and edit as shown

    find_device ()
    if ! test -e ${device_map} ; then
    echo quit | grub –batch –no-floppy –device-map=${device_map} > /dev/null
    #grub-probe –device-map=${device_map} -t device $1 2> /dev/null
    echo /dev/xvda

    update-grub 0
    sed -i ‘s/xvda/xvda2/’ /boot/grub/menu.lst

    I will definitely be looking into changing our default partition mapping in the future to keep grub happy.

  2. Thanks Nathan!
    It would be wonderful if the default partition mapping is updated for the next Debian release. Having PvGrub available is a fantastic feature and greatly appreciated.

  3. A Bug of Grub-probe on Debian Xen VPS…

    My VPS runs its own kernel with PvGrub enabled on the host machine. When apt-get upgrades the kernel package, grub-probe will throw an error message: grub-probe: error: Cannot find a GRUB drive for /dev/xvda1 And abort the upgrade procedure. This is du…

  4. Success 🙂

    I only have xvda1 (no xvda) and it works.
    Thank you very much!

  5. For security purposes, I altered the mknod command to:

    mknod -m 0660 /dev/xvda b 202 0

    as it’s created with 0644 permissions otherwise.

    Cheers for the post – I’ve just made these changes on my VPS to sort this problem out.


  6. With grub2 this doesn’t work.

    When you uncomment #GRUB_TERMINAL=console in /etc/default/grub … run update-grub then aptitude could install the kernel-image without problems in my case.

  7. Thanks for the post Christiaan, I don’t have /etc/default/grub with grub-legacy installed but do on my laptop with grub2 installed.

  8. MammonthVPS have adjusted their xen configuration so that the hacks in this post are no longer required for me.

    From a fresh install this should work:

    Nathan: the minimum steps from a fresh install seem to be; apt-get install grub-legacy ; mkdir /boot/grub ; grub-set-default default ; apt-get install linux-image-xen-686

    My current config shows:
    #cat /boot/grub/
    (hd0) /dev/xvda

    #cat /etc/fstab
    /dev/xvds none swap sw 0 0
    /dev/xvda / ext3 noatime,nodiratime,errors=remount-ro 0 1

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>