After upgrade from Karmic boot fails, mountall and udevd fail

[frenzy]

Binary package hint: mountall

After upgrading from Karmic im no more able to boot, mountall fails with this message

# mountall
libudev: udev_monitor_new_from_netlink: error getting socket: Invalid argument
mountall:mountall.c:2955: Assertion failed in main: udev_monitor = udev_monitor_new_from_netlink (udev, "udev")
Aborted

Udevd is not running, if i try to start it:
# udevd
error initializing netlink socket

To use the system via ssh i need to create /var/run/network, restart networking, create /dev/pts (and mount it)
If i logout from console system reboots

My df looks like this:
# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/cciss/c0d0p1 11416496 9578824 1257732 89% /
udev 518056 40 518016 1% /dev

Mtab
# cat /etc/mtab
/dev/cciss/c0d0p1 / ext3 rw,errors=remount-ro 0 0
proc /proc proc rw 0 0
udev /dev tmpfs rw,mode=0755 0 0
none /dev/pts devpts rw,noexec,nosuid,gid=5,mode=0620 0 0
none /dev/pts devpts rw,noexec,nosuid,gid=5,mode=0620 0 0
none /dev/pts devpts rw,noexec,nosuid,gid=5,mode=0620 0 0
sys /sys sysfs rw 0 0
none /dev/pts devpts rw,noexec,nosuid,gid=5,mode=0620 0 0

System is quite unusable, im not able to install packages either because /dev/shm is not mounted.

[frenzy (johnuopini)]

Forgot to mention not im running 2.6.18-3-686 because im not able to boot with -27 included in lucid, the boot stops with a loading bar and stays there.

[frenzy (johnuopini)]

With latest kernel this problem dissapears. Please close this.

[Bryce Harrington (bryce)]

I ran into this as well. I think it actually is a legitimate issue because if you run into a problem with the lucid kernel, it is entirely sensible that you'd want to boot an earlier kernel version and have it work. I ran into this in a Hardy -> Lucid upgrade.

You can workaround it by issuing the following commands at the tty emergency console;

mount -n -o remount,rw /
mkdir /var/run/network
/etc/init.d/networking restart
mkdir /dev/pts
mount /dev/pts
/etc/init.d/ssh restart
mkdir /dev/shm
mount /dev/shm

You should be able to ssh into the server like usual and start up any other services that may not be running.

[ward (ward-pong)]

I second that this is a real problem.

Anyone running Lucid on a virtualized system, as a guest, will run into this if their hypervisor runs a kernel that's a little older.

Say, Debian Lenny on the bare metal with its 2.6.26-xen kernel, and a Lucid domU.

Can we please get a fix in for this before Lucid is released?

[phaidros (phaidros)]

Yup. Real problem!
Especially in virtualized environments.

I'd love to test fixes!

[Scott James Remnant (Canonical) (canonical-scott)]

Sorry, we simply do not support earlier kernel versions. The plumbing layer is now tightly integrated with the kernel; we had an option to either support kernel upgrades or downgrades, we chose to support upgrades since updating a kernel is common

[bodhi.zazen (bodhi.zazen)]

I do not think this was a request to support an older kernel, I think this is a request to examine the possibility of using Ubuntu 10.05 with "bare metal" virtualization technologies such as Xen, Openvz, Vserver, etc. I believe there is an interest in using Ubuntu at least as a guest OS if not on the host.

[bodhi.zazen (bodhi.zazen)]

Ubuntu 10.04, sorry =)

[Scott James Remnant (Canonical) (canonical-scott)]

On Thu, 2010-04-22 at 14:55 +0000, bodhi.zazen wrote:

> I do not think this was a request to support an older kernel, I think
> this is a request to examine the possibility of using Ubuntu 10.05 with
> "bare metal" virtualization technologies such as Xen, Openvz, Vserver,
> etc. I believe there is an interest in using Ubuntu at least as a guest
> OS if not on the host.
>

Ubuntu works just fine in such virtualisation environments, provided the
host and guest are using the correct kernel versions.

Scott
--
Scott James Remnant
<email address hidden>

[ward (ward-pong)]

> Ubuntu works just fine in such virtualisation environments, provided the
> host and guest are using the correct kernel versions.

Well, that means that Lucid will be a whole lot less relevant on the server side than previous Ubuntu versions.

I'm sure you know that Xen is still not fully incorporated in the kernel.org mainline tree.

That means there are many, many machines out there that run older kernel versions in the dom0, since the Xen patches don't necessarily have the same levels of reliability against the most recent kernels. The situation is a bit better now with the release of Xen 4.0, but the installed base of older dom0s is still out there.

I have dozens of such servers. They run Hardy or Lenny in the dom0. None of them will be able to run Lucid as a domU, except presumably if I run them fully virtualized which is a performance hit (see http://wiki.xensource.com/xenwiki/XenOverview).

What kernel version is required, actually? Is it really *that* impossible to fix this mountall call?

[Scott James Remnant (Canonical) (canonical-scott)]

On Thu, 2010-04-22 at 18:59 +0000, ward wrote:

> What kernel version is required, actually? Is it really *that*
> impossible to fix this mountall call?
>
2.6.32 is required; it's not just mountall, there are dependencies on
this kernel version in upstart, mountall, ureadahead, udev, plymouth, X,
etc.

ie. all the way through the system. mountall just happens to be the
first error you hit

Scott
--
Scott James Remnant
<email address hidden>

[Thomas Zehetbauer (realborg)]

dear scott, can you please explain to my xen hosting provider why they need to upgrade their stable dom0 in order to support ubuntu with new domu kernels? their e-mail is <email address hidden>

I have just replaced upstart, mountall, ureadahead, plymouth,... by a very simple script that does what they fail to do: initialize my server and start the required services. very basic functionality which has been reliably provided by sysv init for more than 20 years...

what do you mean by plumbing layer? sysv init has reliably worked for >20 years with hundreds of different kernels without needing any layers. upstart introduces a plumbing layer and everything breaks

[Matthew Nuzum (newz)]

Hi, just ran into this myself. Upgraded a vm on a Lenny dom0 from hardy to lucid. I only have paravirtualization at my disposal (4 servers, none with hvm support). Does this mean I'm out of luck and need to use an older release when running Ubuntu in a paravirtualized situation? Also, I read through the release notes (http://www.ubuntu.com/getubuntu/releasenotes/1004) and saw no mention of this. I was quite surprised to have this problem since I've run 8.04 and 9.04 without any problems.

[Steven Ellis (steven-openmedia)]

Ive hit the same issue upgrading an Ubuntu 8.04 Xen VM to Lucid 10.04.

The host is running Centos 5.3 and I have a mix of Centos, Debian and Ubuntu VMs. The Debian VMs upgraded to Lenny without incident but It appears I can't run a paravirt Lucid VM on top of Centos.

If I try to boot with a Lucid kernel (2.6.24-21-xen) I get

libudev: udev_monitor_new_from_netlink: error getting socket: Invalid argument
[ 0.505931] wait-for-root[761]: segfault at 00000030 eip b7edef2b esp bfb99f50 error 4
Segmentation fault
chvt: can't open console
There appears to be one or more degraded LVM volumes, and your root device may
depend on the LVM volumes being online. One or more of the following LVM volumes
are degraded:
  read_urandom: /dev/urandom: open failed: No such file or directory
Gave up waiting for root device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
   - Check root= (did the system wait for the right device?)
 - Missing modules (cat /proc/modules; ls /dev)
chvt: can't open console
ALERT! /dev/disk/by-uuid/87fea4c2-db1f-4743-bab0-da95e5c8e594 does not exist. Dropping to a shell!

Anyone had better luck.

[Steven Ellis (steven-openmedia)]

Ok upgrading the Xen VM didn't install a Xen compatible kernel and it reverted to older 8.04 kernels.

Install linux-image-virtual and I'm now up and working

[Matthew Nuzum (newz)]

Steven, can you explain what you did to get it working? I'd like to replicate it if possible.

[Steve Deery (steve-deery)]

I have upgraded from 8.04 to 10.04, I had a problem with ureadahead: init: ureadahead main process (2603) terminated with status 5
libudev: udev_monitor_new_from_netlink: error getting socket:Invalid argument.
Mountall: mountall.c:3204: Assertion failed in main: udev_monitor = udev_monitor_new_from_netlink (udev,"udev")
init: mountall main process (2606) killed by ABRT signal
General error mounting file systems
A maintenance shell will now be started
I attempted to fix this by moving ureadahead.conf to ureadahead.conf.disable as described here: http://ubuntuguide.net/howto-fix-ure...o-ubuntu-10-04.
Now I am presented with the following ;
libudev: udev_monitor_new_from_netlink:error getting socket: invalid argument mountall: mountall.c:3204: Assertion failed in main: udev_monitor = udev_monitor etc etc
init: mountall main process (2639)killed by ABRT signal
General error mounting filesystems etc

[Steve Deery (steve-deery)]

There are quite a number of people on the Ubuntu Forums reporting the same problem but no solution yet. I have been informed that my problem is a duplicate of this bug. Can anyone point me in the right direction, I am keen to gather more info if someone will tell me how. I don't have a VM running or anything just a desk top which has been down for over three weeks now after the upgrade from 8.04 to 10.04.

[Bastian Mäuser (mephisto-mephis)]

Canonical doesn't make it easy for one to stand behind them, if again and again you get a Knife in your back.

Upstart is not a solution aimed at Servers, hence it should be removed from the serveredition. It renders it totally unusable in Virtualized environments, where more dependencies exist..

What are Canonicals plans for that?

[ScottMarlowe (scott-marlowe)]

I am having the exact same problem as Steve Deery. However, it's much worse for me, because as soon as it can't find udev, my USB keyboard is rendered useless. No big deal? Well, it's a Dell PowerEdge 1950 and the only port it supports for the keyboard is USB. urg, so now I can't get into the machine to fix it and get it to finish booting and maybe upgrade the kernel or whatever else to fix this.

Is there a possibility of a fix like 10.04.1LTS that can fix this botched upgrade?

[anurag uniyal (anuraguniyal)]

I am having the same problem, i am not using any virtualization, I upgraded from 8.04 to 10.04 and now can start my machine. I get error

libudev: udev_monitor_new_from_netlink: error getting socket: Invalid argument segfault

and after that my USB keyboard also doesn't work.

[AR (info-xlrs)]

I found a way getting a working 10.04 domU with a hardy dom0.

I used this howto: http://www.agileweboperations.com/xen-debian-lenny-dom0-with-ubuntu-lucid-guest/

The only differences I did in my $DOMU.cfg. This is my config:

bootloader="/usr/bin/pygrub"
#kernel = '/boot/vmlinuz-2.6.24-28-xen'
#ramdisk = '/boot/initrd.img-2.6.24-28-xen'
memory = '4096'

#
# Disk device(s).
#
root = '/dev/xvda2 ro'
disk = [
                  'tap:aio:/data/xen/domains/mail1/disk.img,xvda2,w',
                  'tap:aio:/data/xen/domains/mail1/swap.img,xvda1,w',
              ]

HTH, Axel

[Mark - Syminet (mark-syminet)]

Same problem here. If the newer kernel is required, then why isn't it a dependency when going from Hardy -> Lucid?

[gkforcare (gerke-forcare)]

I encounter the same problem when I upgrade my Hardy to Lucid on my (bare hardware) AMD Athlon. Will try to add more details later.

[gkforcare (gerke-forcare)]

Here are the promis

Thanks go to Bryce Harrington in comment #3 for giving a way around this but it would be good if I could find a permanent way around this. Lucky me for using a PS/2 keyboard ;-)

uname -a gives:
Linux klimrek 2.6.24-27-server #1 SMP Wed Mar 24 10:39:18 UTC 2010 i686 GNU/Linux

cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 4
model name : AMD Athlon(tm) Processor
stepping : 2
cpu MHz : 896.214
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr syscall mmxext 3dnowext 3dnow up
bogomips : 1792.42
clflush size : 32

Can anyone comment on what information might be useful to fix this problem?

[gkforcare (gerke-forcare)]

Dear me, that is the cause of my problem: I'm still running a kernel that is not upstart friendly: 2.6.24-27-server But I cannot use the new kernels as they hang during boot. (Even the 386 version, as opposed to x86) The saga continues, will keep you posted, if I may.

[Henry Hollenberg (hgh)]

I found a solution that worked in my upgrade. I spent the weekend upgrading:
7.04 --> 7.10 --> 8.04 --> 10.04 (Yes it was quite the adventure!)

I saw a few messages about using the old "menu.lst" file during the final upgrade to 10.04 with no
options other than keeping the old file as I recall.

checked /boot/grub/menu.lst and see that it was my old 8.04 menu.lst file. No new kernel. No conversion of
/dev/sda1 to UUID=1234hj-34k435j1-35k13h12 mumbo-jumbo like I had seen on all the previous upgrades.

Found the bottom half of my new 10.04 menu.lst file here:

/var/lib/ucf/cache/:var:run:grub:menu.lst

did a "less" on it and saw it had my new kernel and the UUID stuff (which I don't use) in it.

copied it over:

cp /var/lib/ucf/cache/:var:run:grub:menu.lst /boot/grub/menu.lst

changed the UUID's back to /dev/sda1 and rebooted.

BAM! up and running.

Still need to copy the top half of the saved menu.lst file back over into the "working" menu.lst file to get back my
"will boot in 5 seconds hit Esc to enter GRUB" stuff.