update-rc.d segmentation fault on upgrade from clean feisty to gutsy

[Chris Higgins]

Binary package hint: sysv-rc

Host is a xen host - with a clean feisty image.
Logged into console, followed instructions for clean server image upgrade

apt-get install update-manager-core

do-release-upgrade

A whole chunk of packages won't complete their postinst because update-rc.d is seg faulting

I can install other packages , and remove packages - but apt-get --configure -a fails
all the time with this set of packages

Some of the packages fail because update-rc.d seg faults in the postinst, and the rest
are failing because they have dependencies on the first set.

Examining the postinst for the packages that are failing , I pulled a full command, and
running strace against it -- gives :

# strace /usr/sbin/update-rc.d module-init-tools start 15 S .
[SKIP]
[SKIP]
open("/usr/sbin/update-rc.d", O_RDONLY|O_LARGEFILE) = 3
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfc6ca98) = -1 ENOTTY (Inappropriate ioctl for device)
_llseek(3, 0, [0], SEEK_CUR) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
fstat64(3, {st_mode=S_IFREG|0755, st_size=5573, ...}) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL}, 8) = 0
readlink("/proc/self/exe", "/usr/bin/perl", 4095) = 13
brk(0x8194000) = 0x8194000
getppid() = 6835
read(3, "#! /usr/bin/perl\n#\n# update-rc.d"..., 4096) = 4096
read(3, "= \'S\'; }\n\telsif ($ARGV[0] eq \'st"..., 4096) = 1477
read(3, "", 4096) = 0
close(3) = 0
stat64("/etc/init.d/module-init-tools", {st_mode=S_IFREG|0755, st_size=1399, ...}) = 0
stat64("/etc/rcS.d", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
chdir("/etc/rc0.d") = 0
open(".", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
getdents64(3, /* 12 entries */, 4096) = 408
getdents64(3, /* 0 entries */, 4096) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Process 6836 detached

strace -v gives

getdents64(3, {{d_ino=14855, d_off=691923886, d_type=DT_LNK, d_reclen=40, d_name="S60umountroot"} {d_ino=14849, d_off=818700213, d_type=DT_DIR, d_reclen=24, d_name="."} {d_ino=14853, d_off=1128720051, d_type=DT_LNK, d_reclen=40, d_name="S31umountnfs.sh"} {d_ino=14856, d_off=1223729123, d_type=DT_LNK, d_reclen=32, d_name="S90halt"} {d_ino=14850, d_off=1253351180, d_type=DT_LNK, d_reclen=40, d_name="S15wpa-ifupdown"} {d_ino=14693, d_off=1327966852, d_type=DT_DIR, d_reclen=24, d_name=".."} {d_ino=15408, d_off=1669929854, d_type=DT_LNK, d_reclen=40, d_name="K59mountoverflowtmp"} {d_ino=17550, d_off=1859411755, d_type=DT_REG, d_reclen=32, d_name="README"} {d_ino=14852, d_off=1927655235, d_type=DT_LNK, d_reclen=32, d_name="S30urandom"} {d_ino=14854, d_off=1971293165, d_type=DT_LNK, d_reclen=32, d_name="S40umountfs"} {d_ino=14851, d_off=2082333935, d_type=DT_LNK, d_reclen=32, d_name="S20sendsigs"} {d_ino=14743, d_off=2147483647, d_type=DT_LNK, d_reclen=40, d_name="K25hwclock.sh"}}, 4096) = 408
getdents64(3, {}, 4096) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Process 6838 detached

[Chris Higgins (chris-higgins)]

Oh... really strange..

Left the machine for a while, came back - tried again - and got the same issue.

Tried to dig in further - so the following activities made the problem go away ( but I have no idea why )

# deborphan
libdbus-1-3
libgc1c2
libasound2
liblzo1
klibc-utils
libdb4.3
libiw28
libc6-i686
root@lump:~# deborphan | xargs apt-get -y remove
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  alsa-utils console-setup initramfs-tools klibc-utils libasound2 libc6-i686
  libdb4.3 libdbus-1-3 libgc1c2 libiw28 liblzo1 pcmciautils ubuntu-minimal
  ubuntu-standard udev volumeid w3m wpasupplicant
0 upgraded, 0 newly installed, 18 to remove and 0 not upgraded.
30 not fully installed or removed.
Need to get 0B of archives.
After unpacking 13.5MB disk space will be freed.
(Reading database ... 12875 files and directories currently installed.)
Removing ubuntu-minimal ...
Removing alsa-utils ...
Removing console-setup ...
Removing pcmciautils ...
Removing volumeid ...
Removing initramfs-tools ...
Removing udev ...
Removing ubuntu-standard ...
Removing w3m ...
Segmentation fault
dpkg: error processing w3m (--remove):
 subprocess post-removal script returned error exit status 139
Removing wpasupplicant ...
Removing klibc-utils ...
Removing libasound2 ...
Removing libc6-i686 ...
Removing libdb4.3 ...
Removing libdbus-1-3 ...
Removing libgc1c2 ...
Removing libiw28 ...
Removing liblzo1 ...
Processing triggers for libc6 ...
ldconfig deferred processing now taking place
E: Sub-process /usr/bin/dpkg returned an error code (1)

root@lump:~# apt-get install dbus
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  libdbus-1-3 libx11-6 libx11-data libxau6 libxdmcp6 x11-common
The following packages will be REMOVED:
  w3m
The following NEW packages will be installed:
  dbus libdbus-1-3 libx11-6 libx11-data libxau6 libxdmcp6 x11-common
0 upgraded, 7 newly installed, 1 to remove and 0 not upgraded.
22 not fully installed or removed.
Need to get 1400kB/1529kB of archives.
After unpacking 3248kB of additional disk space will be used.
Do you want to continue [Y/n]?
Get:1 http://nl.archive.ubuntu.com gutsy/main x11-common 1:7.2-5ubuntu13 [358kB]
Get:2 http://nl.archive.ubuntu.com gutsy/main libxau6 1:1.0.3-2 [11.9kB]
Get:3 http://nl.archive.ubuntu.com gutsy/main libxdmcp6 1:1.0.2-2 [16.9kB]
Get:4 http://nl.archive.ubuntu.com gutsy/main libx11-data 2:1.1.1-1ubuntu4 [190kB]
Get:5 http://nl.archive.ubuntu.com gutsy/main libx11-6 2:1.1.1-1ubuntu4 [607kB]
Get:6 http://nl.archive.ubuntu.com gutsy/main dbus 1.1.1-3ubuntu4 [217kB]
Fetched 1400kB in 0s (6045kB/s)
Preconfiguring packages ...
/dev/mem: mmap: Bad address
(Reading database ... 12204 files and directories currently installed.)
Removing w3m ...
Selecting previously deselected package x11-common.
(Reading database ... 12204 files and directories currently installed.)
Unpacking x11-common (from .../x11-common_1%3a7.2-5ubuntu13_i386.deb) ...
/dev/mem: mmap: Bad address
/dev/mem: mmap: Bad address
Selecting previously deselected package ...

[Jonathan Hitchcock (vhata)]

I had the same problem - upgrading from feisty to gutsy on a xen box, the postinst scripts for things like module-init-tools and procps were failing with a segmentation fault:

# dpkg --configure procps
Setting up procps (1:3.2.7-3ubuntu5) ...
Segmentation fault
dpkg: error processing procps (--configure):
subprocess post-installation script returned error exit status 139

On investigation, it was update-rc.d that was causing the segmentation fault:

# grep update-rc /var/lib/dpkg/info/procps.postinst
                        update-rc.d procps remove >/dev/null
        update-rc.d procps.sh start 17 S . >/dev/null

# update-rc.d procps.sh start 17 S .
Segmentation fault

Running deborphan as above, the only libraries I had in common with Chris were:

liblzo1
klibc-utils
libiw28
libc6-i686

I tried to remove them one by one, to see which the culprit was:

# apt-get remove klibc-utils
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.

# dpkg -r klibc-utils
(Reading database ... 45868 files and directories currently installed.)
Removing klibc-utils ...

# dpkg --configure procps
Setting up procps (1:3.2.7-3ubuntu5) ...
Segmentation fault
dpkg: error processing procps (--configure):
 subprocess post-installation script returned error exit status 139
Errors were encountered while processing:
 procps

# dpkg -r libc6-i686
(Reading database ... 45827 files and directories currently installed.)
Removing libc6-i686 ...
Processing triggers for libc6 ...
ldconfig deferred processing now taking place

# dpkg --configure procps
Setting up procps (1:3.2.7-3ubuntu5) ...
error: "kernel.maps_protect" is an unknown key

Processing triggers for libc6 ...
ldconfig deferred processing now taking place

Things were able to proceed after that, although removing libc6-i686 and klibc-utils caused problems later on - they both needed to be manually reinstalled before other packages could complete their configuration.

[Jonathan Hitchcock (vhata)]

Even after this, running update-rc.d (or even psql, the client for postgresql) gave me segfaults.

I have now installed libc6-xen, and the segfaults go away.

The description for libc6-xen says "This set of libraries is optimized for the Xen hypervisor, and will be selected instead when running under Xen." but it was not - ubuntu-minimal has a hard-dependency on libc6-i686...

[Stefan Tzeggai (alfonx-deactivatedaccount)]

We had the same problem while upgrading from edgy to feisty to gutsy. In gutsy dist-upgrade we got the segfaults.

Our solution in quick notes from memory:

* We left out what was mentioned in the second post.
* Only installed libc6-xen with aptitude (accepted the proposed downgrade solution).
Segfaults were gone now, and we were able to configure the list of unconfigured packages!

After that we still had problems with a newer version of libc (...ubuntu10).

Then we did:

* add multiverse to /etc/apt/sources.lst
* aptitude install ubuntu-xen-server

Getting much further now.. but still config probs

* aptitude remove libc6-i686 ubuntu-minimal sounded like a "reasonable" solution.

and indeed now we're able to successfully finish an aptitude upgrade run.
but:
The following packages have been automatically kept back:
  libc6 libc6-dev

guess that we have to wait for the updated libc6-xen version to meet these dependencies.

[Simon Huerlimann (huerlisi)]

Had the same problem on a Xen machine when upgrading to gutsy. After some investigation I was able to blame it on /usr/sbin/update-rc.d around line 96:

        opendir(DIR, ".");

        foreach $_ (readdir(DIR)) {
            next unless (/^[SK]\d\d$bn$/);
            $fn = "$etcd$i.d/$_";

I was able to reproduce the segfault in other directories using the following perl code:
        opendir(DIR, ".");

        while ($a = readdir(DIR)) {
        print "$a\n";
        }
        exit 1;

It seems to bail out after the '..' directory, but I'm not sure it that's always the case...

Installing libc6-xen didn't help...

Pls tell me if I can investigate any further or when I should report this bug on another package (libc6?).

[gherardo (nospam1-piacitelli)]

I occurred here googling a bit. I have what I regard as a related problem.

I upgraded my release to Gutsy, and then I tried to install ubuntu-server-xen for the first time on this machine.

The installation aborted with synaptic complaining that

          Depends: libc6-xen but it is not going to be installed

Then I tried to install libc6-xen first, which failed with

         PreDepends: libc6 (=2.6.1-1ubuntu9) but 2.6.1-1ubuntu10 is to be installed

libc6-xen is optimized for xen hypervisor and used by xen instead of the standard libc6.

The problem then seems to be that libc6-xen has not been updated.

[Věroš Kaplan (veros-kaplan)]

On my hosts, problem disappeared after installing libc6-xen and deinstalling libc-i686 .

It works - at least now. :-)

[Mikko Ohtamaa (mikko-red-innovation)]

Věroš Kaplan, you saved my day! Removing libc-i686 make everything run again.

[Steven Ellis (steven-openmedia)]

Hit the same problems with an LTS upgrade from Dapper to Hardy with a xen kernel.

Had to dpkg -r libc-i686 and then dpkg --configure -a to fix the install

Also remove ubuntu-minimal and install libc-xen and the latest linux-image-xen

[Palantar (palantar)]

Same a Steven Ellis (Thanks!) Just wanted to mention for other ubuntu noobs like me that the line should read:
"Had to dpkg -r libc6-i686 and then dpkg --configure -a"
(note the 6). It cost me an hour, but I sure learned a lot about packages :)