dpkg hangs while unpacking replacement

[John Felso]

Binary package hint: dpkg

I tried to install libmoon (I know, I know) using apt-get. It hung while unpacking the replacement. The last message was: "Unpacking replacement libmoon". Eventually I had to kill it and reboot. From then on, everytime I ran apt-get it would retry the libmoon installation. Eventually I found a command to stop that from happening (I don't remember what it was). I was thinking it was just that one package. However, now its happening on the ifupdown package.

I'm pretty sure that its actually dpkg that's hanging. I tried running the following command and, sure enough, it hangs:

dpkg -i /var/cache/apt/archives/ifupdown_0.6.8ubuntu29_i386.deb

If I turn on debugging, the last line I see is "deferred extract mass sync". Then nothing.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: dpkg 1.15.5.6ubuntu4.1
ProcVersionSignature: Ubuntu 2.6.32-24.41-generic-pae 2.6.32.15+drm33.5
Uname: Linux 2.6.32-24-generic-pae i686
NonfreeKernelModules: nvidia wl
Architecture: i386
Date: Wed Aug 25 15:39:10 2010
InstallationMedia: Ubuntu 10.04 LTS "Lucid Lynx" - Release i386 (20100429)
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.utf8
 SHELL=/bin/bash
SourcePackage: dpkg

[John Felso (felso)]

I found the code that was printing the "deferred extract mass sync" line. The next line was a Sync() call. That appears to be where it was hanging. So I tried the sync command from bash. It hung. Then it dawned on me that I had a corrupted USB drive plugged in (either no partition table to a corrupt one--not sure). As soon as I removed that device, sync began to work and I was able to install packages again.

It doesn't seem right that a corrupt USB drive could cause dpkg to hang. So, I'm not sure if I should close this or keep it open. I guess sync could be modified to handle that situation gracefully, or dpkg could be tweaked to detect a hung sync call. I probably don't have the bandwidth to contribute a patch either way. I'm just glad its working.

[danmb (danmbox)]

It seems that even a slow USB drive causes dpkg to hang. I was copying about 4 Gb of data to a slow (but otherwise intact) USB drive, and dpkg hung until cp finished. I'm guessing the same could happen for a slow NFS mount etc. This is not a good situation.

[David Finch (ubuntu-mytsoftware)]

I get this too when I install something while copying files over the network with rsync, to a raid which was on different drives than the root partition. dpkg freezes at the unpacking step for many minutes, even though the file copying is happening on different drives. dpkg should not be waiting on or syncing filesystems other than the one it's writing to.

Maybe bug 570805 should be reopened, or at least the fix for 570805 sounds like the likely cause of this problem. The fix was described as replacing per-file fsync() calls with per-file sync() calls, which would make it wait for all mounted filesystems on the system to sync, rather than only the file that was just written to. If crash-consistency is the concern here, it really should be syncing only once after the entire unpack is complete IMO, to avoid mixing new files with old in the event of power loss, and to narrow the time window in which a power loss could leave their system in such an intermediate state.

[David Finch (ubuntu-mytsoftware)]

I also get entries like this in /var/log/messages after dpkg's hang on sync():
kernel: [ 4918.834150] dpkg D 0000000000000000 0 6043 6022 0x00000000
kernel: [ 4918.834156] ffff880108747d38 0000000000000082 0000000000015bc0 0000000000015bc0
kernel: [ 4918.834161] ffff88011e0d4890 ffff880108747fd8 0000000000015bc0 ffff88011e0d44d0
kernel: [ 4918.834170] 0000000000015bc0 ffff880108747fd8 0000000000015bc0 ffff88011e0d4890
kernel: [ 4918.834181] Call Trace:
kernel: [ 4918.834194] [<ffffffff815583ad>] schedule_timeout+0x22d/0x300
kernel: [ 4918.834203] [<ffffffff8105df52>] ? enqueue_entity+0x122/0x1a0
kernel: [ 4918.834211] [<ffffffff8105e015>] ? enqueue_task_fair+0x45/0x90
kernel: [ 4918.834219] [<ffffffff81557656>] wait_for_common+0xd6/0x180
kernel: [ 4918.834227] [<ffffffff8105a254>] ? try_to_wake_up+0x284/0x380
kernel: [ 4918.834231] [<ffffffff8105a350>] ? default_wake_function+0x0/0x20
kernel: [ 4918.834240] [<ffffffff815577bd>] wait_for_completion+0x1d/0x20
kernel: [ 4918.834248] [<ffffffff811666d7>] sync_inodes_sb+0x87/0xb0
kernel: [ 4918.834257] [<ffffffff8116afb2>] __sync_filesystem+0x82/0x90
kernel: [ 4918.834265] [<ffffffff8116b099>] sync_filesystems+0xd9/0x130
kernel: [ 4918.834272] [<ffffffff8116b151>] sys_sync+0x21/0x40
kernel: [ 4918.834281] [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b

[Roland Giesler (lifeboy)]

I also have this problem, but it affects Lucid (10.04) and Maveric (10.10). However, I don't have a flashdrive or anything else but an ext4 system drive in a notebook.

Also there are no messages in my /var/log/messages that pertain to apt, dpkg or the like.

It would be nice to learn how to debug this in more detail. Can anyone help?

[Roland Giesler (lifeboy)]

Here is the process that hangs on my machine:

ps ax | grep 2955

 2955 pts/6 Ds+ 0:00 /usr/bin/dpkg --status-fd 27 --unpack --auto-deconfigure /var/cache/apt/archives/libqt4-xml_4%3a4.6.2-0ubuntu5.1_i386.deb /var/cache/apt/archives/libqt4-dbus_4%3a4.6.2-0ubuntu5.1_i386.deb /var/cache/apt/archives/libqt4-network_4%3a4.6.2-0ubuntu5.1_i386.deb /var/cache/apt/archives/skype_2.1.0.81-1ubuntu5_i386.deb

[dborba (dborba)]

Similar issue here. If I have one of my external drives plugged in there is probably a 30% chance apt &/or dpkg will freeze. Additionally I am usually unable to kill the process once it hangs, requiring a full reboot & a dkpg --configure -a to get things working again.

[Brandon Fouts (bfouts+lp)]

Similar issue here. I have a western digital WDBABM0010BBK 1TB external drive. If its plugged in when I run dpkg it takes about 10 - 15 times longer for installations to occur.

[yevaud (krzysztof-dyszczyk)]

Unpacking replacement ...

Lucid (10.04) and after upgrade Maveric (10.10)
Linux 2.6.35-23-generic #37-Ubuntu SMP Fri Nov 5 19:16:29 UTC 2010 x86_64 GNU/Linux
Dell XPS M1530
Model Family: Seagate Momentus 7200.3 series
Device Model: ST9250421ASG
Firmware Version: DEA2
User Capacity: 250,059,350,016 bytes

[14400.560244] INFO: task dpkg:9105 blocked for more than 120 seconds.
[14400.560252] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[14400.560258] dpkg D 00000001001530ac 0 9105 9073 0x00000000
[14400.560269] ffff8800bf05fdf0 0000000000000086 ffffffff00000000 0000000000015980
[14400.560281] ffff8800bf05ffd8 0000000000015980 ffff8800bf05ffd8 ffff880117028000
[14400.560292] 0000000000015980 0000000000015980 ffff8800bf05ffd8 0000000000015980
[14400.560302] Call Trace:
[14400.560318] [<ffffffff815897fd>] rwsem_down_failed_common+0x9d/0x210
[14400.560329] [<ffffffff8117aa10>] ? sync_one_sb+0x0/0x30
[14400.560337] [<ffffffff815899c6>] rwsem_down_read_failed+0x26/0x30
[14400.560348] [<ffffffff812c2a64>] call_rwsem_down_read_failed+0x14/0x30
[14400.560356] [<ffffffff81588c24>] ? down_read+0x24/0x30
[14400.560366] [<ffffffff811552fb>] iterate_supers+0x7b/0xd0
[14400.560373] [<ffffffff8117aa6f>] sys_sync+0x2f/0x70
[14400.560383] [<ffffffff8100a0f2>] system_call_fastpath+0x16/0x1b

[yevaud (krzysztof-dyszczyk)]

Debian `dpkg' package management program version 1.15.8.4ubuntu1 (amd64).

[bloom88 (mutschlechner-f)]

I've the same problem, but I'm using a SSD (Intel X25-M G2 Postville SSDSA2MH080G2R5)

[IlyaGulya (ilyagulya)]

Yes, i have the same problem on 2 notebooks: ASUS N61Vg and HP Compaq 6720s.

[computorist (travis-hume)]

Grrr ... I was frequently seeing this and blamed it on btrfs. I rebuilt my laptop this weekend and just saw the problem this morning using only ext4 filesystems.

Only difference between this machine and past machines that did not have this problem is:

1. this machine has an Intel SSD
2. I'm remote mounting my music collection using sshfs. I used to use openvpn and samba.

[Anatol (anatolat)]

I often experience this bug and it really annoys me. Also I can't kill dpkg process and do normal reboot.
It seems that this bug and https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/606341 are the same.

Linux anatoly-laptop 2.6.35-22-generic #35-Ubuntu SMP Sat Oct 16 20:36:48 UTC 2010 i686 GNU/Linux

[127920.304096] INFO: task dpkg:7626 blocked for more than 120 seconds.
[127920.304104] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[127920.304110] dpkg D c6985f0c 0 7626 6868 0x00000000
[127920.304121] c6985f1c 00000086 00000002 c6985f0c c6985f48 c05d89e0 c08c3700 c08c3700
[127920.304137] 29958238 00007339 c08c3700 c08c3700 29952f48 00007339 00000000 c08c3700
[127920.304153] c08c3700 e02f6580 00000001 e02f6580 c6985f54 efe65e3c c6985f4c c05c89bd
[127920.304168] Call Trace:
[127920.304184] [<c05c89bd>] rwsem_down_failed_common+0x7d/0x1b0
[127920.304192] [<c05c8b3d>] rwsem_down_read_failed+0x1d/0x28
[127920.304200] [<c05c8b83>] call_rwsem_down_read_failed+0x7/0xc
[127920.304208] [<c05c800c>] ? down_read+0x1c/0x20
[127920.304216] [<c021a9f0>] iterate_supers+0x60/0xb0
[127920.304225] [<c023ad90>] ? sync_one_sb+0x0/0x30
[127920.304233] [<c023ade9>] sys_sync+0x29/0x60
[127920.304240] [<c05c9114>] syscall_call+0x7/0xb
[127920.304249] [<c05c0000>] ? cpu_init+0x207/0x249

[C de-Avillez (hggdh2)]

I see here some of the commenters showing a kernel OOPS. Although they are a bit different, some of them report a kernel trace; the common piece is a call to sys_sync.

I wonder if all of you have remote filesystems (like NFS), or USB-mounted disc devices.

[IlyaGulya (ilyagulya)]

Yes, i have the eSATA mounted Hard Drive.

[David Finch (ubuntu-mytsoftware)]

My root / was on an sd card (ext4), which would count as usb, though the only other disk access besides dpkg at the time was rsync writing to physical disks (ext4 mounted to /backup on lvm on md raid 1 made of two sata disks connected to an lsi 1068e sas controller). If not for dpkg's reliance on sync(), the two operations should have occurred completely independent of each other, rather than one freezing the other for what turned out to be several hours.

[JP Meijers (jpm)]

Ubuntu 10.10 fully updated.
Root partition is on a OCZ Vertex 2 SSD.
I have an usb external hdd plugged in which works fine although it sometimes gives hardware errors.

This bug introduced itself since I've been using this external.

[Jeremy Foshee (jeremyfoshee)]

Hi John,

If you could also please test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

    [This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

[Rostislav Stříbrný (rstribrn)]

Also happened to me today. I've got ext4 filesystem on Lenovo T61 laptop.
Running latest Ubuntu 10.10 x86_64 with kernel from kernel-ppa (linux-image-2.6.37-12-generic).

dpkg hangs when unpacking (standard repository) replacement linux-image-2.6.35-25-generic 2.6.35-25.43.

[IcyJ (jeremy-michelle)]

This also happened to me today. I was installing updates and had a Hitachi 250GB external hard drive mounted. I unmounted the drive while updates were in progress and now my update manager is stuck at "unpacking replacement linux-image-2.6.35-25-generic...".

[Greg Pangrazio (pangrazi)]

I was having the same problem on one of my servers running 2.6.32-21-server 64 bit. Once I stopped MySQL, apache2, nagios and snmpd the disk IO slowed to the point where a sync was possible. Once the sync completed the dpkg ended. I did need to run dpkg --configure -a to get dpkg working properly again. Let me know if any additional information would be helpful.

[IlyaGulya (ilyagulya)]

dmesg out:

[873240.404582] INFO: task dpkg:8624 blocked for more than 120 seconds.
[873240.404586] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[873240.404589] dpkg D c05c97f8 0 8624 8392 0x00000000
[873240.404594] eb8c7f1c 00200086 eb8c7ed8 c05c97f8 efdba000 eb8c7f18 c08c4700 c08c4700
[873240.404602] db4ba704 00031a0f c08c4700 c08c4700 00000000 00031a0f ee24bc00 c08c4700
[873240.404610] c08c4700 f1c08cb0 eb8c7f2c f1c08cb0 eb8c7f54 c51d983c eb8c7f4c c05c956d
[873240.404617] Call Trace:
[873240.404625] [<c05c97f8>] ? _raw_spin_lock_irq+0x18/0x20
[873240.404629] [<c05c956d>] rwsem_down_failed_common+0x7d/0x1b0
[873240.404633] [<c05c96ed>] rwsem_down_read_failed+0x1d/0x28
[873240.404637] [<c05c9733>] call_rwsem_down_read_failed+0x7/0xc
[873240.404640] [<c05c8bbc>] ? down_read+0x1c/0x20
[873240.404645] [<c021ae60>] iterate_supers+0x60/0xb0
[873240.404649] [<c023b2c0>] ? sync_one_sb+0x0/0x30
[873240.404653] [<c023b319>] sys_sync+0x29/0x60
[873240.404656] [<c05c9cc4>] syscall_call+0x7/0xb
[873240.404661] [<c05c0000>] ? cpuid4_cache_sysfs_init+0x101/0x105
---------------------
ilyagulya@ilyagulya-asus:~$ lspci | grep SATA
00:1f.2 SATA controller: Intel Corporation ICH9M/M-E SATA AHCI Controller (rev 03)

[IlyaGulya (ilyagulya)]

ilyagulya@ilyagulya-asus:~$ uname -a
Linux ilyagulya-asus 2.6.35-24-generic #42-Ubuntu SMP Thu Dec 2 01:41:57 UTC 2010 i686 GNU/Linux

[Charles Nadeau (charles-nadeau)]

Same problem here while updating my system (Xubuntu 10.10)

charles@ottawa:/$ uname -a
Linux ottawa 2.6.35-26-generic #46-Ubuntu SMP Sun Jan 30 06:59:07 UTC 2011 x86_64 GNU/Linux

charles@ottawa:/$ dpkg --version
Debian `dpkg' package management program version 1.15.8.4ubuntu1 (amd64).

charles@ottawa:/$ ps aux | grep dpkg
root 4019 0.0 1.0 106928 88236 pts/2 Ds+ Feb15 0:00 /usr/bin/dpkg --status-fd 39 --unpack --auto-deconfigure /var/cache/apt/archives/tzdata-java_2011b-0ubuntu0.10.10_all.deb /var/cache/apt/archives/tzdata_2011b-0ubuntu0.10.10_all.deb

charles@ottawa:/$ lspci
00:00.0 Host bridge: nVidia Corporation C55 Host Bridge (rev a2)
00:00.1 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:00.2 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:00.3 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:00.4 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:00.5 RAM memory: nVidia Corporation C55 Memory Controller (rev a2)
00:00.6 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:00.7 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:01.0 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:01.1 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:01.2 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:01.3 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:01.4 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:01.5 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:01.6 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:02.0 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:02.1 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:02.2 RAM memory: nVidia Corporation C55 Memory Controller (rev a1)
00:03.0 PCI bridge: nVidia Corporation C55 PCI Express bridge (rev a1)
00:09.0 RAM memory: nVidia Corporation MCP55 Memory Controller (rev a1)
00:0a.0 ISA bridge: nVidia Corporation MCP55 LPC Bridge (rev a2)
00:0a.1 SMBus: nVidia Corporation MCP55 SMBus (rev a2)
00:0b.0 USB Controller: nVidia Corporation MCP55 USB Controller (rev a1)
00:0b.1 USB Controller: nVidia Corporation MCP55 USB Controller (rev a2)
00:0d.0 IDE interface: nVidia Corporation MCP55 IDE (rev a1)
00:0e.0 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a2)
00:0e.1 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a2)
00:0e.2 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a2)
00:0f.0 PCI bridge: nVidia Corporation MCP55 PCI bridge (rev a2)
00:0f.1 Audio device: nVidia Corporation MCP55 High Definition Audio (rev a2)
00:11.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2)
00:12.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2)
00:14.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a2)
00:15.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a2)
01:00.0 VGA compatible controller: nVidia Corporation G73 [GeForce 7600 GS] (rev a1)
02:06.0 Serial controller: Equinox Systems, Inc. SST-4P Adapter (rev 04)
02:07.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX5081 8-port SATA I PCI-X ...

[Nikita (nikitakv)]

Exact same problem running Ubuntu 64-bit 10.04 on ASUS K61IC-X3 T6600.

Thread about it here:
http://ubuntuforums.org/showthread.php?p=10467368#post10467368

I guess the only resolution is unmount any external io devices before running dpkg?

[Graham (grahamjenson)]

This bug has also affected me. I was upgrading (apt-get upgrade), and it always hung at unpacking libpoppler7, even after restarting.
After removing the USB drive, restarting then running dpkg --configure -a, it successfully installed libpoppler7.

[Rich Wohlstadter (rwohlsta)]

Add me to the affected list as well. In our instance, dpkg will hang when under heavy i/o pressure from nfs mounted volumes

[Jean-Luc Boss (jlb74)]

The same for me, while copying from a large btrfs volume ...

[martin suchanek (martin-suc)]

I have the same problem on ubuntu 12.04 64bit and problem is sync command.

[SamTzu (sami-mattila)]

Looks like this is bugging me on new Kali 2 (Debian) 32bit system.
System freezes every time for about 15 secs after apt-get install (or upgrade) is done.

[Patrick Drotleff (patrick-d)]

Same problem within LXD container on ubuntu 16.04 64bit
Linux patrickd 4.4.0-38-generic #57-Ubuntu SMP Tue Sep 6 15:42:33 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

It's a server.. so there isn't really any external harddrive to remove to fix it :-/