"Loading partman-xfs failed for unknown reasons. Aborting" error in trusty server installations

[Para Siva]

With 20140117 (containing kernel 3.13.0-4-generic) trusty server images the above error message dialog appears during the installation and the following could be seen in the install syslog:

Jan 17 16:49:28 anna[8497]: DEBUG: retrieving partman-target 83ubuntu1
Jan 17 16:49:28 debconf: --> SUBST anna/progress_step_retr PACKAGE partman-target
Jan 17 16:49:28 debconf: Adding [PACKAGE] -> [partman-target]
Jan 17 16:49:28 debconf: <-- 0
Jan 17 16:49:28 debconf: --> PROGRESS INFO anna/progress_step_retr
Jan 17 16:49:28 debconf: <-- 0 OK
Jan 17 16:49:28 anna[8497]: corrupted status flag!!: 0
Jan 17 16:49:28 anna[8497]: (process:10536): tar: write error: No space left on device
Jan 17 16:49:28 anna[8497]: (process:10536): tar: write error: No space left on device
Jan 17 16:49:28 debconf: --> PROGRESS STEP 1
Jan 17 16:49:28 debconf: <-- 0 OK
Jan 17 16:49:28 anna[8497]: DEBUG: retrieving partman-xfs 53
Jan 17 16:49:28 debconf: --> SUBST anna/progress_step_retr PACKAGE partman-xfs
Jan 17 16:49:28 debconf: Adding [PACKAGE] -> [partman-xfs]
Jan 17 16:49:28 debconf: <-- 0
Jan 17 16:49:28 debconf: --> PROGRESS INFO anna/progress_step_retr
Jan 17 16:49:28 debconf: <-- 0 OK
Jan 17 16:49:28 debconf: --> PROGRESS STOP
Jan 17 16:49:28 debconf: <-- 0 OK
Jan 17 16:49:28 debconf: --> SUBST anna/install_failed PACKAGE partman-xfs
Jan 17 16:49:28 debconf: Adding [PACKAGE] -> [partman-xfs]
Jan 17 16:49:28 debconf: <-- 0
Jan 17 16:49:28 debconf: --> INPUT critical anna/install_failed
Jan 17 16:49:28 debconf: <-- 0 question will be asked
Jan 17 16:49:28 debconf: --> GO

Jan 17 16:56:07 debconf: <-- 0 ok
Jan 17 16:56:07 main-menu[2943]: WARNING **: Configuring 'load-cdrom' failed with error code 8
Jan 17 16:56:07 debconf: --> GET debconf/priority
Jan 17 16:56:07 debconf: <-- 0 critical
Jan 17 16:56:07 main-menu[2943]: WARNING **: Menu item 'load-cdrom' failed.
Jan 17 16:56:07 debconf: --> SETTITLE debian-installer/load-cdrom/title
Jan 17 16:56:07 debconf: <-- 0 OK
Jan 17 16:56:07 debconf: --> CAPB
Jan 17 16:56:07 debconf: <-- 0 multiselect backup progresscancel align escape plugin-terminal plugin-detect-keyboard
Jan 17 16:56:07 debconf: --> METAGET debian-installer/load-cdrom/title Description
Jan 17 16:56:07 debconf: <-- 0 Load installer components from CD
Jan 17 16:56:07 debconf: --> SUBST debian-installer/main-menu/item-failure ITEM Load installer components from CD
Jan 17 16:56:07 debconf: Adding [ITEM] -> [Load installer components from CD]
Jan 17 16:56:07 debconf: <-- 0
Jan 17 16:56:07 debconf: --> INPUT critical debian-installer/main-menu/item-failure
Jan 17 16:56:07 debconf: <-- 0 question will be asked

=============
The steps to reproduce:
Install ubuntu server from the the ISO image of 20140117

Impacted jobs are:
http://ci.ubuntu.com/smokeng/trusty/server/i386/20140117/6131/default/
and
http://ci.ubuntu.com/smokeng/trusty/server/amd64/20140117/6130/default/

The installer syslog is attached

[Colin Watson (cjwatson)]

It's running out of space in /. Exactly why is unclear, though; when I unpack coreutils into a subdirectory of /run and use its df and du utilities, df reports / at ~100% use of 240MiB, while du reports only 95MiB used.

[Colin Watson (cjwatson)]

Going back to an i386 image from 20131129 that I happen to have lying around, df reports 39% used on / after retrieving additional installer components, while the absolute figures for used and available are very similar. So I don't think it's that something has grown just over some particular limit; rather, the behaviour of the rootfs seems to have changed.

[Colin Watson (cjwatson)]

Dear kernel team: this appears to be a regression in 3.13.0-4, although I can't quite track down what. Compare the behaviour of these two images:

  http://archive.ubuntu.com/ubuntu/dists/trusty/main/installer-amd64/20101020ubuntu285/images/netboot/mini.iso
  http://archive.ubuntu.com/ubuntu/dists/trusty/main/installer-amd64/20101020ubuntu286/images/netboot/mini.iso

(The first will complain about kernel udebs being missing; you can safely disregard that for the purposes of this bug.)

Take copies of /bin/df and /usr/bin/du on trusty/amd64, and put them somewhere on your network whence you can download them during installation. Now watch "/path/to/df -m" and "/path/to/du -cmsx /" during the "Loading additional components" phase; in the second image, notice how they start out matching but diverge more and more as the phase goes on until there's no more space on rootfs. It looks as though something is holding references to some temporary files so that their space can't be reclaimed, but I can't see what; it doesn't seem to be anything in userspace that I can spot. Help?

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1270228

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Para Siva (psivaa)]

This occurs during the server installation and please see the information and logs given in comments #1 to #4

[Stefan Bader (smb)]

There is one commit that is not yet in 3.13.0-4 but has a very suspicious comment in the commit:

commit d1969a84dd6a44d375aa82bba7d6c38713a429c3
Author: Hugh Dickins <email address hidden>
Date: Thu Jan 16 15:26:48 2014 -0800

    percpu_counter: unbreak __percpu_counter_add()

    Commit 74e72f894d56 ("lib/percpu_counter.c: fix __percpu_counter_add()")
    looked very plausible, but its arithmetic was badly wrong: obvious once
    you see the fix, but maddening to get there from the weird tmpfs ENOSPCs

That would explain why there is ENOSPC reported without du agreeing when counting the usage.

[Andy Whitcroft (apw)]

This has now built and published in 3.13.0-5. Testing seems to indicate the world is a much better place with this kernel. Closing this out.