systemd-fsckd does not allow fsck to finish

[kex]

during normal boots, fsck runs fine with small partitions. But I have two big partitions and systemd seems to abort fsck somewhere between 5 to 10 minutes from start. This results in the partition being checked again at every boot. After manual run in recovery mode, everything is OK again.

See e.g.

 sudo tune2fs -l /dev/mapper/xpcvg-home
tune2fs 1.42.12 (29-Aug-2014)
Filesystem volume name: <none>
Last mounted on: /home
Filesystem UUID: db583624-afb4-41cb-8b2d-640bee2de61b
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent sparse_super large_file uninit_bg
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 65536000
Block count: 262144000
Reserved block count: 13107200
Free blocks: 183474571
Free inodes: 65167292
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 961
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
RAID stride: 128
RAID stripe width: 384
Filesystem created: Mon Jul 25 12:23:50 2011
Last mount time: Sat Feb 20 09:34:41 2016
Last write time: Sat Feb 20 09:34:41 2016
Mount count: 166
Maximum mount count: 157
Last checked: Sun Nov 29 14:02:12 2015
Check interval: 15552000 (6 months)
Next check after: Fri May 27 15:02:12 2016
Lifetime writes: 1113 GB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
First orphan inode: 65143018
Default directory hash: half_md4
Directory Hash Seed: ea4a0233-d8af-4c51-9302-140a4affacdf
Journal backup: inode blocks

journalctl says:
helmi 20 09:30:21 xpc systemd-fsck[720]: /dev/mapper/xpcvg-home has been mounted 165 times without being checked, check forced.
helmi 20 09:34:40 xpc systemd-fsck[720]: fsck: Warning... fsck.ext4 for device /dev/mapper/xpcvg-home exited with signal 13.
helmi 20 09:34:40 xpc systemd-fsck[720]: fsck failed with error code 8.
helmi 20 09:34:40 xpc systemd-fsck[720]: Ignoring error.
helmi 20 09:34:40 xpc systemd[1]: Started File System Check on /dev/mapper/xpcvg-home.

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: systemd 225-1ubuntu9
ProcVersionSignature: Ubuntu 4.2.0-27.32-generic 4.2.8-ckt1
Uname: Linux 4.2.0-27-generic x86_64
ApportVersion: 2.19.1-0ubuntu5
Architecture: amd64
CurrentDesktop: Unity
Date: Sat Feb 20 14:20:24 2016
InstallationDate: Installed on 2012-12-22 (1154 days ago)
InstallationMedia: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
MachineType: ASUS All Series
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-27-generic root=UUID=a21f1300-e2a3-44c9-a667-86fc6184627b ro nomdmonddf nomdmonisw libata.noacpi=1 irqpoll all_generic_ide=1 ide-pci-generic.all-generic-ide=1 usbcore.autosuspend=-1 usbcore.old_scheme_first=1 libata.force=8:1.5G xhci_hcd.quirks=262144
SourcePackage: systemd
UpgradeStatus: Upgraded to wily on 2015-10-23 (119 days ago)
dmi.bios.date: 08/15/2014
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1603
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: MAXIMUS VI EXTREME
dmi.board.vendor: ASUSTeK COMPUTER INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1603:bd08/15/2014:svnASUS:pnAllSeries:pvrSystemVersion:rvnASUSTeKCOMPUTERINC.:rnMAXIMUSVIEXTREME:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: All Series
dmi.product.version: System Version
dmi.sys.vendor: ASUS

[kex (kex-iki)]

Then: running fsck in recovery mode works nice:

root@xpc:~# fsck /dev/mapper/xpcvg-home
fsck – util-linux 2.26.2
e2fsck 1.42.12 (29-Aug-2014)
/dev/mapper/xpcvg-home has been mounted 166 times without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/xpcvg-home: 368728/65536000 files (1.5% non-contiguous), 78669441/262144000 blocks
root@xpc:~# tune2fs -l /dev/mapper/xpcvg-home
tune2fs 1.42.12 (29-Aug-2014)
Filesystem volume name: <none>
Last mounted on: /home
Filesystem UUID: db583624-afb4-41cb-8b2d-640bee2de61b
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent sparse_super large_file uninit_bg
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 65536000
Block count: 262144000
Reserved block count: 13107200
Free blocks: 183474559
Free inodes: 65167272
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 961
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
RAID stride: 128
RAID stripe width: 384
Filesystem created: Mon Jul 25 12:23:50 2011
Last mount time: Sat Feb 20 09:34:41 2016
Last write time: Sat Feb 20 14:39:09 2016
Mount count: 0
Maximum mount count: 157
Last checked: Sat Feb 20 14:39:09 2016
Check interval: 15552000 (6 months)
Next check after: Thu Aug 18 15:39:09 2016
Lifetime writes: 1114 GB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: ea4a0233-d8af-4c51-9302-140a4affacdf
Journal backup: inode blocks

[Martin Pitt (pitti)]

I believe this is fixed by https://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=be10d22 . We should also SRU this fix.

[Martin Pitt (pitti)]

Cherry-picked into Xenial branch: https://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?h=ubuntu-xenial&id=72bc2062

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 231-6git1

---------------
systemd (231-6git1) yakkety; urgency=medium

  Upload current Debian packaging git.

  [ Michael Biebl ]
  * fsckd: Do not exit on idle timeout if there are still clients connected
    systemd-fsckd's event loop terminates if nothing happens for 30 seconds.
    Exiting prematurely while fsck is still running but simply too slow to
    send us progress updates would close the socket and fsck would receive
    SIGPIPE when it writes to the socket. If this happens, the fsck process
    is aborted and the file system check is not completed. (Closes: #788050)
    (LP: #1547844)

  [ Martin Pitt ]
  * 73-usb-net-by-mac.rules: Split kernel command line import line.
    Reportedly this makes the rule actually work on some platforms. Thanks Alp
    Toker! (LP: #1593379)
  * debian/tests/boot-smoke: Only run 5 iterations
  * systemd.postinst: Drop obsolete setcap call for systemd-detect-virt.
    Drop corresponding libcap2-bin dependency.
  * debian/tests/systemd-fsckd: Robustify check for "unit was running"
    (LP: #1624406)
  * debian/extra/set-cpufreq: Use powersave with intel_pstate.
    This is what we did on xenial, and apparently powersave is still actually
    better than performance. Thanks to Doug Smythies for the measurements!
    (LP: #1579278)
  * Ubuntu: Move ondemand.service from static to runtime enablement.
    This makes it easier to keep performance, by disabling ondemand.service.
    Side issue in LP: #1579278

 -- Martin Pitt <email address hidden> Mon, 19 Sep 2016 22:37:51 +0200

[Andy Whitcroft (apw)]

Hello kex, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[kex (kex-iki)]

hi,

the packet fails to install:

kex@xpc:~$ LANG=en sudo apt-get install systemd/xenial-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '229-4ubuntu9' (Ubuntu:16.04/xenial-proposed [amd64]) for 'systemd'
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 systemd : Depends: libsystemd0 (= 229-4ubuntu9) but 229-4ubuntu8 is to be installed
           Recommends: libpam-systemd but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

If I try to install both, a scary list of packages will be removed (I have no courage now to do that):

kex@xpc:~$ LANG=en sudo apt-get install libsystemd0/xenial-proposed systemd/xenial-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '229-4ubuntu9' (Ubuntu:16.04/xenial-proposed [amd64]) for 'libsystemd0'
Selected version '229-4ubuntu9' (Ubuntu:16.04/xenial-proposed [amd64]) for 'systemd'
The following packages were automatically installed and are no longer required:
  account-plugin-tools accountsservice-ubuntu-schemas argyll argyll-ref dbus-property-service gir1.2-gdesktopenums-3.0
  gir1.2-gnomedesktop-3.0 gir1.2-udisks-2.0 gnome-shell-common gstreamer1.0-plugins-base:i386 libaudio2:i386 libcdparanoia0:i386
  libconnectivity-qt1 libdbus-cpp5 libglib2.0-0:i386 libgstreamer-plugins-base1.0-0:i386 libgstreamer1.0-0:i386 libhybris libhybris-utils
  libjpeg62:i386 libmedia1 libmng2:i386 libmysqlclient20:i386 libonline-accounts-client1 libonline-accounts-daemon1 libonline-accounts-qt1
  liboobs-1-5 libopts25 libopus0:i386 liborc-0.4-0:i386 libpkcs11-helper1 libprocess-cpp3 libqgsttools-p1 libqmenumodel0 libqofono-qt5-0
  libqt4-sql:i386 libqt4-sql-mysql:i386 libqt4-xml:i386 libqt5multimedia5-plugins libqt5multimediaquick-p5 libqt5multimediawidgets5
  libqt5systeminfo5 libqt5xmlpatterns5 libqtcore4:i386 libsystemsettings1 libtheora0:i386 libtrust-store2 libubuntuoneauth-2.0-0
  libvisual-0.4-0:i386 libvisual-0.4-plugins:i386 libxss1:i386 libxtst6:i386 libxv1:i386 mate-polkit-common mutter-common
  network-manager-openvpn ntp openvpn plainbox-secure-policy python-gpgme python3-gnupg python3-guacamole python3-jinja2
  python3-markupsafe python3-padme python3-pyparsing python3-xlsxwriter qmenumodel-qml qml-module-qtmultimedia qml-module-qtsysteminfo
  qml-module-ubuntu-connectivity qml-module-ubuntu-onlineaccounts2 qtdeclarative5-gsettings1.0 qtdeclarative5-ofono0.2
  qtdeclarative5-ubuntu-settings-components sqlite3 system-image-common system-image-dbus system-tools-backends ubuntu-keyboard-data
  ubuntuone-credentials-common urfkill
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  apparmor-easyprof apparmor-easyprof-ubuntu click click-apparmor gir1.2-click-0.4 gir1.2-gee-0.8 libboost-log1.58.0 libboos...

[Martin Pitt (pitti)]

You need to install the new libpam-systemd as well at least; there could be more binaries of the systemd source that you have installed. Check with "dpkg -l | grep ^.i.*229-4"

[kex (kex-iki)]

now it worked. the packages I needed:
kex@xpc:~$ sudo dpkg -l | grep 229
ii libpam-systemd:amd64 229-4ubuntu9 amd64 system and service manager - PAM module
ii libsystemd0:amd64 229-4ubuntu9 amd64 systemd utility library
ii libsystemd0:i386 229-4ubuntu9 i386 systemd utility library
ii libudev1:amd64 229-4ubuntu9 amd64 libudev shared library
ii libudev1:i386 229-4ubuntu9 i386 libudev shared library
ii systemd 229-4ubuntu9 amd64 system and service manager
ii systemd-sysv 229-4ubuntu9 amd64 system and service manager - SysV links
ii udev 229-4ubuntu9 amd64 /dev/ and hotplug management daemon

[Martin Pitt (pitti)]

Current SRU got shadowed by a security update, resetting. Will reupload shortly.

[kex (kex-iki)]

this might be unrelated, but since I tried the xenial-proposed packages, the computer has crashed about once a day. Usually I don't get any info on syslog, but I found this on syslog today:
Sep 30 13:32:28 xpc kernel: [19560.717703] INFO: task jbd2/dm-1-8:879 blocked for more than 120 seconds.
Sep 30 13:32:28 xpc kernel: [19560.717706] Tainted: G OE 4.4.0-38-generic #57-Ubuntu
Sep 30 13:32:28 xpc kernel: [19560.717706] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 30 13:32:28 xpc kernel: [19560.717707] jbd2/dm-1-8 D ffff8800af107c98 0 879 2 0x00000000
Sep 30 13:32:28 xpc kernel: [19560.717710] ffff8800af107c98 0000000000000246 ffff8801b8279b80 ffff8801b5d36040
Sep 30 13:32:28 xpc kernel: [19560.717712] ffff8800af108000 ffff8800ae9718b8 ffff8800af107d88 ffff8800af107d70
Sep 30 13:32:28 xpc kernel: [19560.717713] ffff8801b5d36040 ffff8800af107cb0 ffffffff8182c5f5 ffff8800ad776900
Sep 30 13:32:28 xpc kernel: [19560.717714] Call Trace:
Sep 30 13:32:28 xpc kernel: [19560.717720] [<ffffffff8182c5f5>] schedule+0x35/0x80
Sep 30 13:32:28 xpc kernel: [19560.717723] [<ffffffff812ebc40>] jbd2_journal_commit_transaction+0x240/0x1870
Sep 30 13:32:28 xpc kernel: [19560.717727] [<ffffffff810c3d80>] ? wake_atomic_t_function+0x60/0x60
Sep 30 13:32:28 xpc kernel: [19560.717730] [<ffffffff810ec93e>] ? try_to_del_timer_sync+0x5e/0x90
Sep 30 13:32:28 xpc kernel: [19560.717732] [<ffffffff812f0e8a>] kjournald2+0xca/0x250
Sep 30 13:32:28 xpc kernel: [19560.717733] [<ffffffff810c3d80>] ? wake_atomic_t_function+0x60/0x60
Sep 30 13:32:28 xpc kernel: [19560.717735] [<ffffffff812f0dc0>] ? commit_timeout+0x10/0x10
Sep 30 13:32:28 xpc kernel: [19560.717736] [<ffffffff810a08d8>] kthread+0xd8/0xf0
Sep 30 13:32:28 xpc kernel: [19560.717738] [<ffffffff810a0800>] ? kthread_create_on_node+0x1e0/0x1e0
Sep 30 13:32:28 xpc kernel: [19560.717740] [<ffffffff81830a8f>] ret_from_fork+0x3f/0x70
Sep 30 13:32:28 xpc kernel: [19560.717740] [<ffffffff810a0800>] ? kthread_create_on_node+0x1e0/0x1e0
Sep 30 13:32:28 xpc kernel: [19560.717747] INFO: task vdr:1307 blocked for more than 120 seconds.
Sep 30 13:32:28 xpc kernel: [19560.717748] Tainted: G OE 4.4.0-38-generic #57-Ubuntu
Sep 30 13:32:28 xpc kernel: [19560.717748] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 30 13:32:28 xpc kernel: [19560.717749] vdr D ffff880198ac3b08 0 1307 1 0x00000004
Sep 30 13:32:28 xpc kernel: [19560.717750] ffff880198ac3b08 ffff88017b477bc8 ffff8801b827a940 ffff8800aeeb0dc0
Sep 30 13:32:28 xpc kernel: [19560.717751] ffff880198ac4000 ffff8800ae971870 00000000011d3628 ffff8800ae971870
Sep 30 13:32:28 xpc kernel: [19560.717753] ffff8800ad776900 ffff880198ac3b20 ffffffff8182c5f5 ffff8800ae971800
Sep 30 13:32:28 xpc kernel: [19560.717754] Call Trace:
Sep 30 13:32:28 xpc kernel: [19560.717755] [<ffffffff8182c5f5>] schedule+0x35/0x80
Sep 30 13:32:28 xpc kernel: [19560.717757] [<ffffffff812e88da>] wait_transaction_locked+0x8a/0xd0
Sep 30 13:32:28 xpc kernel: [19560.717758] [<ffffffff810c3d80>] ? wake_atomic_t_function+0x60/0x60
Sep 30 13:32:28 xpc kernel: [19560.71...

[kex (kex-iki)]

I have had 229-4ubuntu10 version now for the packages, and the machine has not crashed yet (last crash was Friday morning, so 2 days without crash...

[Chris Halse Rogers (raof)]

Hello kex, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[kex (kex-iki)]

I installed the 229-4ubuntu11 version. If the system is stable till the weekend, and does not hang/crash/livelock/deadlock/whatever happened when I tested the ubuntu9 version, I'll twiddle max mount count to force fsck during the weekend.

[Martin Pitt (pitti)]

Thanks kex for testing! FTR, the kernel oops is unrelated to this issue.

[kex (kex-iki)]

The machine has been stable for a couple of days, and I tested today that fsck finished OK.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 229-4ubuntu11

---------------
systemd (229-4ubuntu11) xenial; urgency=medium

  * 73-usb-net-by-mac.rules: Split kernel command line import line.
    Reportedly this makes the rule actually work on some platforms. Thanks
    Alp Toker! (LP: #1593379)
  * fsckd: Do not exit on idle timeout if there are still clients connected
    (Closes: #788050, LP: #1547844)
  * libnss-*.prerm: Remove possible [key=value] options from NSS modules as
    well. (LP: #1625584)
  * Backport networkd 231. Compared to 229 this has a lot of fixes, some of
    which we need for good netplan support. Backporting them individually
    would be a lot more work and a lot less robust, and we did not use/support
    networkd in 16.04 so far. Drop the other network related patches as they
    are included in this backport now. (LP: #1627641)
  * debian/tests/networkd: Re-enable the the DHCPv6 tests. The DHCPv6
    behaviour is fixed with the above backport now.
  * pid1: process zero-length notification messages again. Just remove the
    assertion, the "n" value was not used anyway. This fixes a local DoS due
    to unprocessed/unclosed fds which got introduced by the previous fix.
    (LP: #1628687)
  * pid1: Robustify manager_dispatch_notify_fd(). If
    manager_dispatch_notify_fd() fails and returns an error then the handling
    of service notifications will be disabled entirely leading to a
    compromised system. (side issue of LP: #1628687)

 -- Martin Pitt <email address hidden> Tue, 04 Oct 2016 21:43:04 +0200

[Martin Pitt (pitti)]

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.