cephfs mounting error after kernel image update on ubuntu 16.04 LTS

Bug #1737506 reported by Thomas Blick
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux-aws (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

I experienced an issue with mounting CephFS shares after upgrading linux-image to linux-image-4.4.0-1043-aws and/or linux-image-4.4.0-103-generic.

After rollback and downgrade back to linux-images linux-image-4.4.0-101-generic and linux-image-4.4.0-1041-aws mounting was fine again.

While having booted with the updated kernel image and while the mount error was present (error message on console: input/output error) I noticed the following logging in syslog - maybe that helps:

Dec 8 08:42:00 ip-172-31-29-78 kernel: [ 9.949106] FS-Cache: Loaded
Dec 8 08:42:00 ip-172-31-29-78 kernel: [ 9.960194] Key type ceph registered
Dec 8 08:42:00 ip-172-31-29-78 kernel: [ 9.960267] libceph: loaded (mon/osd proto 15/24)
Dec 8 08:42:00 ip-172-31-29-78 kernel: [ 9.972192] FS-Cache: Netfs 'ceph' registered for caching
Dec 8 08:42:00 ip-172-31-29-78 kernel: [ 9.972201] ceph: loaded (mds proto 32)
Dec 8 08:42:00 ip-172-31-29-78 kernel: [ 9.995058] libceph: client582051 fsid 0860999e-89dd-483d-8b42-56f7d84bf4e6
Dec 8 08:42:00 ip-172-31-29-78 kernel: [ 9.997806] libceph: mon0 172.31.10.252:6789 session established
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.005674] ceph: problem parsing mds trace -5
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.010254] ceph: mds parse_reply err -5
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.020073] ceph: mdsc_handle_reply got corrupt reply mds0(tid:1)
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031179] header: 00000000: 02 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 ................
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031181] header: 00000010: 1a 00 7f 00 01 00 4c 01 00 00 00 00 00 00 00 00 ......L.........
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031183] header: 00000020: 00 00 00 00 02 00 00 00 00 00 00 00 00 01 00 00 ................
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031184] header: 00000030: 00 27 ed 97 95 .'...
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031185] front: 00000000: 01 01 00 00 00 00 00 00 8f 00 00 00 01 00 01 01 ................
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031186] front: 00000010: 01 00 00 01 00 00 00 00 00 00 00 fe ff ff ff ff ................
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031187] front: 00000020: ff ff ff 00 00 00 00 e0 c5 15 00 00 00 00 00 01 ................
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031188] front: 00000030: 00 00 00 00 00 00 00 55 01 00 00 00 00 00 00 32 .......U.......2
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031189] front: 00000040: 1d 33 00 00 00 00 00 01 00 00 00 00 00 00 00 01 .3..............
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031190] front: 00000050: 00 00 00 00 00 00 00 01 00 00 40 00 01 00 00 00 ..........@.....
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031191] front: 00000060: 00 00 40 00 00 00 00 00 00 00 00 00 00 00 00 00 ..@.............
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031192] front: 00000070: 01 00 00 00 44 bc 16 5a b7 99 78 06 44 bc 16 5a ....D..Z..x.D..Z
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031193] front: 00000080: b7 99 78 06 00 00 00 00 00 00 00 00 00 00 00 00 ..x.............
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031194] front: 00000090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031195] front: 000000a0: ff ff ff ff ff ff ff ff 00 00 00 00 ed 41 00 00 .............A..
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031196] front: 000000b0: f2 01 00 00 f0 01 00 00 01 00 00 00 00 00 00 00 ................
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031197] front: 000000c0: 00 00 00 00 05 00 00 00 00 00 00 00 a1 72 a3 34 .............r.4
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031198] front: 000000d0: 03 00 00 00 b3 1b 01 00 00 00 00 00 6c 3a 00 00 ............l:..
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031199] front: 000000e0: 00 00 00 00 56 50 2a 5a d2 06 94 3b 00 00 00 00 ....VP*Z...;....
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031200] front: 000000f0: 00 00 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ................
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031201] front: 00000100: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031202] front: 00000110: 00 00 00 00 00 00 00 00 30 00 00 00 01 00 00 00 ........0.......
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031203] front: 00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031204] front: 00000130: 00 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 ................
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031205] front: 00000140: 00 00 00 00 00 00 00 00 00 00 00 00 ............
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031206] footer: 00000000: c6 c2 b6 a6 00 00 00 00 00 00 00 00 9d 1f 2d f1 ..............-.
Dec 8 08:42:01 ip-172-31-29-78 kernel: [ 10.031207] footer: 00000010: 44 a5 8c c9 05 D....

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-1043-aws 4.4.0-1043.52
ProcVersionSignature: User Name 4.4.0-1043.52-aws 4.4.98
Uname: Linux 4.4.0-1043-aws x86_64
ApportVersion: 2.20.1-0ubuntu2.14
Architecture: amd64
Date: Mon Dec 11 09:26:11 2017
Ec2AMI: ami-8504fdea
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: eu-central-1b
Ec2InstanceType: t2.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: linux-aws
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Thomas Blick (thbl) wrote :
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux-aws (Ubuntu):
status: New → Confirmed
Revision history for this message
Tim (timw-suqld) wrote :
Download full text (3.5 KiB)

Same issue after kernel upgrade using cloud-image

Linux prod2 4.4.0-103-generic #126-Ubuntu SMP Mon Dec 4 16:23:28 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ apt-cache policy linux-image-virtual
linux-image-virtual:
  Installed: 4.4.0.103.108
  Candidate: 4.4.0.103.108
  Version table:
 *** 4.4.0.103.108 500
        500 http://mirror.overthewire.com.au/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        100 /var/lib/dpkg/status
     4.4.0.21.22 500
        500 http://mirror.overthewire.com.au/ubuntu xenial/main amd64 Packages

[ 403.915557] libceph: client16654110 fsid 66ff6cd5-173c-4272-b23f-fc82629141bf
[ 403.917153] libceph: mon0 10.1.4.111:6789 session established
[ 403.920944] ceph: problem parsing mds trace -5
[ 403.921696] ceph: mds parse_reply err -5
[ 403.922349] ceph: mdsc_handle_reply got corrupt reply mds0(tid:1)
[ 403.923340] header: 00000000: 02 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 ................
[ 403.923342] header: 00000010: 1a 00 7f 00 01 00 4c 01 00 00 00 00 00 00 00 00 ......L.........
[ 403.923343] header: 00000020: 00 00 00 00 02 00 00 00 00 00 00 00 00 01 00 00 ................
[ 403.923344] header: 00000030: 00 27 ed 97 95 .'...
[ 403.923345] front: 00000000: 01 01 00 00 00 00 00 00 0f 04 00 00 01 00 01 01 ................
[ 403.923346] front: 00000010: 01 00 00 01 00 00 00 00 00 00 00 fe ff ff ff ff ................
[ 403.923347] front: 00000020: ff ff ff 00 00 00 00 4c b2 02 00 00 00 00 00 01 .......L........
[ 403.923348] front: 00000030: 00 00 00 00 00 00 00 55 01 00 00 00 00 00 00 52 .......U.......R
[ 403.923349] front: 00000040: eb 48 14 00 00 00 00 01 00 00 00 00 00 00 00 01 .H..............
[ 403.923350] front: 00000050: 00 00 00 00 00 00 00 01 00 00 40 00 01 00 00 00 ..........@.....
[ 403.923351] front: 00000060: 00 00 40 00 00 00 00 00 00 00 00 00 00 00 00 00 ..@.............
[ 403.923352] front: 00000070: 05 00 00 00 ba 23 5b 58 44 a8 f2 16 ba 23 5b 58 .....#[XD....#[X
[ 403.923353] front: 00000080: 44 a8 f2 16 00 00 00 00 00 00 00 00 00 00 00 00 D...............
[ 403.923354] front: 00000090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 403.923355] front: 000000a0: ff ff ff ff ff ff ff ff 00 00 00 00 ed 41 00 00 .............A..
[ 403.923356] front: 000000b0: 00 00 00 00 00 00 00 00 01 00 00 00 01 00 00 00 ................
[ 403.923357] front: 000000c0: 00 00 00 00 01 00 00 00 00 00 00 00 ae 5b 2c 52 .............[,R
[ 403.923358] front: 000000d0: 05 00 00 00 21 00 01 00 00 00 00 00 1d 9f 01 00 ....!...........
[ 403.923359] front: 000000e0: 00 00 00 00 ac 73 32 5a 90 4c 48 20 00 00 00 00 .....s2Z.LH ....
[ 403.923360] front: 000000f0: 00 00 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ................
[ 403.923361] front: 00000100: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
[ 403.923362] front: 00000110: 00 00 00 00 00 00 00 00 30 00 00 00 01 00 00 00 ........0.......
[ 403.923363] front: 00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 403.92...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.