Activity log for bug #1875863

Date Who What changed Old value New value Message
2020-04-29 11:00:16 bugproxy bug added bug
2020-04-29 11:00:19 bugproxy tags architecture-s39064 bugnameltc-185690 severity-high targetmilestone-inin2004
2020-04-29 11:00:20 bugproxy attachment added Kernel messages related to failed mount operation on the client machine https://bugs.launchpad.net/bugs/1875863/+attachment/5363442/+files/ceph-mount-failure-messages.txt
2020-04-29 11:00:22 bugproxy attachment added Proposed fixup https://bugs.launchpad.net/bugs/1875863/+attachment/5363443/+files/ceph-fix-up-endian-bug-in-managing-feature-bits.eml
2020-04-29 11:00:25 bugproxy ubuntu: assignee Skipper Bug Screeners (skipper-screen-team)
2020-04-29 11:00:29 bugproxy affects ubuntu ceph (Ubuntu)
2020-04-29 11:51:47 Frank Heimes bug task added ubuntu-z-systems
2020-04-29 11:51:57 Frank Heimes ubuntu-z-systems: importance Undecided High
2020-04-29 11:52:44 Frank Heimes ubuntu-z-systems: assignee Ubuntu OpenStack (ubuntu-openstack)
2020-04-29 11:55:11 Frank Heimes tags architecture-s39064 bugnameltc-185690 severity-high targetmilestone-inin2004 architecture-s39064 bugnameltc-185690 openstack-ibm severity-high targetmilestone-inin2004
2020-04-29 13:50:11 Frank Heimes ubuntu-z-systems: status New Incomplete
2020-04-29 15:24:36 James Page bug task added linux (Ubuntu)
2020-04-29 15:24:42 James Page ceph (Ubuntu): status New Invalid
2020-04-30 06:54:23 Frank Heimes ubuntu-z-systems: assignee Ubuntu OpenStack (ubuntu-openstack) Skipper Bug Screeners (skipper-screen-team)
2020-04-30 06:54:36 Frank Heimes linux (Ubuntu): assignee Canonical Kernel Team (canonical-kernel-team)
2020-04-30 06:54:44 Frank Heimes bug task deleted ceph (Ubuntu)
2020-04-30 06:55:22 Frank Heimes linux (Ubuntu): importance Undecided High
2020-04-30 19:31:21 Ulrich Weigand bug added subscriber Ulrich Weigand
2020-05-11 09:05:48 Frank Heimes linux (Ubuntu): status New In Progress
2020-05-11 09:05:53 Frank Heimes ubuntu-z-systems: status Incomplete In Progress
2020-05-13 08:12:52 Frank Heimes linux (Ubuntu): status In Progress Fix Committed
2020-05-13 08:12:55 Frank Heimes ubuntu-z-systems: status In Progress Fix Committed
2020-05-13 10:05:05 Frank Heimes nominated for series Ubuntu Groovy
2020-05-13 10:05:05 Frank Heimes bug task added linux (Ubuntu Groovy)
2020-05-13 10:05:05 Frank Heimes nominated for series Ubuntu Focal
2020-05-13 10:05:05 Frank Heimes bug task added linux (Ubuntu Focal)
2020-05-13 10:05:14 Frank Heimes linux (Ubuntu Groovy): status Fix Committed In Progress
2020-05-13 10:05:20 Frank Heimes linux (Ubuntu Focal): status New In Progress
2020-05-13 14:49:24 Frank Heimes description When mounting a ceph volume, mount operation fails with an IO error. The problem is always reproducible. Identified potential root cause as kernel endian bug: In the function handle_session() variable @features always contains little endian order of bytes. Just because The feature mask sent by the MDS is little-endian (bits are packed bytewise from left to right in encode_supported_features()). However, test_bit(), called to check features availability, assumes the host order of bytes in that variable. This leads to problems on big endian architectures. Specifically it is impossible to mount ceph volume on s390. A fixup was proposed to convert little-endian order of bytes to the host one. That fixup was modified by ceph maintainers to use existing unpacking means for the conversion. The resulted patch attached. Related discussion in the ceph-development mailing list: https://marc.info/?l=ceph-devel&m=158815357301332&w=2 SRU Justification: ================== [Impact] * Unable to mount ceph volumes on big endian systems, like s390x. * The mount operation always fails with an IO error. * This is caused by an endiness issue in function handle_session where variable features is always little endian. * But test_bit assumes the host order of bytes, hence causes a problem on big endian systems. [Fix] * 0fa8263367db9287aa0632f96c1a5f93cc478150 0fa8263367db "ceph: fix endianness bug when handling MDS session feature bits" [Test Case] * Setup ceph on s390x. * Try to mount a ceph volume. * If it mounts correctly the patch is applied and working. * Without the patch a mount always fails on big endian / s390x. [Regression Potential] * There is regression potential with having code changes in ceph's session handler, which is common code. * However, the patch was accepted (slightly changed) by the ceph maintainers and with that got upstream accepted, too. * The patch is fairly limited (5 lines removed, 3 added), hence the changes are quite traceable. __________ When mounting a ceph volume, mount operation fails with an IO error. The problem is always reproducible. Identified potential root cause as kernel endian bug: In the function handle_session() variable @features always contains little endian order of bytes. Just because The feature mask sent by the MDS is little-endian (bits are packed bytewise from left to right in encode_supported_features()). However, test_bit(), called to check features availability, assumes the host order of bytes in that variable. This leads to problems on big endian architectures. Specifically it is impossible to mount ceph volume on s390. A fixup was proposed to convert little-endian order of bytes to the host one. That fixup was modified by ceph maintainers to use existing unpacking means for the conversion. The resulted patch attached. Related discussion in the ceph-development mailing list: https://marc.info/?l=ceph-devel&m=158815357301332&w=2
2020-05-14 03:59:07 Khaled El Mously linux (Ubuntu Focal): status In Progress Fix Committed
2020-05-14 05:15:43 Frank Heimes linux (Ubuntu Focal): assignee Canonical Kernel Team (canonical-kernel-team)
2020-05-19 14:32:34 Ubuntu Kernel Bot tags architecture-s39064 bugnameltc-185690 openstack-ibm severity-high targetmilestone-inin2004 architecture-s39064 bugnameltc-185690 openstack-ibm severity-high targetmilestone-inin2004 verification-needed-focal
2020-05-25 15:09:44 bugproxy tags architecture-s39064 bugnameltc-185690 openstack-ibm severity-high targetmilestone-inin2004 verification-needed-focal architecture-s39064 bugnameltc-185690 openstack-ibm severity-high targetmilestone-inin2004 verification-failed-focal
2020-06-08 09:09:15 Frank Heimes tags architecture-s39064 bugnameltc-185690 openstack-ibm severity-high targetmilestone-inin2004 verification-failed-focal architecture-s39064 bugnameltc-185690 openstack-ibm severity-high targetmilestone-inin2004 verification-done-focal
2020-06-09 21:04:04 Launchpad Janitor linux (Ubuntu Focal): status Fix Committed Fix Released
2020-06-09 21:04:04 Launchpad Janitor cve linked 2020-0543
2020-07-28 00:57:39 Launchpad Janitor linux (Ubuntu Groovy): status In Progress Fix Released
2020-07-28 00:57:39 Launchpad Janitor cve linked 2019-16089
2020-07-28 00:57:39 Launchpad Janitor cve linked 2019-19642
2020-07-28 00:57:39 Launchpad Janitor cve linked 2020-11935
2020-07-28 05:52:04 Frank Heimes ubuntu-z-systems: status Fix Committed Fix Released