| 2020-08-14 00:31:20 |
Dan Hill |
bug |
|
|
added bug |
| 2020-08-14 00:31:42 |
Dan Hill |
description |
[Impact]
Upstream tracker: issue#46443 [0].
The ceph-osd service can crash when processing osd map updates.
When the osd encounters a CRC error while processing an incremental map update, it will request a full map update from its peers. In this code path, an uninitialized variable was recently introduced and that will get de-referenced causing a crash.
The uninitialized variable was introduced in nautilus 14.2.10, and octopus 15.2.1.
[Test Case]
# Inject osd_inject_bad_map_crc_probability = 1
sudo ceph daemon osd.{id} config set osd_inject_bad_map_crc_probability 1
# Trigger some osd map updates by restarting a different osd
sudo systemctl restart osd@{diff-id}
[Other Info]
Upstream has released a fix for this issue in Nautilus 14.2.11. The SRU for this point release is being tracked by LP: #1891077
Upstream has merged a fix for this issue in Octopus, but there is no current release target. The ceph packages in focal, groovy, and the ussuri cloud archive are exposed to this critical regression.
[0] https://tracker.ceph.com/issues/46443
[1] https://github.com/ceph/ceph/pull/36340 |
[Impact]
Upstream tracker: issue#46443 [0].
The ceph-osd service can crash when processing osd map updates.
When the osd encounters a CRC error while processing an incremental map update, it will request a full map update from its peers. In this code path, an uninitialized variable was recently introduced and that will get de-referenced causing a crash.
The uninitialized variable was introduced in nautilus 14.2.10, and octopus 15.2.1.
[Test Case]
# Inject osd_inject_bad_map_crc_probability = 1
sudo ceph daemon osd.{id} config set osd_inject_bad_map_crc_probability 1
# Trigger some osd map updates by restarting a different osd
sudo systemctl restart osd@{diff-id}
[Other Info]
Upstream has released a fix for this issue in Nautilus 14.2.11. The SRU for this point release is being tracked by LP: #1891077
Upstream has merged a fix for this issue in Octopus [1], but there is no current release target. The ceph packages in focal, groovy, and the ussuri cloud archive are exposed to this critical regression.
[0] https://tracker.ceph.com/issues/46443
[1] https://github.com/ceph/ceph/pull/36340 |
|
| 2020-08-14 00:32:13 |
Dan Hill |
nominated for series |
|
Ubuntu Groovy |
|
| 2020-08-14 00:32:13 |
Dan Hill |
bug task added |
|
ceph (Ubuntu Groovy) |
|
| 2020-08-14 00:32:13 |
Dan Hill |
nominated for series |
|
Ubuntu Focal |
|
| 2020-08-14 00:32:13 |
Dan Hill |
bug task added |
|
ceph (Ubuntu Focal) |
|
| 2020-08-14 02:08:07 |
Billy Olsen |
bug task added |
|
cloud-archive |
|
| 2020-08-14 02:08:43 |
Billy Olsen |
nominated for series |
|
cloud-archive/ussuri |
|
| 2020-08-14 02:08:43 |
Billy Olsen |
bug task added |
|
cloud-archive/ussuri |
|
| 2020-08-14 02:08:43 |
Billy Olsen |
nominated for series |
|
cloud-archive/victoria |
|
| 2020-08-14 02:08:43 |
Billy Olsen |
bug task added |
|
cloud-archive/victoria |
|
| 2020-08-14 13:07:41 |
Corey Bryant |
cloud-archive/ussuri: status |
New |
Triaged |
|
| 2020-08-14 13:07:43 |
Corey Bryant |
cloud-archive/victoria: status |
New |
Triaged |
|
| 2020-08-14 13:07:45 |
Corey Bryant |
ceph (Ubuntu Focal): status |
New |
Triaged |
|
| 2020-08-14 13:07:47 |
Corey Bryant |
ceph (Ubuntu Groovy): status |
New |
Triaged |
|
| 2020-08-14 13:07:49 |
Corey Bryant |
cloud-archive/ussuri: importance |
Undecided |
Critical |
|
| 2020-08-14 13:07:51 |
Corey Bryant |
cloud-archive/victoria: importance |
Undecided |
Critical |
|
| 2020-08-14 13:07:52 |
Corey Bryant |
ceph (Ubuntu Focal): importance |
Undecided |
Critical |
|
| 2020-08-14 13:07:53 |
Corey Bryant |
ceph (Ubuntu Groovy): importance |
Undecided |
Critical |
|
| 2020-08-19 13:19:13 |
Robie Basak |
ceph (Ubuntu Groovy): status |
Triaged |
Fix Committed |
|
| 2020-08-19 14:44:42 |
Launchpad Janitor |
ceph (Ubuntu Groovy): status |
Fix Committed |
Fix Released |
|
| 2020-08-19 15:16:58 |
Corey Bryant |
description |
[Impact]
Upstream tracker: issue#46443 [0].
The ceph-osd service can crash when processing osd map updates.
When the osd encounters a CRC error while processing an incremental map update, it will request a full map update from its peers. In this code path, an uninitialized variable was recently introduced and that will get de-referenced causing a crash.
The uninitialized variable was introduced in nautilus 14.2.10, and octopus 15.2.1.
[Test Case]
# Inject osd_inject_bad_map_crc_probability = 1
sudo ceph daemon osd.{id} config set osd_inject_bad_map_crc_probability 1
# Trigger some osd map updates by restarting a different osd
sudo systemctl restart osd@{diff-id}
[Other Info]
Upstream has released a fix for this issue in Nautilus 14.2.11. The SRU for this point release is being tracked by LP: #1891077
Upstream has merged a fix for this issue in Octopus [1], but there is no current release target. The ceph packages in focal, groovy, and the ussuri cloud archive are exposed to this critical regression.
[0] https://tracker.ceph.com/issues/46443
[1] https://github.com/ceph/ceph/pull/36340 |
[Impact]
Upstream tracker: issue#46443 [0].
The ceph-osd service can crash when processing osd map updates.
When the osd encounters a CRC error while processing an incremental map update, it will request a full map update from its peers. In this code path, an uninitialized variable was recently introduced and that will get de-referenced causing a crash.
The uninitialized variable was introduced in nautilus 14.2.10, and octopus 15.2.1.
[Test Case]
# Inject osd_inject_bad_map_crc_probability = 1
sudo ceph daemon osd.{id} config set osd_inject_bad_map_crc_probability 1
# Trigger some osd map updates by restarting a different osd
sudo systemctl restart osd@{diff-id}
[Regression Potential]
The code has been updated to leave handle_osd_maps() early if a CRC error is encountered, therefore preventing the commit if the failure is encountered while processing an incremental map update. This will make the full map update take longer but should prevent the crash that resulted in this bug. Additionally _committed_osd_maps() is now coded to abort if first <= last, but it is assumed that code should never be reached.
[Other Info]
Upstream has released a fix for this issue in Nautilus 14.2.11. The SRU for this point release is being tracked by LP: #1891077
Upstream has merged a fix for this issue in Octopus [1], but there is no current release target. The ceph packages in focal, groovy, and the ussuri cloud archive are exposed to this critical regression.
[0] https://tracker.ceph.com/issues/46443
[1] https://github.com/ceph/ceph/pull/36340 |
|
| 2020-08-19 17:09:18 |
Dan Hill |
description |
[Impact]
Upstream tracker: issue#46443 [0].
The ceph-osd service can crash when processing osd map updates.
When the osd encounters a CRC error while processing an incremental map update, it will request a full map update from its peers. In this code path, an uninitialized variable was recently introduced and that will get de-referenced causing a crash.
The uninitialized variable was introduced in nautilus 14.2.10, and octopus 15.2.1.
[Test Case]
# Inject osd_inject_bad_map_crc_probability = 1
sudo ceph daemon osd.{id} config set osd_inject_bad_map_crc_probability 1
# Trigger some osd map updates by restarting a different osd
sudo systemctl restart osd@{diff-id}
[Regression Potential]
The code has been updated to leave handle_osd_maps() early if a CRC error is encountered, therefore preventing the commit if the failure is encountered while processing an incremental map update. This will make the full map update take longer but should prevent the crash that resulted in this bug. Additionally _committed_osd_maps() is now coded to abort if first <= last, but it is assumed that code should never be reached.
[Other Info]
Upstream has released a fix for this issue in Nautilus 14.2.11. The SRU for this point release is being tracked by LP: #1891077
Upstream has merged a fix for this issue in Octopus [1], but there is no current release target. The ceph packages in focal, groovy, and the ussuri cloud archive are exposed to this critical regression.
[0] https://tracker.ceph.com/issues/46443
[1] https://github.com/ceph/ceph/pull/36340 |
[Impact]
Upstream tracker: issue#46443 [0].
The ceph-osd service can crash when processing osd map updates.
When the osd encounters a CRC error while processing an incremental map update, it will request a full map update from its peers. In this code path, an uninitialized variable was recently introduced and that will get de-referenced causing a crash.
The uninitialized variable was introduced in nautilus 14.2.10, and octopus 15.2.1.
[Test Case]
# Inject osd_inject_bad_map_crc_probability = 1
sudo ceph daemon osd.{id} config set osd_inject_bad_map_crc_probability 1
# Trigger some osd map updates by restarting a different osd
sudo systemctl restart osd@{diff-id}
[Regression Potential]
The code has been updated to leave handle_osd_maps() early if a CRC error is encountered, therefore preventing the map commit if the failure is encountered while processing an incremental map update. This will make the full map update take longer but should prevent the crash that resulted in this bug. Additionally, _committed_osd_maps() is now coded to assert if first <= last, but it is assumed that code should never be reached.
[Other Info]
Upstream has released a fix for this issue in Nautilus 14.2.11. The SRU for this point release is being tracked by LP: #1891077
Upstream has merged a fix for this issue in Octopus [1], but there is no current release target. The ceph packages in focal, groovy, and the ussuri cloud archive are exposed to this critical regression.
[0] https://tracker.ceph.com/issues/46443
[1] https://github.com/ceph/ceph/pull/36340 |
|
| 2020-08-20 13:54:44 |
Robie Basak |
ceph (Ubuntu Focal): status |
Triaged |
Fix Committed |
|
| 2020-08-20 13:54:45 |
Robie Basak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2020-08-20 13:54:48 |
Robie Basak |
bug |
|
|
added subscriber SRU Verification |
| 2020-08-20 13:54:50 |
Robie Basak |
tags |
seg sts |
seg sts verification-needed verification-needed-focal |
|
| 2020-08-20 17:26:35 |
Corey Bryant |
cloud-archive/victoria: status |
Triaged |
Invalid |
|
| 2020-08-20 17:27:41 |
Corey Bryant |
cloud-archive/ussuri: status |
Triaged |
Fix Committed |
|
| 2020-08-20 17:27:42 |
Corey Bryant |
tags |
seg sts verification-needed verification-needed-focal |
seg sts verification-needed verification-needed-focal verification-ussuri-needed |
|
| 2020-08-28 08:09:46 |
Ponnuvel Palaniyappan |
tags |
seg sts verification-needed verification-needed-focal verification-ussuri-needed |
seg sts verification-needed verification-needed-focal verification-ussuri-done |
|
| 2020-09-01 20:08:43 |
Kellen Renshaw |
bug |
|
|
added subscriber Kellen Renshaw |
| 2020-09-10 08:14:23 |
Ponnuvel Palaniyappan |
tags |
seg sts verification-needed verification-needed-focal verification-ussuri-done |
seg sts verification-needed verification-needed-done verification-ussuri-done |
|
| 2020-09-10 14:57:32 |
Dan Hill |
tags |
seg sts verification-needed verification-needed-done verification-ussuri-done |
seg sts verification-done verification-done-focal verification-ussuri-done |
|
| 2020-09-15 15:26:51 |
Brian Murray |
tags |
seg sts verification-done verification-done-focal verification-ussuri-done |
seg sts verification-needed verification-needed-focal verification-ussuri-done |
|
| 2020-09-15 18:03:37 |
Ponnuvel Palaniyappan |
attachment added |
|
Package upgrade and test case execution steps & logs https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1891567/+attachment/5411119/+files/ceph15.2.3-focal-verification.txt |
|
| 2020-09-15 18:21:23 |
Ponnuvel Palaniyappan |
tags |
seg sts verification-needed verification-needed-focal verification-ussuri-done |
seg sts verification-done verification-done-focal verification-ussuri-done |
|
| 2020-09-15 18:27:50 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2020-09-15 18:37:53 |
Launchpad Janitor |
ceph (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2020-09-16 14:57:02 |
Corey Bryant |
cloud-archive/ussuri: status |
Fix Committed |
Fix Released |
|
