| 2022-08-17 03:07:29 |
Dan Hill |
bug |
|
|
added bug |
| 2022-08-17 03:08:03 |
Dan Hill |
nominated for series |
|
Ubuntu Jammy |
|
| 2022-08-17 03:08:03 |
Dan Hill |
bug task added |
|
ceph (Ubuntu Jammy) |
|
| 2022-08-17 03:08:03 |
Dan Hill |
nominated for series |
|
Ubuntu Kinetic |
|
| 2022-08-17 03:08:03 |
Dan Hill |
bug task added |
|
ceph (Ubuntu Kinetic) |
|
| 2022-08-17 03:08:43 |
Dan Hill |
bug task added |
|
cloud-archive |
|
| 2022-08-17 03:09:13 |
Dan Hill |
nominated for series |
|
cloud-archive/yoga |
|
| 2022-08-17 03:09:13 |
Dan Hill |
bug task added |
|
cloud-archive/yoga |
|
| 2022-08-17 03:09:32 |
Dan Hill |
tags |
|
seg sts |
|
| 2022-08-17 14:47:25 |
Nicolas Bock |
bug |
|
|
added subscriber Nicolas Bock |
| 2022-08-17 15:57:12 |
Launchpad Janitor |
ceph (Ubuntu): status |
New |
Confirmed |
|
| 2022-08-17 15:57:12 |
Launchpad Janitor |
ceph (Ubuntu Jammy): status |
New |
Confirmed |
|
| 2022-08-17 17:19:08 |
Ponnuvel Palaniyappan |
bug |
|
|
added subscriber Ponnuvel Palaniyappan |
| 2022-08-17 19:03:59 |
Pedro Victor Lourenço Fragola |
bug |
|
|
added subscriber Pedro Victor Lourenço Fragola |
| 2022-08-17 19:52:38 |
Chris MacNaughton |
ceph (Ubuntu Jammy): assignee |
|
Chris MacNaughton (chris.macnaughton) |
|
| 2022-08-17 19:52:41 |
Chris MacNaughton |
ceph (Ubuntu Kinetic): assignee |
|
Chris MacNaughton (chris.macnaughton) |
|
| 2022-08-18 13:47:54 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~chris.macnaughton/ubuntu/+source/ceph/+git/ceph/+merge/428549 |
|
| 2022-08-18 13:48:21 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~chris.macnaughton/ubuntu/+source/ceph/+git/ceph/+merge/428550 |
|
| 2022-08-18 13:49:20 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~chris.macnaughton/ubuntu/+source/ceph/+git/ceph/+merge/428551 |
|
| 2022-08-18 13:53:34 |
Chris MacNaughton |
description |
Ubuntu packaging is configuring `osd_class_dir` with a relative path `CMAKE_INSTALL_LIBDIR` instead of the required absolute path `CMAKE_INSTALL_FULL_LIBDIR` [0].
The default value for `osd_class_dir` changed in Quincy, starting with v17.1.0 [1].
The ceph-osd service relies on the `osd_class_dir` path to find and load class libraries that extend RADOS [2]. When this is set incorrectly, RADOS clients fail with repeated "Operation not supported" errors:
```
2022-08-16T17:42:15.044+0000 7fe375685e40 0 rgw main: ERROR: failed reading data (obj=default.rgw.log:bucket.sync-target-hints.), r=-95
2022-08-16T17:42:15.048+0000 7fe375685e40 0 rgw main: ERROR: failed to read targets index for bucket=:[]) r=-95
2022-08-16T17:42:15.048+0000 7fe375685e40 0 rgw main: ERROR: failed to initialize bucket sync policy handler: get_bucket_sync_hints() on bucket=-- returned r=-95
2022-08-16T17:42:15.048+0000 7fe375685e40 -1 rgw main: ERROR: could not initialize zone policy handler for zone=default
2022-08-16T17:42:15.048+0000 7fe375685e40 0 rgw main: ERROR: failed to start notify service ((95) Operation not supported
2022-08-16T17:42:15.048+0000 7fe375685e40 0 rgw main: ERROR: failed to init services (ret=(95) Operation not supported)
```
The ceph-osd service will also report `_load_class` errors:
```
2022-08-16T19:05:55.562+0000 7f4770ff9700 0 _load_class could not stat class lib/x86_64-linux-gnu/rados-classes/libcls_rbd.so: (2) No such file or directory
```
Admins can resolve this issue by manually setting `osd_class_dir` to the correct value. Run the following command on a ceph-mon:
```
sudo ceph config set global osd_class_dir /usr/lib/x86_64-linux-gnu/rados-classes
```
Then restart all ceph-osd services to pick up the new `osd_class_dir` location.
[0] https://cmake.org/cmake/help/v3.24/module/GNUInstallDirs.html#result-variables
[1] https://github.com/ceph/ceph/commit/3bee4b02611459b9ae949cebf5967e4d83ef55de
[2] https://docs.ceph.com/en/latest/dev/osd-class-path/ |
[Impact]
A Ceph cluster that has upgraded from before 19.04 will break upon upgrade to Ceph Quincy (Jammy and Kinetic).
Ubuntu packaging is configuring `osd_class_dir` with a relative path `CMAKE_INSTALL_LIBDIR` instead of the required absolute path `CMAKE_INSTALL_FULL_LIBDIR` [0].
The default value for `osd_class_dir` changed in Quincy, starting with v17.1.0 [1].
The ceph-osd service relies on the `osd_class_dir` path to find and load class libraries that extend RADOS [2]. When this is set incorrectly, RADOS clients fail with repeated "Operation not supported" errors:
```
2022-08-16T17:42:15.044+0000 7fe375685e40 0 rgw main: ERROR: failed reading data (obj=default.rgw.log:bucket.sync-target-hints.), r=-95
2022-08-16T17:42:15.048+0000 7fe375685e40 0 rgw main: ERROR: failed to read targets index for bucket=:[]) r=-95
2022-08-16T17:42:15.048+0000 7fe375685e40 0 rgw main: ERROR: failed to initialize bucket sync policy handler: get_bucket_sync_hints() on bucket=-- returned r=-95
2022-08-16T17:42:15.048+0000 7fe375685e40 -1 rgw main: ERROR: could not initialize zone policy handler for zone=default
2022-08-16T17:42:15.048+0000 7fe375685e40 0 rgw main: ERROR: failed to start notify service ((95) Operation not supported
2022-08-16T17:42:15.048+0000 7fe375685e40 0 rgw main: ERROR: failed to init services (ret=(95) Operation not supported)
```
The ceph-osd service will also report `_load_class` errors:
```
2022-08-16T19:05:55.562+0000 7f4770ff9700 0 _load_class could not stat class lib/x86_64-linux-gnu/rados-classes/libcls_rbd.so: (2) No such file or directory
```
Admins can resolve this issue by manually setting `osd_class_dir` to the correct value. Run the following command on a ceph-mon:
```
sudo ceph config set global osd_class_dir /usr/lib/x86_64-linux-gnu/rados-classes
```
Then restart all ceph-osd services to pick up the new `osd_class_dir` location.
[0] https://cmake.org/cmake/help/v3.24/module/GNUInstallDirs.html#result-variables
[1] https://github.com/ceph/ceph/commit/3bee4b02611459b9ae949cebf5967e4d83ef55de
[2] https://docs.ceph.com/en/latest/dev/osd-class-path/
[Test Plan]
1. Install Ceph at Bionic
2. Upgrade through to Jammy
a. Confirm that client usage is broken
3. Upgrade to Jammy-proposed
a. Confirm that client usage works again
In addition to client activity, it can be confirmed that the OSDs don't have error logs about failing to load classes.
[Where problems could occur]
Problems could occur as a result of library paths changing, so Ceph functionality should be verified. This will be done with functional tests of Ceph using the Ceph Juju charms. |
|
| 2022-08-18 13:53:44 |
Chris MacNaughton |
merge proposal unlinked |
https://code.launchpad.net/~chris.macnaughton/ubuntu/+source/ceph/+git/ceph/+merge/428549 |
|
|
| 2022-08-18 13:55:43 |
Chris MacNaughton |
ceph (Ubuntu Jammy): importance |
Undecided |
Critical |
|
| 2022-08-18 13:55:44 |
Chris MacNaughton |
ceph (Ubuntu Kinetic): importance |
Undecided |
Critical |
|
| 2022-08-22 15:50:52 |
James Page |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2022-08-29 16:56:13 |
Thobias Trevisan |
bug |
|
|
added subscriber Thobias Trevisan |
| 2022-09-01 15:23:51 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~chris.macnaughton/ubuntu/+source/ceph/+git/ceph/+merge/429303 |
|
| 2022-09-01 15:29:06 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~chris.macnaughton/ubuntu/+source/ceph/+git/ceph/+merge/429304 |
|
| 2022-09-09 12:30:23 |
Timo Aaltonen |
ceph (Ubuntu Jammy): status |
Confirmed |
Fix Committed |
|
| 2022-09-09 12:30:27 |
Timo Aaltonen |
bug |
|
|
added subscriber SRU Verification |
| 2022-09-09 12:30:32 |
Timo Aaltonen |
tags |
seg sts |
seg sts verification-needed verification-needed-jammy |
|
| 2022-09-14 19:26:51 |
Chris MacNaughton |
cloud-archive: status |
New |
Fix Committed |
|
| 2022-09-14 19:26:53 |
Chris MacNaughton |
cloud-archive/yoga: status |
New |
Fix Committed |
|
| 2022-09-20 00:33:27 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~lmlogiudice/ubuntu/+source/ceph/+git/ceph/+merge/430116 |
|
| 2022-09-20 01:07:23 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~lmlogiudice/ubuntu/+source/ceph/+git/ceph/+merge/430118 |
|
| 2022-09-24 18:01:02 |
Launchpad Janitor |
ceph (Ubuntu Kinetic): status |
Confirmed |
Fix Released |
|
| 2022-11-01 10:50:25 |
Pedro Victor Lourenço Fragola |
tags |
seg sts verification-needed verification-needed-jammy |
seg sts verification-done verification-done-jammy |
|
| 2022-11-16 01:27:05 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2022-11-16 01:36:45 |
Launchpad Janitor |
ceph (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2022-11-22 16:08:49 |
Chris MacNaughton |
cloud-archive: status |
Fix Committed |
Fix Released |
|
| 2022-11-22 16:08:52 |
Chris MacNaughton |
cloud-archive/yoga: status |
Fix Committed |
Fix Released |
|
| 2023-01-24 10:05:50 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~peter-sabaini/ubuntu/+source/ceph/+git/ceph/+merge/436235 |
|
