Ceph 12.2.12 restarts services during upgrade

[Kellen Renshaw]

[Impact]

Upgrading from ceph on Ubuntu 18.04 causes the ceph-osd services to be restarted without prompting.

This appears to be in the configure section on the ceph-common,postinst:
# Automatically added by dh_systemd_start/11.1.6ubuntu2
if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then
        if [ -d /run/systemd/system ]; then
                systemctl --system daemon-reload >/dev/null || true
                if [ -n "$2" ]; then
                        _dh_action=restart
                else
                        _dh_action=start
                fi
                deb-systemd-invoke $_dh_action 'ceph.target' >/dev/null || true
        fi
fi
# End automatically added section

This is a change of behaviour compare to debhelper 9.
--restart-after-upgrade is the default in compat 10.

The expected behavior during a package upgrade is to leave all the ceph service states unmodified. They should not be enabled/disabled or stopped/started.

[Test Case]
* Check pid of ceph-osd (before upgrade)
* sudo apt-get update
* sudo apt-get upgrade
* Check pid of ceph-osd (after upgrade)

If the pid changes, it means ceph-osd has been restarted, and the restart or stop/start has been exercised.

[Potential Regression]

Low, the desire is to control the precise timing of the restart.

The larger context is that there may multiple updates (in this case kernel) and one want a single stop/start when the reboot occurred, not 2 (1 due to ceph postinst and 1 for the reboot (to load new kernel) for instance).

The goal is for the admin to choose when (and which) ceph services move to the new code level, after upgrading the package.

[Other Information]

Ubuntu ceph package uses a more modern version of dh (10) as oppose to ceph upstream which uses dh 9[0]. Ceph package also doesn't exclude any ceph services like Ubuntu does[1]

Upstream ceph doesn't have the restart behaviour because they are not there yet (dh 10),but they are likely to take the same road as we do as they upgrade debhelper.

[0] https://github.com/ceph/ceph/blob/master/debian/control#L15
[1] https://github.com/ceph/ceph/blob/master/debian/rules#L86

[Eric Desrochers (slashd)]

# ceph src package : debian/rules
135 dh_systemd_start --no-restart-on-upgrade -Xceph-mon.service -Xceph-osd.service -X ceph-mds.service

It has the --no-restart-on-upgrade flag which is responsible for not stopping the service on upgrade. But there is an exclusion for : ceph-mon, ceph-osd and ceph-mds, meaning they are subject to restart with that exclusion (-X).

What is the expected behaviour here for mon, osd and mds ?

- Eric

[Eric Desrochers (slashd)]

http://manpages.ubuntu.com/manpages/bionic/man1/dh_systemd_start.1.html

       -r, --no-stop-on-upgrade, --no-restart-on-upgrade
           Do not stop service on upgrade.

http://manpages.ubuntu.com/manpages/bionic/en/man7/debhelper.7.html

       -Xitem, --exclude=item
           Exclude an item from processing. This option may be used multiple times, to exclude
           more than one thing. The \fIitem\fR is typically part of a filename, and any file
           containing the specified text will be excluded.

[Dan Hill (hillpd)]

The expected behavior during a package upgrade is to leave all the ceph service states unmodified. They should not be enabled/disabled or stopped/started.

[Eric Desrochers (slashd)]

Then removing the units from the exclusion shoud be suffisant at line 135 in debian/rules.

[Eric Desrochers (slashd)]

Another thing I'm not sure about how it should react this way but worth to explore:

     77 override_dh_installinit:
    ....
    131 # Ensure Debian/Ubuntu specific systemd units are NOT automatically enabled and started
    132 # Enable systemd targets only
    133 dh_systemd_enable -Xceph-mon.service -Xceph-osd.service -X ceph-mds.service
    134 # Start systemd targets only
    135 dh_systemd_start --no-restart-on-upgrade -Xceph-mon.service -Xceph-osd.service -X ceph-mds.service
    136
    137 override_dh_systemd_enable:
    138 # systemd enable done as part of dh_installinit
    139
    140 override_dh_systemd_start:
    141 # systemd start done as part of dh_installinit

Documentation:
Note that dh_systemd_start should be run after dh_installinit so that it can detect
corresponding SysV init scripts. The default sequence in dh does the right thing, this
note is only relevant when you are calling dh_systemd_start manually.

I would first try with the following to place the dh_systemd_[enable|start] execution where it should belong.

     77 override_dh_installinit:
    ....
- 131 # Ensure Debian/Ubuntu specific systemd units are NOT automatically enabled and started
- 132 # Enable systemd targets only
- 133 dh_systemd_enable -Xceph-mon.service -Xceph-osd.service -X ceph-mds.service
- 134 # Start systemd targets only
- 135 dh_systemd_start --no-restart-on-upgrade
    136
    137 override_dh_systemd_enable:
+ # Ensure Debian/Ubuntu specific systemd units are NOT automatically enabled and started
- 138 # systemd enable done as part of dh_installinit
+ 138 dh_systemd_enable -Xceph-mon.service -Xceph-osd.service -X ceph-mds.service
    139
    140 override_dh_systemd_start:
+ # Enable systemd targets only
- 141 # systemd start done as part of dh_installinit
+ 141 dh_systemd_start --no-restart-on-upgrade

That's what I would do first and give that a try.

- Eric

[Eric Desrochers (slashd)]

Another thing you could do as a debug exercise to better understand what is going on is to modify the maintainer scripts in question by adding the following:

- set -e
+ set -exv

-v Print shell input lines as they are read.
-x Print commands and their arguments as they are executed.

and then build a test package(instrumented) from it. It might be useful to see exactly what is happening behind the curtains.

Hope these few tips help !

- Eric

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Eric Desrochers (slashd)]

Upstream ceph, does pretty much the same thing as I explained above ^:
https://github.com/ceph/ceph/blob/master/debian/rules#L81-L92

They don't do any exclusion for osd, mds and mon like the the ceph source package in Ubuntu.

I'll start testing.

[Eric Desrochers (slashd)]

By default compat 10

--restart-after-upgrade seems to be the default in compat 10.
Which explain why the behaviour observed are different between upstream (dh 9) & Ubuntu (dh 10)
(I build the Ubuntu ceph package with dh 9, and there is no trace of restart action)

It's definitely a change of behaviour cause with the introduction of dh 10 in the Ceph package.

* dh_systemd_start should be run after dh_installinit, as it is right now it is running inside dh_installinit.

* --no-restart-on-upgrade still works, but deprecated, to avoid confusion a new alias has been created: --no-stop-on-upgrade

* I think the source package may need both : --no-stop-on-upgrade --no-restart-after-upgrade in order to hopefully stop restarting.

This is where I am at the moment.

- Eric

[Eric Desrochers (slashd)]

Hopefully that should do the trick:

override_dh_installinit:
.....
- # Ensure Debian/Ubuntu specific systemd units are NOT automatically enabled and started
- # Enable systemd targets only
- dh_systemd_enable -Xceph-mon.service -Xceph-osd.service -X ceph-mds.service
- # Start systemd targets only
- dh_systemd_start --no-restart-on-upgrade

override_dh_systemd_enable:
- # systemd start done as part of dh_installinit
+ dh_systemd_enable -Xceph-mon.service -Xceph-osd.service -X ceph-mds.service

override_dh_systemd_start:
- # systemd start done as part of dh_installinit
+ dh_systemd_start --no-stop-on-upgrade --no-restart-after-upgrade

[Eric Desrochers (slashd)]

# Before the upgrade : 12.2.12-0ubuntu0.18.04.1
ceph 15176 1 0 17:57 ? 00:00:59 /usr/bin/ceph-osd -f ....

# After the upgrade: 12.2.12-0ubuntu0.18.04.2 # ==> It's a test pkg (not in the archive yet)
ceph 15176 1 0 17:57 ? 00:01:00 /usr/bin/ceph-osd -f .....

I'll submit the MP to Cloud team for approval by end of week.

- Eric

[James Page (james-page)]

Setting as High across the board - if this is important in bionic, then its important for disco and eoan as well as they feed the UCA for bionic.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 14.2.2-0ubuntu2

---------------
ceph (14.2.2-0ubuntu2) eoan; urgency=medium

  [ Eric Desrochers ]
  * Ensure that daemons are not automatically restarted during package
    upgrades (LP: #1840347):
    - d/rules: Use "--no-restart-after-upgrade" and "--no-stop-on-upgrade"
      instead of "--no-restart-on-upgrade".
    - d/rules: Drop exclusion for ceph-[osd,mon,mds] for restarts.

  [ Jesse Williamson ]
  * d/p/civetweb-755-1.8-somaxconn-configurable*.patch: Backport changes
    to civetweb to allow tuning of SOMAXCONN in Ceph RADOS Gateway
    deployments (LP: #1838109).

  [ James Page ]
  * d/p/ceph-volume-wait-for-lvs.patch: Cherry pick inflight fix to
    ensure that required wal and db devices are present before
    activating OSD's (LP: #1828617).

  [ Steve Beattie ]
  * SECURITY UPDATE: RADOS gateway remote denial of service
    - d/p/CVE-2019-10222.patch: rgw: asio: check the remote endpoint
      before processing requests.
    - CVE-2019-10222

 -- James Page <email address hidden> Thu, 29 Aug 2019 13:54:25 +0100

[Mark Schouten (mark-prevented)]

root@osdnode01:~# apt-cache policy ceph-common
ceph-common:
  Installed: 12.2.11-1xenial
  Candidate: 12.2.12-0ubuntu0.18.04.2
  Version table:
     12.2.12-0ubuntu0.18.04.2 500
        500 http://nl.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages

Isn't this fixed now in bionic as well?

[Eric Desrochers (slashd)]

@Mark Schouten (mark-prevented),

The fix is waiting for SRU approval in the bionic upload queue, then it will start building in bionic-proposed for the testing phase. At this point Bionic status will be change to "Fix Committed", and this will mark the beginning of the testing phase.

- Eric

[Łukasz Zemczak (sil2100)]

Hello Kellen, or anyone else affected,

Accepted ceph into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/13.2.6-0ubuntu0.19.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Kellen, or anyone else affected,

Accepted ceph into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/12.2.12-0ubuntu0.18.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Eric Desrochers (slashd)]

[VERIFICATION BIONIC]

* Before the upgrade:
#ps
ceph 14495 1 0 14:40 ? 00:00:32 /usr/bin/ceph-osd -f --cluster ceph --id 3 --setuser ceph --setgroup ceph

# pidof ceph-osd
14495

#dpkg
ii ceph 12.2.12-0ubuntu0.18.04.2 amd64 distributed storage and file system
ii ceph-base 12.2.12-0ubuntu0.18.04.2 amd64 common ceph daemon libraries and management tools
ii ceph-common 12.2.12-0ubuntu0.18.04.2 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-mgr 12.2.12-0ubuntu0.18.04.2 amd64 manager for the ceph distributed file system
ii ceph-mon 12.2.12-0ubuntu0.18.04.2 amd64 monitor server for the ceph storage system
ii ceph-osd 12.2.12-0ubuntu0.18.04.2 amd64 OSD server for the ceph storage system
ii libcephfs2 12.2.12-0ubuntu0.18.04.2 amd64 Ceph distributed file system client library
ii python-cephfs 12.2.12-0ubuntu0.18.04.2 amd64 Python libraries for the Ceph libcephfs library
ii python-rados 12.2.12-0ubuntu0.18.04.2 amd64 Python libraries for the Ceph librados library
ii python-rbd 12.2.12-0ubuntu0.18.04.2 amd64 Python libraries for the Ceph librbd library

* After the upgrade to the proposed package:

#ps
ceph 14495 1 0 14:40 ? 00:00:32 /usr/bin/ceph-osd -f --cluster ceph --id 3 --setuser ceph --setgroup ceph

# pidof ceph-osd
14495

# dpkg -l | grep -i ceph
ii ceph 12.2.12-0ubuntu0.18.04.3 amd64 distributed storage and file system
ii ceph-base 12.2.12-0ubuntu0.18.04.3 amd64 common ceph daemon libraries and management tools
ii ceph-common 12.2.12-0ubuntu0.18.04.3 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-mgr 12.2.12-0ubuntu0.18.04.3 amd64 manager for the ceph distributed file system
ii ceph-mon 12.2.12-0ubuntu0.18.04.3 amd64 monitor server for the ceph storage system
ii ceph-osd 12.2.12-0ubuntu0.18.04.3 amd64 OSD server for the ceph storage system
ii libcephfs2 12.2.12-0ubuntu0.18.04.3 amd64 Ceph distributed file system client library
ii python-cephfs 12.2.12-0ubuntu0.18.04.3 amd64 Python libraries for the Ceph libcephfs library
ii python-rados 12.2.12-0ubuntu0.18.04.3 amd64 Python libraries for the Ceph librados library
ii python-rbd 12.2.12-0ubuntu0.18.04.3 amd64 Python libraries for th...

[Eric Desrochers (slashd)]

[VERIFICATION DISCO]

* Before upgrade [13.2.6-0ubuntu0.19.04.3]

#pidof ceph-osd
26937

#ps
ceph 26937 1 0 18:10 ? 00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph

* After upgrade [13.2.6-0ubuntu0.19.04.4

#pidof ceph-osd
26937

#ps
ceph 26937 1 0 18:10 ? 00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph

This is the expected behaviour, osd didn't restart ON nor AFTER upgrade, which now leave the decision to the sysadmin of the ceph cluster to decide when the daemons restart will happen via their maintenance window policy.

[James Page (james-page)]

Hello Kellen, or anyone else affected,

Accepted ceph into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[James Page (james-page)]

Hello Kellen, or anyone else affected,

Accepted ceph into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 13.2.6-0ubuntu0.19.04.4

---------------
ceph (13.2.6-0ubuntu0.19.04.4) disco; urgency=medium

  [ Eric Desrochers ]
  * Ensure that daemons are not automatically restarted during package
    upgrades (LP: #1840347):
    - d/rules: Use "--no-restart-after-upgrade" and "--no-stop-on-upgrade"
      instead of "--no-restart-on-upgrade".
    - d/rules: Drop exclusion for ceph-[osd,mon,mds] for restarts.

  [ James Page ]
  * d/p/ceph-volume-wait-for-lvs.patch: Cherry pick inflight fix to
    ensure that required wal and db devices are present before
    activating OSD's (LP: #1828617).

  [ Jesse Williamson ]
  * d/p/civetweb-755-1.8-somaxconn-configurable*.patch: Backport changes
    to civetweb to allow tuning of SOMAXCONN in Ceph RADOS Gateway
    deployments (LP: #1838109).

 -- James Page <email address hidden> Fri, 30 Aug 2019 10:10:04 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 12.2.12-0ubuntu0.18.04.3

---------------
ceph (12.2.12-0ubuntu0.18.04.3) bionic; urgency=medium

  [ James Page ]
  * d/p/ceph-volume-wait-for-lvs.patch: Cherry pick inflight fix to
    ensure that required wal and db devices are present before
    activating OSD's (LP: #1828617).

  [ Jesse Williamson ]
  * d/p/civetweb-755-1.8-somaxconn-configurable*.patch: Backport changes
    to civetweb to allow tuning of SOMAXCONN in Ceph RADOS Gateway
    deployments (LP: #1838109).

  [ James Page ]
  * d/p/rgw-gc-use-aio.patch: Cherry pick fix to switch to using AIO for
    garbage collection of objects in the Ceph RADOS Gateway
    (LP: #1838858).

  [ Eric Desrochers ]
  * Ensure that daemons are not automatically restarted during package
    upgrades (LP: #1840347):
    - d/rules: Use "--no-restart-after-upgrade" and "--no-stop-on-upgrade"
      instead of "--no-restart-on-upgrade".
    - d/rules: Drop exclusion for ceph-[osd,mon,mds] for restarts.

 -- James Page <email address hidden> Fri, 30 Aug 2019 10:11:09 +0100

[James Page (james-page)]

Hello Kellen, or anyone else affected,

Accepted ceph into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[James Page (james-page)]

This bug is still pending verification for UCA pockets.

[James Page (james-page)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package ceph - 13.2.6-0ubuntu0.19.04.4~cloud0
---------------

 ceph (13.2.6-0ubuntu0.19.04.4~cloud0) bionic-stein; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceph (13.2.6-0ubuntu0.19.04.4) disco; urgency=medium
 .
   [ Eric Desrochers ]
   * Ensure that daemons are not automatically restarted during package
     upgrades (LP: #1840347):
     - d/rules: Use "--no-restart-after-upgrade" and "--no-stop-on-upgrade"
       instead of "--no-restart-on-upgrade".
     - d/rules: Drop exclusion for ceph-[osd,mon,mds] for restarts.
 .
   [ James Page ]
   * d/p/ceph-volume-wait-for-lvs.patch: Cherry pick inflight fix to
     ensure that required wal and db devices are present before
     activating OSD's (LP: #1828617).
 .
   [ Jesse Williamson ]
   * d/p/civetweb-755-1.8-somaxconn-configurable*.patch: Backport changes
     to civetweb to allow tuning of SOMAXCONN in Ceph RADOS Gateway
     deployments (LP: #1838109).

[James Page (james-page)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package ceph - 13.2.6-0ubuntu0.18.10.3~cloud0
---------------

 ceph (13.2.6-0ubuntu0.18.10.3~cloud0) bionic; urgency=medium
 .
   [ Eric Desrochers ]
   * Ensure that daemons are not automatically restarted during package
     upgrades (LP: #1840347):
     - d/rules: Use "--no-restart-after-upgrade" and "--no-stop-on-upgrade"
       instead of "--no-restart-on-upgrade".
     - d/rules: Drop exclusion for ceph-[osd,mon,mds] for restarts.
 .
   [ James Page ]
   * d/p/ceph-volume-wait-for-lvs.patch: Cherry pick inflight fix to
     ensure that required wal and db devices are present before
     activating OSD's (LP: #1828617).
 .
   [ Jesse Williamson ]
   * d/p/civetweb-755-1.8-somaxconn-configurable*.patch: Backport changes
     to civetweb to allow tuning of SOMAXCONN in Ceph RADOS Gateway
     deployments (LP: #1838109).

[Eric Desrochers (slashd)]

[VERIFICATION QUEENS]

* Before upgrade

#pidof ceph-osd

22291

* After upgrade

#pidof ceph-osd
26937

This is the expected behaviour, osd didn't restart ON nor AFTER upgrade, which now leave the decision to the sysadmin of the ceph cluster to decide when the daemons restart will happen via their maintenance window policy.

[James Page (james-page)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package ceph - 12.2.12-0ubuntu0.18.04.3~cloud0
---------------

 ceph (12.2.12-0ubuntu0.18.04.3~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceph (12.2.12-0ubuntu0.18.04.3) bionic; urgency=medium
 .
   [ James Page ]
   * d/p/ceph-volume-wait-for-lvs.patch: Cherry pick inflight fix to
     ensure that required wal and db devices are present before
     activating OSD's (LP: #1828617).
 .
   [ Jesse Williamson ]
   * d/p/civetweb-755-1.8-somaxconn-configurable*.patch: Backport changes
     to civetweb to allow tuning of SOMAXCONN in Ceph RADOS Gateway
     deployments (LP: #1838109).
 .
   [ James Page ]
   * d/p/rgw-gc-use-aio.patch: Cherry pick fix to switch to using AIO for
     garbage collection of objects in the Ceph RADOS Gateway
     (LP: #1838858).
 .
   [ Eric Desrochers ]
   * Ensure that daemons are not automatically restarted during package
     upgrades (LP: #1840347):
     - d/rules: Use "--no-restart-after-upgrade" and "--no-stop-on-upgrade"
       instead of "--no-restart-on-upgrade".
     - d/rules: Drop exclusion for ceph-[osd,mon,mds] for restarts.