[SRU] nova-* services do not start if rsyslog is not yet started

[Liang Chen]

[Impact]

 * If Nova services are configured to log to syslog (use_syslog=True) they
   will currently fail with ECONNREFUSED if they cannot connect to syslog.
   This patch adds support for allowing nova to retry connecting a
   configurable number of times before print an error message and continuing
   with startup.

[Test Case]

 * Configure nova with use_syslog=True in nova.conf, stop rsyslog service and
   restart nova services. Check that upstart nova logs to see retries
   occurring then start rsyslog and observe connection succeed and
   nova-compute startup.

[Regression Potential]

 * None

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "nova-2014.1.4-0ubuntu3-lp1459046.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Michael Terry (mterry)]

Don't each of these start lines need a different WAITER value?

[Liang Chen (cbjchen)]

Michael,

Yeah, they do. Thanks a lot for pointing it out! I am going to have a thorough testing, and will update the patch soon.

Thanks,
Liang

[Martin Pitt (pitti)]

This is apparently an SRU fix for trusty. Does that also affect later releases? In particular, does it affect vivid/wily, as we don't use upstart there but systemd? In particular, these would need an After=syslog.service, is that the case? If not, this needs to be fixed as well, and as per the SRU rules a bug has to be fixed in the development series first. Thanks!

[Liang Chen (cbjchen)]

Hi Martin,

Yeah. This an SRU fix for trusty. It doesn't affect later releases, as newer releases use openstack juno/kilo which doesn't have this problem.
That reminded me to try a backport instead of working on a upstart job patch. And it's indeed simpler and smaller. Thanks!

Thanks,
Liang

[James Page (james-page)]

Hi Liang

Two comment; firstly your cherry pick is not quite clean - it retains the existing syslog config code after the try/except wrapper introduced upstream.

Secondly I'm not sure that this really resolves the issue - if syslog is not started prior to a nova service, nova will just not use syslog for logging - which feels wrong to me.

I think a better fix might be to retry configuring syslog a few times - that way we don't have to update all upstart configurations, as the common code in the nova codebase will deal with the race.

[James Page (james-page)]

Removing sponsors for now - lets get the fix right and then go for sponsorship.

[Liang Chen (cbjchen)]

Hi James,

Sorry, I got messed up with quilt. I have resubmit the SRU at https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1348244, as it has the major fix that I backported in this SRU. And along with the backport, I also implemented the retry logic as you suggested. Thank you. I will mark this bug as invalid and continue the SRU in the original bug.

Thanks,
Liang

[Edward Hope-Morley (hopem)]

The patch you have provided here for SRU (that is also now part of the SRU submission in bug 1348244) does not appear to have been submitted upstream. So, does this affect openstack versions greater than Icehouse? It looks like Nova logging was switched to use oslo.log as of Kilo so presumably Juno is also affected by this?

Assuming that Kilo+ is not affected and since this code has been moved out of tree anyway I think we should apply this to both I and J and make sure that patch commit message references this bug so that we have a context reference.

[Edward Hope-Morley (hopem)]

stable/icehouse patch uploaded to https://review.openstack.org/#/c/193105/

[Edward Hope-Morley (hopem)]

stable/juno patch uploaded to https://review.openstack.org/#/c/193110/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/193214

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/193215

[Liang Chen (cbjchen)]

Kilo & master status:

Master works without any problem. oslo.log 1.2.0 will start writing to syslog as soon as rsyslogd is brought up.
Kilo uses oslo.log 1.1.0 which just ignores use_syslog setting when rsyslogd is not running or not started before nova-* services.

I have proposed the backport (https://review.openstack.org/#/c/193633/) to oslo.log stable/kilo from oslo.log master, and will get to the maintainer do a 1.0.1 release once merged.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/193633
Committed: https://git.openstack.org/cgit/openstack/oslo.log/commit/?id=9a6c22663b57b7f2d9071767dadf28e59075a75e
Submitter: Jenkins
Branch: stable/kilo

commit 9a6c22663b57b7f2d9071767dadf28e59075a75e
Author: Julien Danjou <email address hidden>
Date: Mon Nov 24 18:35:36 2014 +0100

    Replace RFCSysLogHandler by a syslog() based one

    Rather than using the Python provided logging module that reimplents the
    whole syslog protocol and pointing it to /dev/log (which is not
    portable), we use the system calls from the syslog module, which are
    portable and default to the behaviour we actually want.

    Closes-Bug: 1385295
    Closes-Bug: 1391428

    Partial-Bug: 1459046

    (cherry picked from commit 33f5c6f94d94d5464fea38229d880f177b05aff7)

    Conflicts:
     oslo_log/log.py
     requirements.txt

    Change-Id: I2e47c896841cbd43d1f5504f2b8b897fc432cd0b

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jeremy Stanley (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/193105
Reason: This branch has reached end of life and is being deleted.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jeremy Stanley (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/193209
Reason: This branch has reached end of life and is being deleted.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jeremy Stanley (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/193214
Reason: This branch has reached end of life and is being deleted.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jeremy Stanley (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/193215
Reason: This branch has reached end of life and is being deleted.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Edward Hope-Morley (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/193110

[James Page (james-page)]

Confirmed as fixed in Wily (oslo.log 1.2.0)

[James Page (james-page)]

As Utopic goes EOL this month, marking Utopic task as 'Won't Fix' - we will target this directly to the Cloud Archive for Juno

[James Page (james-page)]

Raised bug tasks for python-oslo.log, as we have fix to land for vivid/kilo

[Brian Murray (brian-murray)]

Hello Liang, or anyone else affected,

Accepted nova into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/1:2014.1.5-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Edward Hope-Morley (hopem)]

The package uploaded to proposed has failed to build and I see a lot of these messages in the buildlog:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/sphinx/ext/autodoc.py", line 335, in import_object
    __import__(self.modname)
  File "/«PKGBUILDDIR»/nova/tests/__init__.py", line 37, in <module>
    % os.environ.get('EVENTLET_NO_GREENDNS'))
ImportError: eventlet imported before nova/cmd/__init__ (env var set to None)

[Edward Hope-Morley (hopem)]

fwiw i just tried building this myself and it succeeded so not sure why the proposed pocket fails:

https://launchpad.net/~hopem/+archive/ubuntu/trusty-sru-testing-lp1459046/+build/7667621

[Edward Hope-Morley (hopem)]

Turns out the problem here a dodgy python-oslo.messaging in proposed from bug 1362863. This has now been removed so hopefully a re-spin will succeed.

[Edward Hope-Morley (hopem)]

New proposed build was successful. I have now deployed and tested that this work as expected.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 1:2014.1.5-0ubuntu1.2

---------------
nova (1:2014.1.5-0ubuntu1.2) trusty; urgency=medium

  * Add rsyslog retry support (LP: #1459046)
    - d/p/add-support-for-syslog-connect-retries.patch
  * Add vm clean shutdown support (LP: #1196924)
    - d/p/clean-shutdown.patch

 -- Edward Hope-Morley <email address hidden> Thu, 16 Jul 2015 11:55:57 +0100

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Davanum Srinivas (DIMS) (dims-v)]

Fixed in oslo.log (which Nova uses)