[SRU] nova-* services do not start if rsyslog is not yet started

Bug #1459046 reported by Liang Chen on 2015-05-26
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Unassigned
Ubuntu Cloud Archive
High
Unassigned
Icehouse
High
Unassigned
Juno
High
Unassigned
Kilo
High
Unassigned
Liberty
Undecided
Unassigned
oslo.log
Undecided
Liang Chen
nova (Ubuntu)
Undecided
Unassigned
Trusty
High
Edward Hope-Morley
Utopic
High
Edward Hope-Morley
Vivid
Undecided
Unassigned
Wily
Undecided
Unassigned
python-oslo.log (Ubuntu)
High
Unassigned
Vivid
High
Unassigned
Wily
High
Unassigned

Bug Description

[Impact]

 * If Nova services are configured to log to syslog (use_syslog=True) they
   will currently fail with ECONNREFUSED if they cannot connect to syslog.
   This patch adds support for allowing nova to retry connecting a
   configurable number of times before print an error message and continuing
   with startup.

[Test Case]

 * Configure nova with use_syslog=True in nova.conf, stop rsyslog service and
   restart nova services. Check that upstart nova logs to see retries
   occurring then start rsyslog and observe connection succeed and
   nova-compute startup.

[Regression Potential]

 * None

Related branches

Liang Chen (cbjchen) on 2015-05-26
Changed in nova (Ubuntu):
assignee: nobody → Liang Chen (cbjchen)
status: New → In Progress
Liang Chen (cbjchen) on 2015-05-26
description: updated
Liang Chen (cbjchen) on 2015-05-27
description: updated
Liang Chen (cbjchen) on 2015-05-27
description: updated

The attachment "nova-2014.1.4-0ubuntu3-lp1459046.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Michael Terry (mterry) wrote :

Don't each of these start lines need a different WAITER value?

Liang Chen (cbjchen) wrote :

Michael,

Yeah, they do. Thanks a lot for pointing it out! I am going to have a thorough testing, and will update the patch soon.

Thanks,
Liang

Martin Pitt (pitti) wrote :

This is apparently an SRU fix for trusty. Does that also affect later releases? In particular, does it affect vivid/wily, as we don't use upstart there but systemd? In particular, these would need an After=syslog.service, is that the case? If not, this needs to be fixed as well, and as per the SRU rules a bug has to be fixed in the development series first. Thanks!

Changed in nova (Ubuntu Trusty):
status: New → In Progress
Liang Chen (cbjchen) on 2015-05-29
Changed in nova (Ubuntu Trusty):
assignee: nobody → Liang Chen (cbjchen)
Liang Chen (cbjchen) wrote :

Hi Martin,

Yeah. This an SRU fix for trusty. It doesn't affect later releases, as newer releases use openstack juno/kilo which doesn't have this problem.
That reminded me to try a backport instead of working on a upstart job patch. And it's indeed simpler and smaller. Thanks!

Thanks,
Liang

James Page (james-page) wrote :

Hi Liang

Two comment; firstly your cherry pick is not quite clean - it retains the existing syslog config code after the try/except wrapper introduced upstream.

Secondly I'm not sure that this really resolves the issue - if syslog is not started prior to a nova service, nova will just not use syslog for logging - which feels wrong to me.

I think a better fix might be to retry configuring syslog a few times - that way we don't have to update all upstart configurations, as the common code in the nova codebase will deal with the race.

James Page (james-page) wrote :

Removing sponsors for now - lets get the fix right and then go for sponsorship.

Liang Chen (cbjchen) wrote :

Hi James,

Sorry, I got messed up with quilt. I have resubmit the SRU at https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1348244, as it has the major fix that I backported in this SRU. And along with the backport, I also implemented the retry logic as you suggested. Thank you. I will mark this bug as invalid and continue the SRU in the original bug.

Thanks,
Liang

Changed in nova (Ubuntu):
status: In Progress → Invalid
Edward Hope-Morley (hopem) wrote :

The patch you have provided here for SRU (that is also now part of the SRU submission in bug 1348244) does not appear to have been submitted upstream. So, does this affect openstack versions greater than Icehouse? It looks like Nova logging was switched to use oslo.log as of Kilo so presumably Juno is also affected by this?

Assuming that Kilo+ is not affected and since this code has been moved out of tree anyway I think we should apply this to both I and J and make sure that patch commit message references this bug so that we have a context reference.

Edward Hope-Morley (hopem) wrote :

stable/icehouse patch uploaded to https://review.openstack.org/#/c/193105/

Changed in nova (Ubuntu):
status: Invalid → In Progress
Edward Hope-Morley (hopem) wrote :

stable/juno patch uploaded to https://review.openstack.org/#/c/193110/

Changed in nova:
status: New → In Progress
assignee: nobody → Edward Hope-Morley (hopem)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/193215

Kilo & master status:

Master works without any problem. oslo.log 1.2.0 will start writing to syslog as soon as rsyslogd is brought up.
Kilo uses oslo.log 1.1.0 which just ignores use_syslog setting when rsyslogd is not running or not started before nova-* services.

I have proposed the backport (https://review.openstack.org/#/c/193633/) to oslo.log stable/kilo from oslo.log master, and will get to the maintainer do a 1.0.1 release once merged.

Reviewed: https://review.openstack.org/193633
Committed: https://git.openstack.org/cgit/openstack/oslo.log/commit/?id=9a6c22663b57b7f2d9071767dadf28e59075a75e
Submitter: Jenkins
Branch: stable/kilo

commit 9a6c22663b57b7f2d9071767dadf28e59075a75e
Author: Julien Danjou <email address hidden>
Date: Mon Nov 24 18:35:36 2014 +0100

    Replace RFCSysLogHandler by a syslog() based one

    Rather than using the Python provided logging module that reimplents the
    whole syslog protocol and pointing it to /dev/log (which is not
    portable), we use the system calls from the syslog module, which are
    portable and default to the behaviour we actually want.

    Closes-Bug: 1385295
    Closes-Bug: 1391428

    Partial-Bug: 1459046

    (cherry picked from commit 33f5c6f94d94d5464fea38229d880f177b05aff7)

    Conflicts:
     oslo_log/log.py
     requirements.txt

    Change-Id: I2e47c896841cbd43d1f5504f2b8b897fc432cd0b

tags: added: in-stable-kilo

Change abandoned by Jeremy Stanley (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/193105
Reason: This branch has reached end of life and is being deleted.

Change abandoned by Jeremy Stanley (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/193209
Reason: This branch has reached end of life and is being deleted.

Change abandoned by Jeremy Stanley (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/193214
Reason: This branch has reached end of life and is being deleted.

Change abandoned by Jeremy Stanley (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/193215
Reason: This branch has reached end of life and is being deleted.

Change abandoned by Edward Hope-Morley (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/193110

summary: - nova-* services do not start if rsyslog is not yet started
+ [SRU] nova-* services do not start if rsyslog is not yet started
Changed in nova (Ubuntu Trusty):
assignee: Liang Chen (cbjchen) → Edward Hope-Morley (hopem)
Changed in nova (Ubuntu):
assignee: Liang Chen (cbjchen) → Edward Hope-Morley (hopem)
Changed in nova:
status: In Progress → New
assignee: Edward Hope-Morley (hopem) → nobody
Changed in nova (Ubuntu Utopic):
status: New → In Progress
assignee: nobody → Edward Hope-Morley (hopem)
Changed in nova (Ubuntu):
importance: Undecided → High
Changed in nova (Ubuntu Utopic):
importance: Undecided → High
Changed in nova (Ubuntu Trusty):
importance: Undecided → High
description: updated
description: updated
no longer affects: oslo.log
Changed in oslo.log:
status: New → Fix Committed
Liang Chen (cbjchen) on 2015-07-16
Changed in oslo.log:
assignee: nobody → Liang Chen (cbjchen)
James Page (james-page) wrote :

Confirmed as fixed in Wily (oslo.log 1.2.0)

Changed in nova (Ubuntu Utopic):
status: In Progress → Won't Fix
Changed in nova (Ubuntu):
status: In Progress → Invalid
Changed in python-oslo.log (Ubuntu Trusty):
status: New → Invalid
Changed in python-oslo.log (Ubuntu Utopic):
status: New → Invalid
Changed in python-oslo.log (Ubuntu):
status: New → Fix Released
Changed in nova (Ubuntu Vivid):
status: New → Invalid
Changed in python-oslo.log (Ubuntu Vivid):
status: New → Confirmed
James Page (james-page) wrote :

As Utopic goes EOL this month, marking Utopic task as 'Won't Fix' - we will target this directly to the Cloud Archive for Juno

James Page (james-page) wrote :

Raised bug tasks for python-oslo.log, as we have fix to land for vivid/kilo

Changed in python-oslo.log (Ubuntu Vivid):
importance: Undecided → High
Changed in python-oslo.log (Ubuntu Wily):
importance: Undecided → High
Changed in nova (Ubuntu Wily):
importance: High → Undecided
assignee: Edward Hope-Morley (hopem) → nobody

Hello Liang, or anyone else affected,

Accepted nova into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/1:2014.1.5-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in nova (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Edward Hope-Morley (hopem) wrote :

The package uploaded to proposed has failed to build and I see a lot of these messages in the buildlog:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/sphinx/ext/autodoc.py", line 335, in import_object
    __import__(self.modname)
  File "/«PKGBUILDDIR»/nova/tests/__init__.py", line 37, in <module>
    % os.environ.get('EVENTLET_NO_GREENDNS'))
ImportError: eventlet imported before nova/cmd/__init__ (env var set to None)

Edward Hope-Morley (hopem) wrote :

fwiw i just tried building this myself and it succeeded so not sure why the proposed pocket fails:

https://launchpad.net/~hopem/+archive/ubuntu/trusty-sru-testing-lp1459046/+build/7667621

tags: added: sts
Edward Hope-Morley (hopem) wrote :

Turns out the problem here a dodgy python-oslo.messaging in proposed from bug 1362863. This has now been removed so hopefully a re-spin will succeed.

Edward Hope-Morley (hopem) wrote :

New proposed build was successful. I have now deployed and tested that this work as expected.

tags: added: verification-done
removed: verification-needed
Changed in oslo.log:
milestone: none → 1.7.0
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 1:2014.1.5-0ubuntu1.2

---------------
nova (1:2014.1.5-0ubuntu1.2) trusty; urgency=medium

  * Add rsyslog retry support (LP: #1459046)
    - d/p/add-support-for-syslog-connect-retries.patch
  * Add vm clean shutdown support (LP: #1196924)
    - d/p/clean-shutdown.patch

 -- Edward Hope-Morley <email address hidden> Thu, 16 Jul 2015 11:55:57 +0100

Changed in nova (Ubuntu Trusty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Fixed in oslo.log (which Nova uses)

Changed in nova:
status: New → Fix Committed
Thierry Carrez (ttx) on 2015-09-03
Changed in nova:
milestone: none → liberty-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2015-10-15
Changed in nova:
milestone: liberty-3 → 12.0.0
Changed in python-oslo.log (Ubuntu Vivid):
status: Confirmed → Won't Fix
no longer affects: python-oslo.log (Ubuntu Trusty)
no longer affects: python-oslo.log (Ubuntu Utopic)
James Page (james-page) on 2016-09-08
Changed in cloud-archive:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers