nova-consoleauth (2:12.0.2-0ubuntu1~cloud0) upgrade leaves nova-consoleauth stopped

Bug #1572320 reported by Jill Rouleau
26
This bug affects 5 people
Affects Status Importance Assigned to Milestone
nova (Ubuntu)
Confirmed
High
James Page

Bug Description

Latest set of upgrades leaves nova-consoleauth service stopped. Service must manually be restarted. Trusty/Liberty.

Start-Date: 2016-04-19 21:15:39
Upgrade: nova-novncproxy:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), nova-common:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), python-nova:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), nova-api-ec2:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), nova-api-os-compute:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), nova-consoleauth:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), nova-conductor:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), nova-scheduler:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), python-keystone:amd64 (8.0.1-0ubuntu1~cloud0, 8.1.0-0ubuntu1~cloud0), nova-cert:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), nova-objectstore:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0)
End-Date: 2016-04-19 21:16:05

Appears to be related to nova-consoleauth.log messages:

2016-04-19 21:15:44.688 119419 WARNING oslo_messaging.server [req-7686c9be-ab85-4c84-af4e-2df15f68f511 - - - - -] start/stop/wait must be called in the same thread
2016-04-19 21:15:44.690 119419 WARNING oslo_messaging.server [req-7686c9be-ab85-4c84-af4e-2df15f68f511 - - - - -] start/stop/wait must be called in the same thread

This is rolling through multiple landscape-managed clouds at the moment (probably over the next day) in case there is any other logging that would be useful.

Revision history for this message
Jill Rouleau (jillrouleau) wrote :

This is happening again with the latest set of updates:

Start-Date: 2016-05-17 22:11:48
Upgrade: libstdc++-4.8-dev:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), nova-novncproxy:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), libasan0:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), libquadmath0:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), gcc-4.8-base:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), cpp-4.8:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), nova-common:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), libgomp1:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), libarchive13:amd64 (3.1.2-7ubuntu2.1, 3.1.2-7ubuntu2.2), libtsan0:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), python-nova:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), neutron-common:amd64 (7.0.3-0ubuntu1~cloud0, 7.0.4-0ubuntu1~cloud0), libatomic1:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), nova-api-ec2:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), neutron-plugin-ml2:amd64 (7.0.3-0ubuntu1~cloud0, 7.0.4-0ubuntu1~cloud0), g++-4.8:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), binutils:amd64 (2.24-5ubuntu14, 2.24-5ubuntu14.1), python-neutron-fwaas:amd64 (7.0.0-0ubuntu1~cloud0, 7.0.2-0ubuntu1~cloud0), libgcc-4.8-dev:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), gcc-4.8:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), neutron-server:amd64 (7.0.3-0ubuntu1~cloud0, 7.0.4-0ubuntu1~cloud0), nova-api-os-compute:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), nova-consoleauth:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), nova-conductor:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), python-neutron:amd64 (7.0.3-0ubuntu1~cloud0, 7.0.4-0ubuntu1~cloud0), libgfortran3:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), nova-scheduler:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), nova-cert:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), nova-objectstore:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), libstdc++6:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), libitm1:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3)
End-Date: 2016-05-17 22:12:23

2016-05-17 22:11:58.561 1776154 WARNING oslo_messaging.server [req-61be1a66-e2a9-4f70-b3ff-172f89d72020 - - - - -] start/stop/wait must be called in the same thread
2016-05-17 22:11:58.563 1776154 WARNING oslo_messaging.server [req-61be1a66-e2a9-4f70-b3ff-172f89d72020 - - - - -] start/stop/wait must be called in the same thread

nova-consoleauth service is left in a 'stopped' state after update, this is an outage condition.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in nova (Ubuntu):
status: New → Confirmed
Revision history for this message
Brad Marshall (brad-marshall) wrote :

This just occured on another cloud, this time with nova-consoleauth 1:2015.1.4-0ubuntu2.

James Page (james-page)
Changed in nova (Ubuntu):
assignee: nobody → James Page (james-page)
importance: Undecided → High
Revision history for this message
James Page (james-page) wrote :

On the assumption that you are running multiple nova-cloud-controller service units, I need to understand which of the HA options you are using for nova-consoleauth, as I think its what the charm is doing in terms of managing the process via upstart that is causing the problem; specifically:

  single-nova-consoleauth
  console-access-protocol

would be useful configuration options to see please (I'm assuming that you are also using the hacluster charm for HA).

Changed in nova (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Jill Rouleau (jillrouleau) wrote :

    nova-cloud-controller:
      charm: nova-cloud-controller
      num_units: 3
      to: *NCC_3x_TO
      options:
        <<: *OS_REL_ORIGIN
        # config-flags will likely be overloaded by additional settings like
        # quota_RESOURCENAME=nnn,scheduler_default_filters=...
        config-flags: disk_allocation_ratio=1.0
        cpu-allocation-ratio: 4.0
        ram-allocation-ratio: 0.9
        network-manager: Neutron
        console-access-protocol: novnc
        quantum-security-groups: "yes"
        region: *REGION
        vip: *NCC_VIP
        vip_cidr: *VIP_CIDR
    ncc-hacluster:
      charm: hacluster
      options:
        cluster_count: 3

from juju get:

  single-nova-consoleauth
default: true
  console-access-protocol
novnc

All three of the clouds that hit this with the most recent update are running
lp:charms/trusty/nova-cloud-controller;revno=173
lp:charms/trusty/hacluster;revno=54

REV (axis77cuttingedge)
Changed in nova (Ubuntu):
status: Incomplete → New
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in nova (Ubuntu):
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.