nova-consoleauth (2:12.0.2-0ubuntu1~cloud0) upgrade leaves nova-consoleauth stopped

[Jill Rouleau]

Latest set of upgrades leaves nova-consoleauth service stopped. Service must manually be restarted. Trusty/Liberty.

Start-Date: 2016-04-19 21:15:39
Upgrade: nova-novncproxy:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), nova-common:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), python-nova:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), nova-api-ec2:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), nova-api-os-compute:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), nova-consoleauth:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), nova-conductor:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), nova-scheduler:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), python-keystone:amd64 (8.0.1-0ubuntu1~cloud0, 8.1.0-0ubuntu1~cloud0), nova-cert:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0), nova-objectstore:amd64 (12.0.1-0ubuntu1~cloud0, 12.0.2-0ubuntu1~cloud0)
End-Date: 2016-04-19 21:16:05

Appears to be related to nova-consoleauth.log messages:

2016-04-19 21:15:44.688 119419 WARNING oslo_messaging.server [req-7686c9be-ab85-4c84-af4e-2df15f68f511 - - - - -] start/stop/wait must be called in the same thread
2016-04-19 21:15:44.690 119419 WARNING oslo_messaging.server [req-7686c9be-ab85-4c84-af4e-2df15f68f511 - - - - -] start/stop/wait must be called in the same thread

This is rolling through multiple landscape-managed clouds at the moment (probably over the next day) in case there is any other logging that would be useful.

[Jill Rouleau (jillrouleau)]

This is happening again with the latest set of updates:

Start-Date: 2016-05-17 22:11:48
Upgrade: libstdc++-4.8-dev:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), nova-novncproxy:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), libasan0:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), libquadmath0:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), gcc-4.8-base:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), cpp-4.8:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), nova-common:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), libgomp1:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), libarchive13:amd64 (3.1.2-7ubuntu2.1, 3.1.2-7ubuntu2.2), libtsan0:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), python-nova:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), neutron-common:amd64 (7.0.3-0ubuntu1~cloud0, 7.0.4-0ubuntu1~cloud0), libatomic1:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), nova-api-ec2:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), neutron-plugin-ml2:amd64 (7.0.3-0ubuntu1~cloud0, 7.0.4-0ubuntu1~cloud0), g++-4.8:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), binutils:amd64 (2.24-5ubuntu14, 2.24-5ubuntu14.1), python-neutron-fwaas:amd64 (7.0.0-0ubuntu1~cloud0, 7.0.2-0ubuntu1~cloud0), libgcc-4.8-dev:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), gcc-4.8:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), neutron-server:amd64 (7.0.3-0ubuntu1~cloud0, 7.0.4-0ubuntu1~cloud0), nova-api-os-compute:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), nova-consoleauth:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), nova-conductor:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), python-neutron:amd64 (7.0.3-0ubuntu1~cloud0, 7.0.4-0ubuntu1~cloud0), libgfortran3:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), nova-scheduler:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), nova-cert:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), nova-objectstore:amd64 (12.0.2-0ubuntu1~cloud0, 12.0.3-0ubuntu1~cloud0), libstdc++6:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3), libitm1:amd64 (4.8.4-2ubuntu1~14.04.1, 4.8.4-2ubuntu1~14.04.3)
End-Date: 2016-05-17 22:12:23

2016-05-17 22:11:58.561 1776154 WARNING oslo_messaging.server [req-61be1a66-e2a9-4f70-b3ff-172f89d72020 - - - - -] start/stop/wait must be called in the same thread
2016-05-17 22:11:58.563 1776154 WARNING oslo_messaging.server [req-61be1a66-e2a9-4f70-b3ff-172f89d72020 - - - - -] start/stop/wait must be called in the same thread

nova-consoleauth service is left in a 'stopped' state after update, this is an outage condition.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Brad Marshall (brad-marshall)]

This just occured on another cloud, this time with nova-consoleauth 1:2015.1.4-0ubuntu2.

[James Page (james-page)]

On the assumption that you are running multiple nova-cloud-controller service units, I need to understand which of the HA options you are using for nova-consoleauth, as I think its what the charm is doing in terms of managing the process via upstart that is causing the problem; specifically:

  single-nova-consoleauth
  console-access-protocol

would be useful configuration options to see please (I'm assuming that you are also using the hacluster charm for HA).

[Jill Rouleau (jillrouleau)]

    nova-cloud-controller:
      charm: nova-cloud-controller
      num_units: 3
      to: *NCC_3x_TO
      options:
        <<: *OS_REL_ORIGIN
        # config-flags will likely be overloaded by additional settings like
        # quota_RESOURCENAME=nnn,scheduler_default_filters=...
        config-flags: disk_allocation_ratio=1.0
        cpu-allocation-ratio: 4.0
        ram-allocation-ratio: 0.9
        network-manager: Neutron
        console-access-protocol: novnc
        quantum-security-groups: "yes"
        region: *REGION
        vip: *NCC_VIP
        vip_cidr: *VIP_CIDR
    ncc-hacluster:
      charm: hacluster
      options:
        cluster_count: 3

from juju get:

  single-nova-consoleauth
default: true
  console-access-protocol
novnc

All three of the clouds that hit this with the most recent update are running
lp:charms/trusty/nova-cloud-controller;revno=173
lp:charms/trusty/hacluster;revno=54

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.