neutron-api stays blocked: Services not running that should be: neutron-server

[Bas de Bruijne]

In testrun:
https://solutions.qa.canonical.com/testruns/testRun/ad989e74-d085-40bb-b79a-c39f51180e2a

with FCE console:
https://oil-jenkins.canonical.com/job/fce_build/1270//console

2 out of 3 neutron-api using stay blocked:
------------------------------------------------
neutron-api/0* active idle 0/lxd/6 10.246.166.106 9696/tcp Unit is ready
  filebeat/21 active idle 10.246.166.106 Filebeat ready.
  hacluster-neutron-api/0* active idle 10.246.166.106 Unit is ready and clustered
  landscape-client/21 maintenance idle 10.246.166.106 Need computer-title and juju-info to proceed
  logrotated/15 active idle 10.246.166.106 Unit is ready.
  neutron-api-mysql-router/0* active idle 10.246.166.106 Unit is ready
  nrpe/21 active idle 10.246.166.106 icmp,5666/tcp Ready (source version/commit cs-nrpe-...)
  public-policy-routing/3 active idle 10.246.166.106 Unit is ready
  telegraf/21 active idle 10.246.166.106 9103/tcp Monitoring neutron-api/0 (source version/commit 26e531a)
neutron-api/1 blocked idle 1/lxd/7 10.246.167.64 9696/tcp Services not running that should be: neutron-server
  filebeat/31 active idle 10.246.167.64 Filebeat ready.
  hacluster-neutron-api/1 active idle 10.246.167.64 Unit is ready and clustered
  landscape-client/35 maintenance idle 10.246.167.64 Need computer-title and juju-info to proceed
  logrotated/23 active idle 10.246.167.64 Unit is ready.
  neutron-api-mysql-router/1 active idle 10.246.167.64 Unit is ready
  nrpe/31 active idle 10.246.167.64 icmp,5666/tcp Ready (source version/commit cs-nrpe-...)
  public-policy-routing/6 active idle 10.246.167.64 Unit is ready
  telegraf/35 active idle 10.246.167.64 9103/tcp Monitoring neutron-api/1 (source version/commit 26e531a)
neutron-api/2 blocked idle 2/lxd/7 10.246.165.62 9696/tcp Services not running that should be: neutron-server
  filebeat/33 active idle 10.246.165.62 Filebeat ready.
  hacluster-neutron-api/2 active idle 10.246.165.62 Unit is ready and clustered
  landscape-client/33 maintenance idle 10.246.165.62 Need computer-title and juju-info to proceed
  logrotated/27 active idle 10.246.165.62 Unit is ready.
  neutron-api-mysql-router/2 active idle 10.246.165.62 Unit is ready
  nrpe/33 active idle 10.246.165.62 icmp,5666/tcp Ready (source version/commit cs-nrpe-...)
  public-policy-routing/9 active idle 10.246.165.62 Unit is ready
  telegraf/31 active idle 10.246.165.62 9103/tcp Monitoring neutron-api/2 (source version/commit 26e531a)
------------------------------------------------

The only thing I can see standing out in the logs is:
------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect
    return fn()
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 303, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 760, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 492, in checkout
    rec = pool._do_get()
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
    self._dec_overflow()
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
    return self._create_connection()
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 437, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 639, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 482, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib/python3/dist-packages/pymysql/__init__.py", line 94, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 325, in __init__
    self.connect()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 599, in connect
    self._request_authentication()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 871, in _request_authentication
    auth_packet = self._process_auth(plugin_name, auth_packet)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 942, in _process_auth
    pkt = self._read_packet()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 684, in _read_packet
    packet.check_error()
  File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python3/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1045, "Access denied for user 'neutron'@'10.246.170.69' (using password: YES)")
------------------------------------------------

This message is from about 3h before the end of the testrun, but it is possible that the units are not recovering from it.

Crashdumps can be found here:
https://oil-jenkins.canonical.com/artifacts/ad989e74-d085-40bb-b79a-c39f51180e2a/index.html

[Bas de Bruijne (basdbruijne)]

This is still an active bug, subscribing Abdullah for follow up questions

[Alex Kavanagh (ajkavanagh)]

As per usual, please add versions of relevant software to the bug.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Neutron API Charm because there has been no activity for 60 days.]