Brief Description
-----------------
After swact to controller-1, reapply stx-openstack without any charts change, the reapply will hang on "Generating application overrides" for 10+ minutes, with RPC reposnse timeout error in sysinv log.
- The application does eventually apply successfully.
- This issue does not happen on subsequent reapply from controller-1
- This issue does not happen when reapply after swacting to controller-0
Severity
--------
Minor
Steps to Reproduce
------------------
- install and config system, deploy stx-openstack application
- swact active controller to controller-1
- Run "system application-apply stx-openstack" from controller-1 after swact.
Expected Behavior
------------------
- Reapply succeeds very quickly (normally within 1 minute)
Actual Behavior
----------------
- Reapply gets stuck at generating application overrides for about 10 minutes, and eventually completes
Reproducibility
---------------
Reproducible
System Configuration
--------------------
Multi-node system
Branch/Pull Time/Commit
-----------------------
f/stein as of 2019-02-25
Timestamp/Logs
--------------
[2019-02-26 18:59:55,310] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-swact controller-0'
[2019-02-26 19:02:14,727] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-apply stx-openstack'
[2019-02-26 19:17:47,162] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
+---------------+-----------------+------------------------+----------+----------------------------------+
| application | manifest name | manifest file | status | progress |
+---------------+-----------------+------------------------+----------+----------------------------------+
| stx-openstack | armada-manifest | manifest-no-tests.yaml | applying | generating application overrides |
+---------------+-----------------+------------------------+----------+----------------------------------+
controller-1:~$
sysinv.log:
2019-02-26 19:02:16.477 178122 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-02-26 19:02:17.007 178122 INFO sysinv.helm.neutron [req-6f73c478-0811-4605-8b93-f44da59f9c6a admin admin] _get_neutron_ml2_config={'ml2': {'physical_network_mtus': 'group0-data0:1500'}}
2019-02-26 19:04:00.408 3710 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout.
2019-02-26 19:04:00.417 3710 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.2:5672
2019-02-26 19:05:00.478 3710 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "<unknown>"
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task Traceback (most recent call last):
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 182, in run_periodic_tasks
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task task(self, context)
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1035, in _agent_audit
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task force_updates=None)
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 315, in inner
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task return f(*args, **kwargs)
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1134, in agent_audit
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid)
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 234, in _update_ttys_dcd_status
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id)
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1174, in get_host_ttys_dcd
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task ihost_id=ihost_id))
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method'))
2019-02-26 19:05:00.478 3710 TRACE sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "<unknown>"
Marking as release gating; low priority given the operation complete and is only slow right after a swact operation.