neutron-lbaasv2-agent MessagingTimeout trying to contact HAPROXY plugin after pike to queens upgrade

Bug #1817390 reported by Drew Freiberger on 2019-02-23
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack neutron-gateway charm
Undecided
Unassigned

Bug Description

with 18.11 charms, upgrading from xenial-pike to xenial-queens, lbaasv2-agents failed to start and connect to api layer.

errors in neutron-lbaasv2-agent.log show:
2019-02-23 04:59:56.435 1396118 DEBUG neutron.common.rpc [-] Creating Consumer connection for Service n-lbaasv2_agent start /usr/lib/python2.7/dist-packages/neutron/common/rpc.py:293
2019-02-23 05:00:56.438 1396118 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:276
2019-02-23 05:00:56.439 1396118 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.001s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:288
2019-02-23 05:00:56.476 1396118 ERROR neutron.common.rpc [-] Timeout in RPC method get_ready_devices. Waiting for 54 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID c2238fbe1d9843cdb438ef94c6e78581
2019-02-23 05:00:56.476 1396118 WARNING neutron.common.rpc [-] Increasing timeout for get_ready_devices calls to 120 seconds. Restart the agent to restore it to the default value.: MessagingTimeout: Timed out waiting for a reply to message ID c2238fbe1d9843cdb438ef94c6e78581

enabling debug/verbose logging, can't tell where this connection should be going.

I see n-lbaasv2-plugin queue is full and not being consumed.

Also seeing possibly related:

2019-02-23 04:59:56.340 1396118 WARNING stevedore.named [req-5317e433-25e6-46c5-8066-f59ba94bf73a - - - - -] Could not load neutron_lbaas.drivers.haproxy.namespace_driver.HaproxyNSDriver

summary: - neutron-lbaasv2-agent MessagingTimeout
+ neutron-lbaasv2-agent MessagingTimeout trying to contact HAPROXY plugin
+ after pike to queens upgrade
Pete Vander Giessen (petevg) wrote :

@afreiberger is this problem still occurring in the latest charm release? If so, would you please attach bundles and crashdumps?

Changed in charm-neutron-gateway:
status: New → Incomplete
status: Incomplete → New
status: New → Incomplete
Drew Freiberger (afreiberger) wrote :

As an update, I just performed a similar upgrade using 19.04 latest charms and did not witness this issue.

Changed in charm-neutron-gateway:
status: Incomplete → Confirmed
status: Confirmed → In Progress
status: In Progress → New
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers