service did not come up on NovaLink restart, exception fetching license status for vios

Bug #1782192 reported by Matthew Edmonds
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
networking-powervm
New
Undecided
Unassigned

Bug Description

Several hours after restarting a NovaLink partition, it was noticed that the neutron api logs (on the separate management node) were not able to communicate with the neutron agent on one of the compute nodes:

Jul 17 11:09:11 ip9-114-251-95 neutron-server[32460]: WARNING neutron.plugins.ml2.drivers.mech_agent [None req-de80026a-1666-4334-98af-a2ae86a04750 service neutron] Refusing to bind port d8d6a33f-3783-49e6-bb0d-3bba704fb141 to dead agent: {'binary': u'networking-powervm-sharedethernet-agent', 'description': None, 'availability_zone': None, 'heartbeat_timestamp': datetime.datetime(2018, 7, 13, 20, 46, 18), 'admin_state_up': True, 'alive': False, 'topic': u'N/A', 'host': u'neo21', 'agent_type': u'PowerVM Shared Ethernet agent', 'resource_versions': {}, 'created_at': datetime.datetime(2018, 7, 11, 14, 2, 32), 'started_at': datetime.datetime(2018, 7, 11, 14, 2, 32), 'id': '9e6f3f28-e36d-43a5-b1a5-d2548be4411f', 'configurations': {u'bridge_mappings': {u'default': u'c37d98e8-ccc3-3316-bddc-102189356e6a'}, u'devices': 0}}

Logging into that NovaLink compute node, systemctl showed <email address hidden> as enabled, but journalctl didn't have any logs. I found the following in /var/log/messages with a timestamp that would have been just a few minutes after the NovaLink was restarted.

Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00mTraceback (most recent call last):
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/usr/local/bin/networking-powervm-sea-agent", line 10, in <module>
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m sys.exit(main())
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/opt/stack/networking-powervm/networking_powervm/plugins/ibm/agent/powervm/sea_agent.py", line 286, in main
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m agent = SharedEthernetNeutronAgent()
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/opt/stack/networking-powervm/networking_powervm/plugins/ibm/agent/powervm/agent_base.py", line 265, in __init__
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m self.br_map = self.parse_bridge_mappings()
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/opt/stack/networking-powervm/networking_powervm/plugins/ibm/agent/powervm/sea_agent.py", line 97, in parse_bridge_mappings
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m ACONF.bridge_mappings)
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/opt/stack/networking-powervm/networking_powervm/plugins/ibm/agent/powervm/utils.py", line 63, in parse_sea_mappings
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m nb_wraps = list_bridges(adapter, host_uuid)
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/pypowervm/utils/retry.py", line 251, in __retry
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m _raise_exc()
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/pypowervm/utils/retry.py", line 237, in __retry
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m resp = func(*args, **kwds)
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/opt/stack/networking-powervm/networking_powervm/plugins/ibm/agent/powervm/utils.py", line 303, in list_bridges
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m parent_uuid=host_uuid)
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/pypowervm/wrappers/entry_wrapper.py", line 857, in get
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m read_kwargs)
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/pypowervm/wrappers/entry_wrapper.py", line 893, in _read_child
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m child_type=cls.schema_type, **read_kwargs)
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/pypowervm/adapter.py", line 762, in read
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m sensitive=sensitive, helpers=helpers)
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/pypowervm/adapter.py", line 799, in read_by_path
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m sensitive, helpers=helpers)
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/pypowervm/adapter.py", line 823, in _read_by_path
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m sensitive=sensitive)
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/pypowervm/adapter.py", line 645, in _request
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m resp = func(method, path, **kwds)
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/pypowervm/helpers/log_helper.py", line 150, in log_req_resp
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m response = func(*args, **kwds)
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/pypowervm/helpers/vios_busy.py", line 60, in wrapper
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m resp = func(*args, **kwds)
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m File "/usr/local/lib/python2.7/dist-packages/pypowervm/adapter.py", line 432, in request
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00m raise self._get_httperror(resp)
Jul 17 06:30:08 neo21 networking-powervm-sea-agent[1701]: ERROR neutron #033[01;35m#033[00mHttpError: HTTP error 500 for method GET on path /rest/api/uom/ManagedSystem/daf0fca1-d881-3d39-a5f3-fc221870a008/NetworkBridge?group=None: Internal Server Error -- VIOS0005 VIOS_LICENSE: Generic exception occurred in fetching license status for vios vios1 with ID 2 in CEC 8247-21L*212A58A

Restarting the pvm-q-sea-agt service seems to have fixed the issue.

It looks to me like there is a race condition on start up which led to this unchecked exception that messed up the service.

found in a devstack environment using stable/queens.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.