Debian: lldp tool leads to a KeyError in sysinv logs

[Al Bailey]

Brief Description
-----------------
Debian AIO-SX sysinv logs show a KeyError related to lldp.

Severity
--------
Minor

Steps to Reproduce
------------------
Install and unlock an AIO-SX system for Debian.
Check the sysinv.logs

Expected Behavior
------------------
No KeyError

Actual Behavior
----------------
KeyError

Reproducibility
---------------
100%

System Configuration
--------------------
AIO-SX

Branch/Pull Time/Commit
-----------------------
March 21, 2022

Last Pass
---------
N/A

Timestamp/Logs
--------------
sysinv 2022-03-22 14:24:19.411 57499 ERROR sysinv.agent.lldp.manager [-] Sysinv LLDP agent driver 'lldpd' failed in lldp_neighbours_list
sysinv 2022-03-22 14:24:19.411 57499 ERROR sysinv.agent.lldp.manager [-] 0: KeyError: 0
2022-03-22 14:24:19.411 57499 ERROR sysinv.agent.lldp.manager Traceback (most recent call last):
2022-03-22 14:24:19.411 57499 ERROR sysinv.agent.lldp.manager File "/usr/lib/python3/dist-packages/sysinv/agent/lldp/manager.py", line 147, in lldp_neighbours_list
2022-03-22 14:24:19.411 57499 ERROR sysinv.agent.lldp.manager return self._call_drivers_and_return_array("lldp_neighbours_list",
2022-03-22 14:24:19.411 57499 ERROR sysinv.agent.lldp.manager File "/usr/lib/python3/dist-packages/sysinv/agent/lldp/manager.py", line 75, in _call_drivers_and_return_array
2022-03-22 14:24:19.411 57499 ERROR sysinv.agent.lldp.manager ret = ret + method()
2022-03-22 14:24:19.411 57499 ERROR sysinv.agent.lldp.manager File "/usr/lib/python3/dist-packages/sysinv/agent/lldp/drivers/lldpd/driver.py", line 291, in lldp_neighbours_list
2022-03-22 14:24:19.411 57499 ERROR sysinv.agent.lldp.manager lldp = data['lldp'][0]
2022-03-22 14:24:19.411 57499 ERROR sysinv.agent.lldp.manager KeyError: 0
2022-03-22 14:24:19.411 57499 ERROR sysinv.agent.lldp.manager
sysinv 2022-03-22 14:24:19.419 57499 ERROR sysinv.agent.lldp.manager [-] 0: KeyError: 0

Test Activity
-------------
Feature Testing

Workaround
----------
The code needs to use json0 instead of json when invoking the tool.

This is due to the default format being changed when lldp 0.9.2 was released.
Debian is running 1.0.11
Centos is still running 0.9.0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/834780

[Al Bailey (albailey1974)]

The proposed fix enables the code to get further but then encounters a python3 conversion issue

I will investigate this as part of this bug.

2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last):
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python3/dist-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task task(self, context)
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python3/dist-packages/sysinv/agent/manager.py", line 1259, in _agent_audit
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task self.agent_audit(context, host_uuid=self._ihost_uuid,
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs)
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python3/dist-packages/sysinv/agent/manager.py", line 1361, in agent_audit
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task self.host_lldp_get_and_report(icontext, rpcapi, self._ihost_uuid)
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python3/dist-packages/sysinv/agent/manager.py", line 486, in host_lldp_get_and_report
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task rpcapi.lldp_agent_update_by_host(context,
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python3/dist-packages/sysinv/conductor/rpcapi.py", line 240, in lldp_agent_update_by_host
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task return self.call(context,
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python3/dist-packages/sysinv/openstack/common/rpc/proxy.py", line 121, in call
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task result = rpc.call(context, real_topic, msg, timeout)
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python3/dist-packages/sysinv/openstack/common/rpc/__init__.py", line 139, in call
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task return _get_impl().call(CONF, context, topic, msg, timeout)
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python3/dist-packages/sysinv/openstack/common/rpc/impl_kombu.py", line 813, in call
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task return rpc_amqp.call(
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python3/dist-packages/sysinv/openstack/common/rpc/amqp.py", line 619, in call
2022-03-22 19:30:48.065 1077855 ERROR sysinv.openstack.common.periodic_task rv = list(rv)
2022-03-22 19:...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/836080

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/836080
Committed: https://opendev.org/starlingx/config/commit/c66caa6b357720dd45492c0d3dacec3b2232f32c
Submitter: "Zuul (22348)"
Branch: master

commit c66caa6b357720dd45492c0d3dacec3b2232f32c
Author: Al Bailey <email address hidden>
Date: Sun Mar 27 18:09:02 2022 +0000

    Adding unit tests for lldp agent driver activities

    The lldp CLI returns different data on Debian than Centos
    and so new unit tests are being written to validate the
    code that processes those results

    The unit tests are based on json data files that are
    generated for those versions of the tool.

    This submission covers the 0.9.0 version of the tool
    which is currently the version for STX 6.0 on Centos

    Current level of coverage is 70% for the lldp driver,
    with the majority of the uncovered lines relating
    to the input data not having vlans.

    Partial-Bug: 1965978
    Signed-off-by: Al Bailey <email address hidden>
    Change-Id: I8827e3652cdaa04e4c42dffaa85b92bee0e382f0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/834780
Committed: https://opendev.org/starlingx/config/commit/192867300e628fa44420e66050ac602483de1eea
Submitter: "Zuul (22348)"
Branch: master

commit 192867300e628fa44420e66050ac602483de1eea
Author: Al Bailey <email address hidden>
Date: Tue Mar 22 17:00:33 2022 +0000

    Fix lldp format in debian environment

    Centos and Debian are using different versions of lldp tool.

    Centos (python2) uses an older lldp 0.9.0
    Debian (python3) uses a newer lldp 1.0.11

    When calling the lldp tool passing 'json' flag, the results
    are formatted differently in those versions of the tool.

    A "json0" flag was added for backward compatability but
    does not exist in 0.9.0.

    For this solution:
    'json' argument is kept for the older lldp on Centos.
    'json0' argument is used for the newer lldp on Debian.

    The results have some slight variations, so some special
    handling in the processing methods is being added.

    For Centos the results will be handled exactly the same.
    For Debian, certain areas of the data need to be handled
    differently.

    Test Plan:
      PASS: Verify the KeyErrors are no longer generated in Debian
      PASS: Verify no new exceptions are generated in Debian.
      PASS: Verify Centos still works.
      PASS: Verify the controllerconfig changes can still be parsed

    Closes-Bug: 1965978
    Signed-off-by: Al Bailey <email address hidden>
    Change-Id: I133e934f58fcc03c2625ec818094acea23e21c12