Distributed cloud - dcorch.log showing error logs for ntp and ptp
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
Medium
|
Gerry Kopec |
Bug Description
Brief Description
-----------------
When running distributed cloud system notice frequent ptp and ntp error logs in dcorch.log during sync audit. Logs are "tuple index out of range: IndexError: tuple index out of range", "get_ptp_resources error enabled" and "get_ntp_resources error enabled". These happen multiple times for each subcloud during every audit cycle.
Further investigation of code indicates that ptp and ntp coexistence data schema changes (https:/
Severity
--------
Major
Steps to Reproduce
------------------
Set up DC system with subclouds. Observe /var/log/
Expected Behavior
------------------
Expect clean audit run without errors.
Actual Behavior
----------------
Seeing error logs
Reproducibility
---------------
Reproducible
System Configuration
-------
All-in-one duplex plus worker, DC system controller with 10 subclouds
Branch/Pull Time/Commit
-------
2019-12-09_20-00-00
Last Pass
---------
Unknown
Timestamp/Logs
--------------
2019-12-16 22:21:31.031 1249743 INFO dcorch.
2019-12-16 22:21:31.032 1249743 ERROR dcorch.
2019-12-16 22:21:31.032 1249743 ERROR dcorch.
2019-12-16 22:21:31.032 1249743 ERROR dcorch.
2019-12-16 22:21:31.032 1249743 ERROR dcorch.
2019-12-16 22:21:31.032 1249743 ERROR dcorch.
2019-12-16 22:21:31.032 1249743 ERROR dcorch.
2019-12-16 22:21:31.032 1249743 ERROR dcorch.
2019-12-16 22:21:31.032 1249743 ERROR dcorch.
2019-12-16 22:21:31.032 1249743 ERROR dcorch.
2019-12-16 22:21:31.032 1249743 ERROR dcorch.
...
2019-12-16 22:21:34.747 1249743 INFO dcorch.
...
2019-12-16 22:21:37.863 1249743 INFO dcorch.
2019-12-16 22:21:37.863 1249743 ERROR dcorch.
2019-12-16 22:21:37.863 1249743 ERROR dcorch.
2019-12-16 22:21:37.863 1249743 ERROR dcorch.
2019-12-16 22:21:37.863 1249743 ERROR dcorch.
2019-12-16 22:21:37.863 1249743 ERROR dcorch.
2019-12-16 22:21:37.863 1249743 ERROR dcorch.
2019-12-16 22:21:37.863 1249743 ERROR dcorch.
2019-12-16 22:21:37.863 1249743 ERROR dcorch.
2019-12-16 22:21:37.863 1249743 ERROR dcorch.
2019-12-16 22:21:37.863 1249743 ERROR dcorch.
...
2019-12-16 22:21:38.633 1249743 INFO dcorch.
Test Activity
-------------
DC engineering
Workaround
----------
n/a
Looking at the code some problem areas are: drivers/ openstack/ sysinv_ v1.py:
LOG.debug( "get_ptp uuid=%s enabled=%s mode=%s "
"transport= %s mechanism=%s" %
(ptp. uuid, ptp.enabled, ptp.mode,
ptp. transport, ptp.mechanism))
LOG.debug( "get_ntp uuid=%s enabled=%s ntpservers=%s" %
(intp. uuid, intp.enabled, intp.ntpservers))
dcorch/
- get_ptp() -- ptp.enabled no longer exists
- get_ntp() -- intp.enabled no longer exists
It appears the above exceptions are handled and the "error enabled" logs are generated but ptp/ntp is now set to None and that then triggers other problems.
dcorch/ engine/ sync_services/ sysinv. py:
LOG.info( "get_resource_ id {} NO uuid resource_ type={} ".format(
resource_ type))
- get_resource_id() - NO uuid log expects 2 parms but is only give one and throws exception
- get_resource_info() - resource=None is not handled, resource._info will throw exception
dumps = jsonutils. dumps({ "payload" : resource._info})
dumps = jsonutils. dumps(resource. _info)
LOG. info("get_ resource_ info resource_type={} dumps={}".format(
resource_ type, dumps),
extra= self.log_ extra)
if resource_type in payload_resources:
if 'payload' not in resource._info:
else:
return dumps