Error messages observed in /var/log/sysinv.log file

[Alexandru Dimofte]

Brief Description
-----------------
I observed some error messages in /var/log/sysinv.log file on all configurations. This are visible in any build after ussuri was merged. However the Daily Sanity is GREEN and this issues seems to not have a big impact.

Severity
--------
<Minor: System/Feature is usable with minor issue>

Steps to Reproduce
------------------
Login into controller-0 on any configuration, using a layered build from master branch after ussuri merge.
Execute: cat /var/log/sysinv.log | grep error

Expected Behavior
------------------
Ideal would be to not see any error message.

Actual Behavior
----------------
Error messages observed:
1.
ing again in 1 seconds.: error: [Errno 111] ECONNREFUSED
sysinv 2020-07-10 18:59:34.037 74028 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
sysinv 2020-07-10 18:59:35.044 74050 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.: error: [Errno 111] ECONNREFUSED
sysinv 2020-07-10 18:59:35.044 74028 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.: error: [Errno 111] ECONNREFUSED

2.
sysinv 2020-07-10 19:19:28.039 85066 WARNING wsme.api [-] Client-side error: No disk with id 348cd8f4-8357-48c5-a986-bce82b5c0b0b: DiskNotFound: No disk with id 348cd8f4-8357-48c5-a986-bce82b5c0b0b
sysinv 2020-07-10 19:19:43.655 120176 WARNING sysinv.api.controllers.v1.rest_api [-] URLError Error e=<urlopen error [Errno 111] ECONNREFUSED>: URLError: <urlopen error [Errno 111] ECONNREFUSED>
2020-07-10 19:19:43.660 74028 ERROR sysinv.openstack.common.rpc.amqp error=None)
2020-07-10 19:19:43.660 74028 ERROR sysinv.openstack.common.rpc.amqp error=error))
sysinv 2020-07-10 19:19:51.651 85066 WARNING wsme.api [-] Client-side error: No disk with id 49df7ab4-d633-46c3-a7bb-4eb16c979f78: DiskNotFound: No disk with id 49df7ab4-d633-46c3-a7bb-4eb16c979f78

3.
sysinv 2020-07-10 19:35:43.735 104173 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"25763f55-f3fb-44dd-b3aa-c69fd353ee14","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":3,"quorum":[0],"quorum_names":["controller-0"],"monmap":{"epoch":1,"fsid":"25763f55-f3fb-44dd-b3aa-c69fd353ee14","modified":"2020-07-10 19:27:06.196102","created":"2020-07-10 19:27:06.196102","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller-0","addr":"192.168.204.2:6789/0","public_addr":"192.168.204.2:6789/0"}]},"osdmap":{"osdmap":{"epoch":6,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[],"num_pgs":0,"num_pools":0,"num_objects":0,"data_bytes":0,"bytes_used":112189440,"bytes_avail":958541803520,"bytes_total":958653992960},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":58,"active_gid":4135,"active_name":"controller-0","active_addr":"192.168.204.2:6804/103669","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":true,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:7999/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'140490042174928'}

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
All 4 configurations are affected.

Branch/Pull Time/Commit
-----------------------
The issue can be observed on any master layered build after ussuri merge. Example:
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/20200711T013416Z/outputs/iso/

Last Pass
---------
Before ussuri was merged.

Timestamp/Logs
--------------
Will be uploaded.

Test Activity
-------------
This issue was not observed running any test. It was a human observation.

Workaround
----------
-

[Ghada Khalil (gkhalil)]

Assigning to the distro.openstack PL as this appears related to the Ussuri rebase

[Ghada Khalil (gkhalil)]

Marking as stx.4.0 for now until there is a read-out from the openstack team. This is an issue introduced after the rebase.

[yong hu (yhu6)]

@Austin please help address this issue.

[Austin Sun (sunausti)]

Hi Alex:
   I think these 3 issues you reported are normal case.
   for item 1) This is normal as rabbitmq service is not fully start yet . once rabbitmq started, sysinv will work properly .
   for item 2) would you like to check if /dev/disk/by-path/pci-0000:00:11.5-ata-2.0-part6 is in your setup ?
   and you can check by command 'system host-disk-partition-list controller-0' , i think the partition id will be there .
   for item 3) this should just ceph client report , it is not sysinv issue too.

   I set the status to invalid now, if you still have some question, we can discuss.

Thanks.
BR
Austin Sun.