Upgrade: controller-1 failed inventory report with sysinv-agent failure

[Reynaldo P Gomes]

Brief Description
-----------------
controller-1 failed to report initial inventory after upgrade to 23.09 from 22.12.

Severity
-----------------
Critical

Steps to Reproduce
-----------------
From 22.12 upgrade to master (23.09), follow the step to system host-upgrade controller-1. After controller-1 pxeboot and reboot, controller-1 stays in the state of installing (won't complete initial inventory report).

From sysinv.log in controller-1, found below error message:

Expected Behavior
-----------------
controller-1 should complete its initial inventory report

Actual Behavior
-----------------
controller-1 stays as "installing" forever

Reproducibility
-----------------
100%

System Configuration
-----------------
DX configuration

Branch/Pull Time/Commit
-----------------------
Master branch

Last Pass
-----------------
Earlier this year

Timestamp/Logs
-----------------
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task Traceback (most recent call last):
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task task(self, context)
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sysinv/agent/manager.py", line 1423, in _inventory_audit
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task self.agent_audit(context, host_uuid=self._ihost_uuid,
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task return f(*args, **kwargs)
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sysinv/agent/manager.py", line 1508, in agent_audit
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task self.ihost_inv_get_and_report(icontext)
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sysinv/agent/manager.py", line 895, in ihost_inv_get_and_report
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task self._ihost_cstates_available = ihost.get('cstates_available') or None
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sysinv/objects/base.py", line 443, in get
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task return self[key]
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sysinv/objects/base.py", line 422, in _getitem_
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task return getattr(self, name)
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sysinv/objects/base.py", line 56, in getter
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task self.obj_load_attr(name)
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sysinv/objects/base.py", line 347, in obj_load_attr
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task raise NotImplementedError(
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task NotImplementedError: Cannot load 'cstates_available' in the base class
2023-10-04 17:02:29.374 1331 ERROR oslo_service.periodic_task
sysinv 2023-10-04 17:03:29.393 1331 INFO sysinv.agent.manager [req-f88e4468-4006-4104-9cb8-819a9b415bc5 None None] _report_to_conductor initial_reports_required={'disk', 'kernel', 'numa', 'lvg', 'port', 'cpu', 'memory', 'pv', 'pci_device'}

Test Activity
-----------------
Developer Testing

Workaround
-----------------
No workaround. code change is required

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/897586

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/897586
Committed: https://opendev.org/starlingx/config/commit/4c207964938d435d97a89554b9fea10fc71665c3
Submitter: "Zuul (22348)"
Branch: master

commit 4c207964938d435d97a89554b9fea10fc71665c3
Author: Reynaldo P Gomes <email address hidden>
Date: Fri Oct 6 15:39:45 2023 -0300

    Fix fields missing error during upgrade process

    This change fixes the error during the upgrade process when the host
    under update tries to access 2 new fields (min_cpu_mhz_allowed and
    cstates_available) that do not exists on the active controller yet.

    TEST: AIO-DX successfully upgraded from 22.12 to master.

    Closes-Bug: 2038667
    Change-Id: I80f123bb44321d025217d292e7d826e23d8c7233
    Signed-off-by: Reynaldo P Gomes <email address hidden>