| 2024-02-12 23:47:38 |
Ghada Khalil |
description |
Brief Description
Noticed some exceptions in sysinv.log on upgrade related to DiskNotFound
The upgrade appears to be left in this state
sysinv 2023-03-28 17:09:52.533 158668 INFO sysinv.conductor.manager [-] Upgrade in progress - defer platform managed application activity
sysinv 2023-03-28 17:10:52.534 158668 INFO sysinv.conductor.manager [-] Upgrade in progress - defer platform managed application activity
...
sysinv 2023-03-28 19:30:53.004 158668 INFO sysinv.conductor.manager [-] Upgrade in progress - defer platform managed application activity
Severity
Provide the severity of the defect.
<Critical: System/Feature is not usable after the defect>
<Major: System/Feature is usable but degraded>
<Minor: System/Feature is usable with minor issue>
Steps to Reproduce
HPE
(Note: Partitions had been created on the controllers prior to upgrade I believe)
[Upgrade controller-0 here]
2023-03-28T16:34:23.148 controller-1 -bash: info HISTORY: PID=166700 UID=42425 system host-upgrade controller-0
[Looks like unlock of controller here]
2023-03-28T17:00:30.377 controller-1 -bash: info HISTORY: PID=166700 UID=42425 system host-unlock controller-0
Expected Behavior
Expect upgrade without errors in sysinv.log
Actual Behavior
[Error in sysinv.log here]
96d24e24845f
sysinv 2023-03-28 17:08:47.247 103812 INFO sysinv.agent.manager [-] Sysinv Agent platform update by host:
{'availability': 'available', 'config_applied': '6e9e0d4a-0ea2-4b80-a46e-4470b9719df4', 'iscsi_initiator_name': 'iqn.1993-08.org.debian:01:96d24e24845f', 'first_report': True}
sysinv 2023-03-28 17:08:50.143 103812 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._inventory_audit: Remote error: DiskNotFound No disk with id 46fd21bf-4407-491d-9870-7fcf057dfe89
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/sysinv/db/sqlalchemy/api.py", line 2854, in _partition_get
result = query.one()
File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3500, in one
raise orm_exc.NoResultFound("No row was found for one()")
sqlalchemy.orm.exc.NoResultFound: No row was found for one()
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/sysinv/conductor/manager.py", line 5003, in ipv_update_by_ihost
partition = self.dbapi.partition_get(
....
sysinv.logsysinv 2023-03-28 17:08:49.781 158668 INFO sysinv.conductor.manager [-] Partition create on host: 1. Details:
{'forihostid': 1, 'status': 1, 'device_node': '/dev/mapper/mpathb-part1', 'device_path': '/dev/disk/by-id/wwn-0x60002ac0000000000000000800029aaa-part1', 'start_mib': '1', 'end_mib': '581633', 'size_mib': '581632', 'type_guid': 'ba5eba11-0000-1111-2222-000000000001', 'type_name': 'LVM Physical Volume', 'uuid': '8f5e7cb1-ccc1-4055-b997-f92bbadd812c', 'idisk_id': 11, 'idisk_uuid': '7d626936-6cf8-419d-8a9b-20fced07e07c'}
sysinv 2023-03-28 17:08:49.961 158668 INFO sysinv.conductor.manager [-] LVG uuid: 4e47e388-ce5b-4ca0-ab25-5ab2c142cdd5 changed UUID from hzFevZ-dY92-miB4-Z5Bl-GfU2-4znE-qZ4mmL to fpqOK8-Y92q-yEid-ZtGt-Ks8A-x3pn-D9498C
sysinv 2023-03-28 17:08:50.140 158668 ERROR zerorpc.core [-] : sysinv.common.exception.DiskNotFound: No disk with id 46fd21bf-4407-491d-9870-7fcf057dfe89
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core Traceback (most recent call last):
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core File "/usr/lib/python3/dist-packages/sysinv/db/sqlalchemy/api.py", line 2854, in _partition_get
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core result = query.one()
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3500, in one
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core raise orm_exc.NoResultFound("No row was found for one()")
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core sqlalchemy.orm.exc.NoResultFound: No row was found for one()
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core During handling of the above exception, another exception occurred:
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core Traceback (most recent call last):
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core File "/usr/lib/python3/dist-packages/sysinv/conductor/manager.py", line 5003, in ipv_update_by_ihost
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core partition = self.dbapi.partition_get(
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core File "/usr/lib/python3/dist-packages/sysinv/db/sqlalchemy/objects.py", line 29, in wrapper
Reproducibility
<Reproducible/Intermittent/Seen once>
State if the issue is 100% reproducible, intermittent or seen once. If it is intermittent, state the frequency of occurrence
Last Pass
TBD
Timestamp/Logs
see times inline above
Alarms
$ fm alarm-list
-------------------------------------------------------------------------------------------------------------------------
Alarm ID Reason Text Entity ID Severity Time Stamp
-------------------------------------------------------------------------------------------------------------------------
280.002 subcloud1 load sync_status is out-of-sync subcloud=subcloud1. major 2023-03-28T17
resource=load :32:23.376003
750.006 A configuration change requires a reapply of the cert-manager k8s_application= warning 2023-03-28T17
application. cert-manager :09:01.352747
750.006 A configuration change requires a reapply of the platform-integ-apps k8s_application= warning 2023-03-28T17
application. platform-integ-apps :09:01.133454
500.101 Developer patch certificate is enabled host=controller critical 2023-03-28T16
:32:15.886883
$ system host-disk-list controller-0
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
uuid device_node device_num device_type size_gib available_gib rpm serial_id device_path
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
18a50617-16bc-4640-9781-e0755c52e50a /dev/mapper/mpatha 64768 SSD 1229.0 0.0 N/A 360002ac0000000000000000700029aaa /dev/disk/by-id/wwn-0x60002ac0000000000000000700029aaa
7d626936-6cf8-419d-8a9b-20fced07e07c /dev/mapper/mpathb 64773 SSD 2867.0 2298.981 N/A 360002ac0000000000000000800029aaa /dev/disk/by-id/wwn-0x60002ac0000000000000000800029aaa
9e8aab0a-1f3a-4386-bde1-e9ade337372e /dev/mapper/mpathc 64775 SSD 1024.0 0.0 N/A 360002ac0000000000000001d00029aaa /dev/disk/by-id/wwn-0x60002ac0000000000000001d00029aaa
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[sysadmin@controller-1 log(keystone_admin)]$ system host-disk-list controller-1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
uuid device_node device_num device_type size_gib available_gib rpm serial_id device_path
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
8c62f9e2-c49a-43d1-aa9f-c890b7c19ccc /dev/mapper/mpatha 64768 SSD 1229.0 0.0 N/A 360002ac0000000000000000900029aaa /dev/disk/by-id/wwn-0x60002ac0000000000000000900029aaa
166f228f-400a-4789-8536-83497f06293c /dev/mapper/mpathb 64773 SSD 2867.0 2298.981 N/A 360002ac0000000000000000a00029aaa /dev/disk/by-id/wwn-0x60002ac0000000000000000a00029aaa
a546f167-429b-4a25-a46c-87ba94ef0e81 /dev/mapper/mpathc 64775 SSD 1024.0 0.0 N/A 360002ac0000000000000001e00029aaa /dev/disk/by-id/wwn-0x60002ac0000000000000001e00029aaa
---------------------------------------------------------------------------
[sysadmin@controller-1 log(keystone_admin)]$ system host-disk-partition-list controller-0
System platform upgrade is in progress.
The command may display the target configuration that has not yet been applied to the host.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
uuid device_path device_node type_guid type_name size_gib status
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
baebfc2e-488e-49ed-9d33-0c413aed81ad /dev/disk/by-id/wwn-0x60002ac0000000000000000700029aaa-part4 /dev/mapper/mpatha-part4 e6d6d379-f507-44c2-a23c-238f2a3df928 Linux LVM 1197.409 In-Use
8f5e7cb1-ccc1-4055-b997-f92bbadd812c /dev/disk/by-id/wwn-0x60002ac0000000000000000800029aaa-part1 /dev/mapper/mpathb-part1 ba5eba11-0000-1111-2222-000000000001 LVM Physical Volume 568.0 Ready
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[sysadmin@controller-1 log(keystone_admin)]$ system host-disk-partition-list controller-1
System platform upgrade is in progress.
The command may display the target configuration that has not yet been applied to the host.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
uuid device_path device_node type_guid type_name size_gib status
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
f1567b03-f0ce-45e0-af08-f39a2a4c3e8c /dev/disk/by-id/wwn-0x60002ac0000000000000000900029aaa-part4 /dev/mapper/mpatha-part4 e6d6d379-f507-44c2-a23c-238f2a3df928 Linux LVM 1197.409 In-Use
e1b5657a-9b51-4b08-bbcf-241663f583fd /dev/disk/by-id/wwn-0x60002ac0000000000000000a00029aaa-part1 /dev/mapper/mpathb-part1 ba5eba11-0000-1111-2222-000000000001 LVM Physical Volume 568.0 In-Use
Test Activity
Feature Testing (Upgrade)
Workaround
Describe workaround if available |
Brief Description
Noticed some exceptions in sysinv.log on upgrade related to DiskNotFound
The upgrade appears to be left in this state
sysinv 2023-03-28 17:09:52.533 158668 INFO sysinv.conductor.manager [-] Upgrade in progress - defer platform managed application activity
sysinv 2023-03-28 17:10:52.534 158668 INFO sysinv.conductor.manager [-] Upgrade in progress - defer platform managed application activity
...
sysinv 2023-03-28 19:30:53.004 158668 INFO sysinv.conductor.manager [-] Upgrade in progress - defer platform managed application activity
Severity
<Minor: System/Feature is usable with minor issue>
Steps to Reproduce
HPE
(Note: Partitions had been created on the controllers prior to upgrade I believe)
[Upgrade controller-0 here]
2023-03-28T16:34:23.148 controller-1 -bash: info HISTORY: PID=166700 UID=42425 system host-upgrade controller-0
[Looks like unlock of controller here]
2023-03-28T17:00:30.377 controller-1 -bash: info HISTORY: PID=166700 UID=42425 system host-unlock controller-0
Expected Behavior
Expect upgrade without errors in sysinv.log
Actual Behavior
[Error in sysinv.log here]
96d24e24845f
sysinv 2023-03-28 17:08:47.247 103812 INFO sysinv.agent.manager [-] Sysinv Agent platform update by host:
{'availability': 'available', 'config_applied': '6e9e0d4a-0ea2-4b80-a46e-4470b9719df4', 'iscsi_initiator_name': 'iqn.1993-08.org.debian:01:96d24e24845f', 'first_report': True}
sysinv 2023-03-28 17:08:50.143 103812 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._inventory_audit: Remote error: DiskNotFound No disk with id 46fd21bf-4407-491d-9870-7fcf057dfe89
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/sysinv/db/sqlalchemy/api.py", line 2854, in _partition_get
result = query.one()
File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3500, in one
raise orm_exc.NoResultFound("No row was found for one()")
sqlalchemy.orm.exc.NoResultFound: No row was found for one()
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/sysinv/conductor/manager.py", line 5003, in ipv_update_by_ihost
partition = self.dbapi.partition_get(
....
sysinv.logsysinv 2023-03-28 17:08:49.781 158668 INFO sysinv.conductor.manager [-] Partition create on host: 1. Details:
{'forihostid': 1, 'status': 1, 'device_node': '/dev/mapper/mpathb-part1', 'device_path': '/dev/disk/by-id/wwn-0x60002ac0000000000000000800029aaa-part1', 'start_mib': '1', 'end_mib': '581633', 'size_mib': '581632', 'type_guid': 'ba5eba11-0000-1111-2222-000000000001', 'type_name': 'LVM Physical Volume', 'uuid': '8f5e7cb1-ccc1-4055-b997-f92bbadd812c', 'idisk_id': 11, 'idisk_uuid': '7d626936-6cf8-419d-8a9b-20fced07e07c'}
sysinv 2023-03-28 17:08:49.961 158668 INFO sysinv.conductor.manager [-] LVG uuid: 4e47e388-ce5b-4ca0-ab25-5ab2c142cdd5 changed UUID from hzFevZ-dY92-miB4-Z5Bl-GfU2-4znE-qZ4mmL to fpqOK8-Y92q-yEid-ZtGt-Ks8A-x3pn-D9498C
sysinv 2023-03-28 17:08:50.140 158668 ERROR zerorpc.core [-] : sysinv.common.exception.DiskNotFound: No disk with id 46fd21bf-4407-491d-9870-7fcf057dfe89
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core Traceback (most recent call last):
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core File "/usr/lib/python3/dist-packages/sysinv/db/sqlalchemy/api.py", line 2854, in _partition_get
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core result = query.one()
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3500, in one
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core raise orm_exc.NoResultFound("No row was found for one()")
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core sqlalchemy.orm.exc.NoResultFound: No row was found for one()
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core During handling of the above exception, another exception occurred:
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core Traceback (most recent call last):
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core File "/usr/lib/python3/dist-packages/sysinv/conductor/manager.py", line 5003, in ipv_update_by_ihost
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core partition = self.dbapi.partition_get(
2023-03-28 17:08:50.140 158668 ERROR zerorpc.core File "/usr/lib/python3/dist-packages/sysinv/db/sqlalchemy/objects.py", line 29, in wrapper
Reproducibility
<Reproducible/Intermittent/Seen once>
State if the issue is 100% reproducible, intermittent or seen once. If it is intermittent, state the frequency of occurrence
Last Pass
TBD
Timestamp/Logs
see times inline above
Alarms
$ fm alarm-list
-------------------------------------------------------------------------------------------------------------------------
Alarm ID Reason Text Entity ID Severity Time Stamp
-------------------------------------------------------------------------------------------------------------------------
280.002 subcloud1 load sync_status is out-of-sync subcloud=subcloud1. major 2023-03-28T17
resource=load :32:23.376003
750.006 A configuration change requires a reapply of the cert-manager k8s_application= warning 2023-03-28T17
application. cert-manager :09:01.352747
750.006 A configuration change requires a reapply of the platform-integ-apps k8s_application= warning 2023-03-28T17
application. platform-integ-apps :09:01.133454
500.101 Developer patch certificate is enabled host=controller critical 2023-03-28T16
:32:15.886883
$ system host-disk-list controller-0
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
uuid device_node device_num device_type size_gib available_gib rpm serial_id device_path
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
18a50617-16bc-4640-9781-e0755c52e50a /dev/mapper/mpatha 64768 SSD 1229.0 0.0 N/A 360002ac0000000000000000700029aaa /dev/disk/by-id/wwn-0x60002ac0000000000000000700029aaa
7d626936-6cf8-419d-8a9b-20fced07e07c /dev/mapper/mpathb 64773 SSD 2867.0 2298.981 N/A 360002ac0000000000000000800029aaa /dev/disk/by-id/wwn-0x60002ac0000000000000000800029aaa
9e8aab0a-1f3a-4386-bde1-e9ade337372e /dev/mapper/mpathc 64775 SSD 1024.0 0.0 N/A 360002ac0000000000000001d00029aaa /dev/disk/by-id/wwn-0x60002ac0000000000000001d00029aaa
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[sysadmin@controller-1 log(keystone_admin)]$ system host-disk-list controller-1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
uuid device_node device_num device_type size_gib available_gib rpm serial_id device_path
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
8c62f9e2-c49a-43d1-aa9f-c890b7c19ccc /dev/mapper/mpatha 64768 SSD 1229.0 0.0 N/A 360002ac0000000000000000900029aaa /dev/disk/by-id/wwn-0x60002ac0000000000000000900029aaa
166f228f-400a-4789-8536-83497f06293c /dev/mapper/mpathb 64773 SSD 2867.0 2298.981 N/A 360002ac0000000000000000a00029aaa /dev/disk/by-id/wwn-0x60002ac0000000000000000a00029aaa
a546f167-429b-4a25-a46c-87ba94ef0e81 /dev/mapper/mpathc 64775 SSD 1024.0 0.0 N/A 360002ac0000000000000001e00029aaa /dev/disk/by-id/wwn-0x60002ac0000000000000001e00029aaa
---------------------------------------------------------------------------
[sysadmin@controller-1 log(keystone_admin)]$ system host-disk-partition-list controller-0
System platform upgrade is in progress.
The command may display the target configuration that has not yet been applied to the host.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
uuid device_path device_node type_guid type_name size_gib status
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
baebfc2e-488e-49ed-9d33-0c413aed81ad /dev/disk/by-id/wwn-0x60002ac0000000000000000700029aaa-part4 /dev/mapper/mpatha-part4 e6d6d379-f507-44c2-a23c-238f2a3df928 Linux LVM 1197.409 In-Use
8f5e7cb1-ccc1-4055-b997-f92bbadd812c /dev/disk/by-id/wwn-0x60002ac0000000000000000800029aaa-part1 /dev/mapper/mpathb-part1 ba5eba11-0000-1111-2222-000000000001 LVM Physical Volume 568.0 Ready
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[sysadmin@controller-1 log(keystone_admin)]$ system host-disk-partition-list controller-1
System platform upgrade is in progress.
The command may display the target configuration that has not yet been applied to the host.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
uuid device_path device_node type_guid type_name size_gib status
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
f1567b03-f0ce-45e0-af08-f39a2a4c3e8c /dev/disk/by-id/wwn-0x60002ac0000000000000000900029aaa-part4 /dev/mapper/mpatha-part4 e6d6d379-f507-44c2-a23c-238f2a3df928 Linux LVM 1197.409 In-Use
e1b5657a-9b51-4b08-bbcf-241663f583fd /dev/disk/by-id/wwn-0x60002ac0000000000000000a00029aaa-part1 /dev/mapper/mpathb-part1 ba5eba11-0000-1111-2222-000000000001 LVM Physical Volume 568.0 In-Use
Test Activity
Feature Testing (Upgrade)
Workaround
Describe workaround if available |
|
