Comment 47 for bug 1829403

Hi Austin,

I took a look at yow-cgcs-wildcat-99-103 yesterday. The database and hiera data were not updated on both compute-0 and compute-2, although there were logs from 2019-07-19 showing both database and hiera data were updated. There are 2 possible sources that could cause this failure:
One, the database update failed without error logs, resulting in the hiera data not being updated. Second, the sysinv-agent reporting the default memory inventory prior to hiera data update, resetting the vswitch huge pages to 0 in the database.

To recover, I locked /unlocked the compute-0 to re-populate the vswitch huge pages and it was successful.

# compute-2
2019-07-19 20:49:47.430 103746 INFO sysinv.api.controllers.v1.host [-] Updating mem values of host(compute-2) node(5): {'vm_hugepages_nr_4K': 1379328, 'vm_hugepages_nr_2M': 24246, 'vswitch_hugepages_nr': 1}
2019-07-19 20:49:47.459 103738 INFO sysinv.api.controllers.v1.host [-] compute-1 1. delta_handle ['uptime', 'task']
2019-07-19 20:49:47.530 103746 INFO sysinv.api.controllers.v1.host [-] host(compute-2) node(6): vm_mem_mib=60218,vm_mem_mib_possible (from agent) = 63242
2019-07-19 20:49:47.530 103746 INFO sysinv.api.controllers.v1.host [-] Updating mem values of host(compute-2) node(6): {'vm_hugepages_nr_4K': 1541632, 'vm_hugepages_nr_2M': 27098, 'vswitch_hugepages_nr': 1}
2019-07-19 20:49:50.612 102604 INFO sysinv.puppet.puppet [req-df405ec9-1017-4b2d-861a-e4790ad819d6 admin admin] Updating hiera for host: compute-2 with config_uuid: None

# compute-0
2019-07-19 20:50:01.085 103738 INFO sysinv.api.controllers.v1.host [-] Updating mem values of host(compute-0) node(3): {'vm_hugepages_nr_4K': 1379840, 'vm_hugepages_nr_2M': 24247, 'vswitch_hugepages_nr': 1}
2019-07-19 20:50:01.187 103738 INFO sysinv.api.controllers.v1.host [-] host(compute-0) node(4): vm_mem_mib=60215,vm_mem_mib_possible (from agent) = 63238
2019-07-19 20:50:01.188 103738 INFO sysinv.api.controllers.v1.host [-] Updating mem values of host(compute-0) node(4): {'vm_hugepages_nr_4K': 1541888, 'vm_hugepages_nr_2M': 27096, 'vswitch_hugepages_nr': 1}
2019-07-19 20:50:05.532 102604 INFO sysinv.puppet.puppet [req-3de697a6-37d1-4c78-bc0f-6005f67b27b9 admin admin] Updating hiera for host: compute-0 with config_uuid: None