Activity log for bug #1882251

Date Who What changed Old value New value Message
2020-06-05 14:08:45 Peng Peng bug added bug
2020-06-05 14:09:21 Peng Peng tags stx.retestneeded
2020-06-05 14:35:58 Peng Peng description Brief Description ----------------- During system initializing, after both controller nodes had been installed, but compute nodes kept offline. Severity -------- Major Steps to Reproduce ------------------ install multi nodes system Expected Behavior ------------------ all compute nodes are online Actual Behavior ---------------- compute nodes are offline Reproducibility --------------- Reproducible 2 of 2 System Configuration -------------------- Multi-node system Lab-name: wcp_8-12 Branch/Pull Time/Commit ----------------------- 2020-06-04_20-00-00 Last Pass --------- 2020-06-03_20-00-00 Timestamp/Logs -------------- [2020-06-05 03:48:25,548] 2197 INFO MainThread fresh_install_helper.wait_for_deploy_mgr_hosts_config:: Waiting for Deploy Mgr to configure and unlock hosts: ['controller-1', 'compute-0', 'compute-1'] ... [2020-06-05 03:48:25,548] 58 INFO MainThread kube_helper.exec_kube_cmd:: exec_kube_cmd:kubectl get hosts -n=deployment -o=wide [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get hosts -n=deployment -o=wide NAME ADMINISTRATIVE OPERATIONAL AVAILABILITY PROFILE INSYNC RECONCILED compute-0 unlocked disabled failed compute-1-profile false false compute-1 unlocked disabled failed compute-1-profile false false compute-2 unlocked disabled failed compute-2-profile false false controller-0 unlocked enabled available controller-0-profile true true controller-1 unlocked enabled available controller-0-profile true true Test Activity ------------- Sanity Brief Description ----------------- During system initializing, after both controller nodes had been installed, but compute nodes kept offline. Severity -------- Major Steps to Reproduce ------------------ install multi nodes system Expected Behavior ------------------ all compute nodes are online Actual Behavior ---------------- compute nodes are offline Reproducibility --------------- Reproducible 2 of 2 System Configuration -------------------- Multi-node system Lab-name: wcp_8-12 Branch/Pull Time/Commit ----------------------- 2020-06-04_20-00-00 Last Pass --------- 2020-06-03_20-00-00 Timestamp/Logs -------------- [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get hosts -n=deployment -o=wide NAME ADMINISTRATIVE OPERATIONAL AVAILABILITY PROFILE INSYNC RECONCILED compute-0 unlocked disabled failed compute-1-profile false false compute-1 unlocked disabled failed compute-1-profile false false compute-2 unlocked disabled failed compute-2-profile false false controller-0 unlocked enabled available controller-0-profile true true controller-1 unlocked enabled available controller-0-profile true true [sysadmin@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | compute-0 | worker | unlocked | disabled | failed | | 3 | compute-1 | worker | unlocked | disabled | failed | | 4 | compute-2 | worker | unlocked | disabled | failed | | 5 | controller-1 | controller | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-show compute-0 +-----------------------+----------------------------------------------------------------+ | Property | Value | +-----------------------+----------------------------------------------------------------+ | action | none | | administrative | unlocked | | availability | failed | | bm_ip | 2620:10a:a001:a102::134 | | bm_type | dynamic | | bm_username | root | | boot_device | /dev/disk/by-path/pci-0000:18:00.0-scsi-0:0:8:0 | | capabilities | {} | | clock_synchronization | ntp | | config_applied | 6fc32a18-c718-4a2c-b1e5-7c1ba3c65e72 | | config_status | None | | config_target | 6fc32a18-c718-4a2c-b1e5-7c1ba3c65e72 | | console | ttyS0,115200n8 | | created_at | 2020-06-05T03:33:18.372476+00:00 | | device_image_update | None | | hostname | compute-0 | | id | 2 | | install_output | text | | install_state | completed | | install_state_info | None | | inv_state | inventoried | | invprovision | provisioning | | location | {} | | mgmt_ip | face::31f0:b4b7:17d6:b129 | | mgmt_mac | 3c:fd:fe:b1:26:d8 | | operational | disabled | | personality | worker | | reboot_needed | False | | reserved | False | | rootfs_device | /dev/disk/by-path/pci-0000:18:00.0-scsi-0:0:8:0 | | serialid | None | | software_load | 20.06 | | subfunctions | worker,lowlatency | | task | Configuration failure, threshold reached, Lock/Unlock to retry | | tboot | false | | ttys_dcd | None | | updated_at | 2020-06-05T14:34:08.504008+00:00 | | uptime | 37323 | | uuid | 9a7b34ac-28c9-4854-a97d-a546d5f59952 | | vim_progress_status | None | +-----------------------+----------------------------------------------------------------+ Test Activity ------------- Sanity
2020-06-05 15:16:00 Yang Liu summary compute nodes offline during system initialize worker nodes failed after initial unlock
2020-06-05 15:17:35 Yang Liu description Brief Description ----------------- During system initializing, after both controller nodes had been installed, but compute nodes kept offline. Severity -------- Major Steps to Reproduce ------------------ install multi nodes system Expected Behavior ------------------ all compute nodes are online Actual Behavior ---------------- compute nodes are offline Reproducibility --------------- Reproducible 2 of 2 System Configuration -------------------- Multi-node system Lab-name: wcp_8-12 Branch/Pull Time/Commit ----------------------- 2020-06-04_20-00-00 Last Pass --------- 2020-06-03_20-00-00 Timestamp/Logs -------------- [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get hosts -n=deployment -o=wide NAME ADMINISTRATIVE OPERATIONAL AVAILABILITY PROFILE INSYNC RECONCILED compute-0 unlocked disabled failed compute-1-profile false false compute-1 unlocked disabled failed compute-1-profile false false compute-2 unlocked disabled failed compute-2-profile false false controller-0 unlocked enabled available controller-0-profile true true controller-1 unlocked enabled available controller-0-profile true true [sysadmin@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | compute-0 | worker | unlocked | disabled | failed | | 3 | compute-1 | worker | unlocked | disabled | failed | | 4 | compute-2 | worker | unlocked | disabled | failed | | 5 | controller-1 | controller | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-show compute-0 +-----------------------+----------------------------------------------------------------+ | Property | Value | +-----------------------+----------------------------------------------------------------+ | action | none | | administrative | unlocked | | availability | failed | | bm_ip | 2620:10a:a001:a102::134 | | bm_type | dynamic | | bm_username | root | | boot_device | /dev/disk/by-path/pci-0000:18:00.0-scsi-0:0:8:0 | | capabilities | {} | | clock_synchronization | ntp | | config_applied | 6fc32a18-c718-4a2c-b1e5-7c1ba3c65e72 | | config_status | None | | config_target | 6fc32a18-c718-4a2c-b1e5-7c1ba3c65e72 | | console | ttyS0,115200n8 | | created_at | 2020-06-05T03:33:18.372476+00:00 | | device_image_update | None | | hostname | compute-0 | | id | 2 | | install_output | text | | install_state | completed | | install_state_info | None | | inv_state | inventoried | | invprovision | provisioning | | location | {} | | mgmt_ip | face::31f0:b4b7:17d6:b129 | | mgmt_mac | 3c:fd:fe:b1:26:d8 | | operational | disabled | | personality | worker | | reboot_needed | False | | reserved | False | | rootfs_device | /dev/disk/by-path/pci-0000:18:00.0-scsi-0:0:8:0 | | serialid | None | | software_load | 20.06 | | subfunctions | worker,lowlatency | | task | Configuration failure, threshold reached, Lock/Unlock to retry | | tboot | false | | ttys_dcd | None | | updated_at | 2020-06-05T14:34:08.504008+00:00 | | uptime | 37323 | | uuid | 9a7b34ac-28c9-4854-a97d-a546d5f59952 | | vim_progress_status | None | +-----------------------+----------------------------------------------------------------+ Test Activity ------------- Sanity Brief Description ----------------- During system install and config, both controller nodes had been installed and unlocked successfully, but compute nodes failed after unlock. Severity -------- Major Steps to Reproduce ------------------ install multi nodes system Expected Behavior ------------------ all compute nodes are available after unlock Actual Behavior ---------------- compute nodes failed after initial unlock Reproducibility --------------- Reproducible 2 of 2 System Configuration -------------------- Multi-node system Lab-name: wcp_8-12 Branch/Pull Time/Commit ----------------------- 2020-06-04_20-00-00 Last Pass --------- 2020-06-03_20-00-00 Timestamp/Logs -------------- [sysadmin@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | compute-0 | worker | unlocked | disabled | failed | | 3 | compute-1 | worker | unlocked | disabled | failed | | 4 | compute-2 | worker | unlocked | disabled | failed | | 5 | controller-1 | controller | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system host-show compute-0 +-----------------------+----------------------------------------------------------------+ | Property | Value | +-----------------------+----------------------------------------------------------------+ | action | none | | administrative | unlocked | | availability | failed | | bm_ip | 2620:10a:a001:a102::134 | | bm_type | dynamic | | bm_username | root | | boot_device | /dev/disk/by-path/pci-0000:18:00.0-scsi-0:0:8:0 | | capabilities | {} | | clock_synchronization | ntp | | config_applied | 6fc32a18-c718-4a2c-b1e5-7c1ba3c65e72 | | config_status | None | | config_target | 6fc32a18-c718-4a2c-b1e5-7c1ba3c65e72 | | console | ttyS0,115200n8 | | created_at | 2020-06-05T03:33:18.372476+00:00 | | device_image_update | None | | hostname | compute-0 | | id | 2 | | install_output | text | | install_state | completed | | install_state_info | None | | inv_state | inventoried | | invprovision | provisioning | | location | {} | | mgmt_ip | face::31f0:b4b7:17d6:b129 | | mgmt_mac | 3c:fd:fe:b1:26:d8 | | operational | disabled | | personality | worker | | reboot_needed | False | | reserved | False | | rootfs_device | /dev/disk/by-path/pci-0000:18:00.0-scsi-0:0:8:0 | | serialid | None | | software_load | 20.06 | | subfunctions | worker,lowlatency | | task | Configuration failure, threshold reached, Lock/Unlock to retry | | tboot | false | | ttys_dcd | None | | updated_at | 2020-06-05T14:34:08.504008+00:00 | | uptime | 37323 | | uuid | 9a7b34ac-28c9-4854-a97d-a546d5f59952 | | vim_progress_status | None | +-----------------------+----------------------------------------------------------------+ Test Activity ------------- Sanity
2020-06-05 16:49:58 Ghada Khalil starlingx: assignee Bob Church (rchurch)
2020-06-05 17:01:06 Ghada Khalil tags stx.retestneeded stx. stx.retestneeded
2020-06-05 17:01:40 Ghada Khalil tags stx. stx.retestneeded stx.4.0 stx.containers stx.metal stx.retestneeded
2020-06-05 17:02:11 Ghada Khalil starlingx: importance Undecided High
2020-06-05 17:02:24 Ghada Khalil bug added subscriber Daniel Badea
2020-06-05 17:02:28 Ghada Khalil starlingx: status New Triaged
2020-06-05 19:57:04 Bob Church starlingx: status Triaged Fix Released
2020-06-10 15:21:42 Peng Peng tags stx.4.0 stx.containers stx.metal stx.retestneeded stx.4.0 stx.containers stx.metal