StarlingX

  1. Bug #1835807
  2. Activity log

Activity log for bug #1835807

Date Who What changed Old value New value Message
2019-07-08 15:58:51 Anujeyan Manokeran bug added bug
2019-07-08 16:00:29 Anujeyan Manokeran description Brief Description ----------------- neutron-l3-agent and neutron-dhcp-agent never recovered after compute-3 was force reboot. Below shows before and after compute got rebooted. Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne network agent list' [2019-07-07 04:53:48,062] 423 DEBUG MainThread ssh.expect :: Output: +--------------------------------------+--------------------+-----------+-------------------+-------+-------+---------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+--------------------+-----------+-------------------+-------+-------+---------------------------+ | 02266b80-8c4e-40b7-a9fa-72d6695a1d35 | Metadata agent | compute-4 | None | :-) | UP | neutron-metadata-agent | | 0398ee2d-75b8-4493-8f2a-76e9efe13946 | DHCP agent | compute-2 | nova | :-) | UP | neutron-dhcp-agent | | 0488ea35-3833-4e95-8c89-57cb3badefbe | L3 agent | compute-3 | nova | :-) | UP | neutron-l3-agent | | 084b585c-3b5a-45df-90be-51f66a1745d1 | NIC Switch agent | compute-2 | None | :-) | UP | neutron-sriov-nic-agent | | 0dc1dd0e-928d-4827-926f-923c3b645b33 | L3 agent | compute-0 | nova | :-) | UP | neutron-l3-agent | | 167b07dc-76ad-44f2-acd1-ae73f977af8d | NIC Switch agent | compute-4 | None | :-) | UP | neutron-sriov-nic-agent | | 1b8f8bb7-a9cd-4bd7-b429-82eb6a57e5ed | Open vSwitch agent | compute-0 | None | :-) | UP | neutron-openvswitch-agent | | 1d09fdbd-b6bf-468f-9162-21e1984cca40 | L3 agent | compute-1 | nova | :-) | UP | neutron-l3-agent | | 2ac2053c-073c-4d75-bbac-6367937b9b7b | Open vSwitch agent | compute-3 | None | :-) | UP | neutron-openvswitch-agent | | 2b3e4e26-7364-4d55-aac8-9e75d5bd517c | NIC Switch agent | compute-3 | None | :-) | UP | neutron-sriov-nic-agent | | 45a5a758-df01-4d74-b197-f60c023c8342 | Open vSwitch agent | compute-2 | None | :-) | UP | neutron-openvswitch-agent | | 4684d7f7-6606-4888-ab83-c97867f81cc1 | DHCP agent | compute-3 | nova | :-) | UP | neutron-dhcp-agent | | 46ad03ef-5656-4911-a1dc-20dd48f0c0ae | NIC Switch agent | compute-0 | None | :-) | UP | neutron-sriov-nic-agent | | 4c4413b2-72ce-4dd0-a609-d35dca96a33c | Metadata agent | compute-0 | None | :-) | UP | neutron-metadata-agent | | 5593bdb5-8fb0-4589-9417-4eea8200af18 | DHCP agent | compute-4 | nova | :-) | UP | neutron-dhcp-agent | | 58290c99-d3b3-43ac-9d32-84e4197d52f4 | L3 agent | compute-2 | nova | :-) | UP | neutron-l3-agent | | 603e727d-0f28-4010-84ab-31fc799f67c1 | Metadata agent | compute-3 | None | :-) | UP | neutron-metadata-agent | | 639282a0-0f1b-4ef5-8352-a3ea90a0b797 | Open vSwitch agent | compute-4 | None | :-) | UP | neutron-openvswitch-agent | | 6e13479a-565d-44a1-8b54-30280342362d | Metadata agent | compute-2 | None | :-) | UP | neutron-metadata-agent | | 79ac04e7-4f7b-4015-be3c-9ff0a411914b | Open vSwitch agent | compute-1 | None | :-) | UP | neutron-openvswitch-agent | | 844028c3-b522-4e12-83dd-093cb35e7b04 | L3 agent | compute-4 | nova | :-) | UP | neutron-l3-agent | | 864fbc99-cf8b-422f-bb7b-2ac942340629 | NIC Switch agent | compute-1 | None | :-) | UP | neutron-sriov-nic-agent | | a647d5ea-012a-4940-8dc4-4db4cd17b619 | DHCP agent | compute-0 | nova | :-) | UP | neutron-dhcp-agent | | ad20d257-e40c-48f8-ae17-088c6cd64b18 | Metadata agent | compute-1 | None | :-) | UP | neutron-metadata-agent | | caad4860-bf10-48b6-bd2a-ec6fb1b10c2a | DHCP agent | compute-1 | nova | :-) | UP | neutron-dhcp-agent | +--------------------------------------+--------------------+-----------+-------------------+-------+-------+---------------------------+ Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-list' [2019-07-07 04:54:57,648] 423 DEBUG MainThread ssh.expect :: Output: +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | compute-0 | worker | unlocked | enabled | available | | 3 | compute-1 | worker | unlocked | enabled | available | | 4 | compute-2 | worker | unlocked | enabled | available | | 5 | compute-3 | worker | unlocked | enabled | available | | 6 | compute-4 | worker | unlocked | enabled | available | | 7 | controller-1 | controller | unlocked | enabled | available | | 8 | storage-0 | storage | unlocked | enabled | available | | 9 | storage-1 | storage | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ controller-1:~$ [2019-07-07 04:54:57,649] 301 DEBUG MainThread ssh.send :: Send 'echo $?' [2019-07-07 04:54:57,751] 423 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2019-07-07 04:54:57,752] 282 DEBUG MainThread system_helper.get_hosts:: Filtered hosts: ['controller-0', 'compute-0', 'compute-1', 'compute-2', 'compute-3', 'compute-4', 'controller-1', 'storage-0', 'storage-1'] [2019-07-07 04:54:57,752] 1534 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for wcp_113_121 [2019-07-07 04:54:57,752] 466 DEBUG MainThread ssh.exec_cmd:: Executing command... [2019-07-07 04:54:57,753] 301 DEBUG MainThread ssh.send :: Send 'whoami' [2019-07-07 04:54:57,857] 423 DEBUG MainThread ssh.expect :: Output: sysadmin controller-1:~$ [2019-07-07 04:54:57,857] 301 DEBUG MainThread ssh.send :: Send '' [2019-07-07 04:54:57,959] 423 DEBUG MainThread ssh.expect :: Output: controller-1:~$ [2019-07-07 04:54:57,960] 1185 INFO MainThread ssh.connect :: Attempt to connect to compute-3 from 128.224.150.45... [2019-07-07 04:54:57,960] 301 DEBUG MainThread ssh.send :: Send '/usr/bin/ssh -o RSAAuthentication=no -o PubkeyAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null sysadmin@compute-3' [2019-07-07 04:54:58,131] 423 DEBUG MainThread ssh.expect :: Output: Warning: Permanently added 'compute-3,192.168.222.103' (ECDSA) to the list of known hosts. sysadmin@compute-3's password: [2019-07-07 04:54:58,131] 301 DEBUG MainThread ssh.send :: Send 'Li69nux*' [2019-07-07 04:54:58,398] 423 DEBUG MainThread ssh.expect :: Output: Last login: Sat Jul 6 23:06:06 2019 from controller-0 /etc/motd.d/00-header: compute-3:~$ [2019-07-07 04:54:58,712] 165 INFO MainThread host_helper.reboot_hosts:: Rebooting compute-3 [2019-07-07 04:54:58,713] 301 DEBUG MainThread ssh.send :: Send 'sudo reboot -f' [2019-07-07 04:54:58,826] 423 DEBUG MainThread ssh.expect :: Output: Password: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-list' [2019-07-07 04:55:40,600] 423 DEBUG MainThread ssh.expect :: Output: +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | compute-0 | worker | unlocked | enabled | available | | 3 | compute-1 | worker | unlocked | enabled | available | | 4 | compute-2 | worker | unlocked | enabled | available | | 5 | compute-3 | worker | unlocked | disabled | offline | | 6 | compute-4 | worker | unlocked | enabled | available | | 7 | controller-1 | controller | unlocked | enabled | available | | 8 | storage-0 | storage | unlocked | enabled | available | | 9 | storage-1 | storage | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ end 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-list' [2019-07-07 05:00:05,427] 423 DEBUG MainThread ssh.expect :: Output: +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | compute-0 | worker | unlocked | enabled | available | | 3 | compute-1 | worker | unlocked | enabled | available | | 4 | compute-2 | worker | unlocked | enabled | available | | 5 | compute-3 | worker | unlocked | enabled | available | | 6 | compute-4 | worker | unlocked | enabled | available | | 7 | controller-1 | controller | unlocked | enabled | available | | 8 | storage-0 | storage | unlocked | enabled | available | | 9 | storage-1 | storage | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list' [2019-07-07 05:00:43,303] 423 DEBUG MainThread ssh.expect :: Output: +----+---------------------+-----------------+-----------------+-------+ | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | +----+---------------------+-----------------+-----------------+-------+ | 4 | compute-1 | QEMU | 192.168.223.77 | up | | 7 | compute-2 | QEMU | 192.168.223.176 | up | | 10 | compute-0 | QEMU | 192.168.223.171 | up | | 13 | compute-3 | QEMU | 192.168.223.237 | up | | 16 | compute-4 | QEMU | 192.168.223.64 | up | +----+---------------------+-----------------+-----------------+-------+ Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne network agent list' [2019-07-07 05:03:19,248] 423 DEBUG MainThread ssh.expect :: Output: +--------------------------------------+--------------------+-----------+-------------------+-------+-------+---------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+--------------------+-----------+-------------------+-------+-------+---------------------------+ | 02266b80-8c4e-40b7-a9fa-72d6695a1d35 | Metadata agent | compute-4 | None | :-) | UP | neutron-metadata-agent | | 0398ee2d-75b8-4493-8f2a-76e9efe13946 | DHCP agent | compute-2 | nova | :-) | UP | neutron-dhcp-agent | | 0488ea35-3833-4e95-8c89-57cb3badefbe | L3 agent | compute-3 | nova | XXX | DOWN | neutron-l3-agent | | 084b585c-3b5a-45df-90be-51f66a1745d1 | NIC Switch agent | compute-2 | None | :-) | UP | neutron-sriov-nic-agent | | 0dc1dd0e-928d-4827-926f-923c3b645b33 | L3 agent | compute-0 | nova | :-) | UP | neutron-l3-agent | | 167b07dc-76ad-44f2-acd1-ae73f977af8d | NIC Switch agent | compute-4 | None | :-) | UP | neutron-sriov-nic-agent | | 1b8f8bb7-a9cd-4bd7-b429-82eb6a57e5ed | Open vSwitch agent | compute-0 | None | :-) | UP | neutron-openvswitch-agent | | 1d09fdbd-b6bf-468f-9162-21e1984cca40 | L3 agent | compute-1 | nova | :-) | UP | neutron-l3-agent | | 2ac2053c-073c-4d75-bbac-6367937b9b7b | Open vSwitch agent | compute-3 | None | :-) | UP | neutron-openvswitch-agent | | 2b3e4e26-7364-4d55-aac8-9e75d5bd517c | NIC Switch agent | compute-3 | None | :-) | UP | neutron-sriov-nic-agent | | 45a5a758-df01-4d74-b197-f60c023c8342 | Open vSwitch agent | compute-2 | None | :-) | UP | neutron-openvswitch-agent | | 4684d7f7-6606-4888-ab83-c97867f81cc1 | DHCP agent | compute-3 | nova | :-) | DOWN | neutron-dhcp-agent | | 46ad03ef-5656-4911-a1dc-20dd48f0c0ae | NIC Switch agent | compute-0 | None | :-) | UP | neutron-sriov-nic-agent | | 4c4413b2-72ce-4dd0-a609-d35dca96a33c | Metadata agent | compute-0 | None | :-) | UP | neutron-metadata-agent | | 5593bdb5-8fb0-4589-9417-4eea8200af18 | DHCP agent | compute-4 | nova | :-) | UP | neutron-dhcp-agent | | 58290c99-d3b3-43ac-9d32-84e4197d52f4 | L3 agent | compute-2 | nova | :-) | UP | neutron-l3-agent | | 603e727d-0f28-4010-84ab-31fc799f67c1 | Metadata agent | compute-3 | None | :-) | UP | neutron-metadata-agent | | 639282a0-0f1b-4ef5-8352-a3ea90a0b797 | Open vSwitch agent | compute-4 | None | :-) | UP | neutron-openvswitch-agent | | 6e13479a-565d-44a1-8b54-30280342362d | Metadata agent | compute-2 | None | :-) | UP | neutron-metadata-agent | | 79ac04e7-4f7b-4015-be3c-9ff0a411914b | Open vSwitch agent | compute-1 | None | :-) | UP | neutron-openvswitch-agent | | 844028c3-b522-4e12-83dd-093cb35e7b04 | L3 agent | compute-4 | nova | :-) | UP | neutron-l3-agent | | 864fbc99-cf8b-422f-bb7b-2ac942340629 | NIC Switch agent | compute-1 | None | :-) | UP | neutron-sriov-nic-agent | | a647d5ea-012a-4940-8dc4-4db4cd17b619 | DHCP agent | compute-0 | nova | :-) | UP | neutron-dhcp-agent | | ad20d257-e40c-48f8-ae17-088c6cd64b18 | Metadata agent | compute-1 | None | :-) | UP | neutron-metadata-agent | | caad4860-bf10-48b6-bd2a-ec6fb1b10c2a | DHCP agent | compute-1 | nova | :-) | UP | neutron-dhcp-agent | +--------------------------------------+--------------------+-----------+-------------------+-------+-------+---------------------------+ controller-1:~$ [2019-07-07 05:03:19,248] 301 DEBUG MainThread ssh.send :: Send 'echo $?' Severity -------- Major Steps to Reproduce ------------------ 1. Verify system health by checking alarm, hosts in available state and network agent list 2. force reboot compute wait for the hosts become available 3. As description says neutron-l3-agent and neutron-dhcp-agent never came up. Expected Behavior ------------------ After force reboot network agent list shows all the binary up. Actual Behavior ---------------- As per description 2 binary’s are not up. Reproducibility --------------- Tried only once in latest load. System Configuration -------------------- storage system Branch/Pull Time/Commit ----------------------- 20190706T013000Z Last Pass --------- Not known Timestamp/Logs -------------- 2019-07-07T04:54:58.000 Test Activity ------------- Regression test Brief Description -----------------   neutron-l3-agent and neutron-dhcp-agent was down and never recovered after compute-3 was force rebooted. Below shows before it was up and after compute got rebooted it went down. Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne network agent list' [2019-07-07 04:53:48,062] 423 DEBUG MainThread ssh.expect :: Output: +--------------------------------------+--------------------+-----------+-------------------+-------+-------+---------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+--------------------+-----------+-------------------+-------+-------+---------------------------+ | 02266b80-8c4e-40b7-a9fa-72d6695a1d35 | Metadata agent | compute-4 | None | :-) | UP | neutron-metadata-agent | | 0398ee2d-75b8-4493-8f2a-76e9efe13946 | DHCP agent | compute-2 | nova | :-) | UP | neutron-dhcp-agent | | 0488ea35-3833-4e95-8c89-57cb3badefbe | L3 agent | compute-3 | nova | :-) | UP | neutron-l3-agent | | 084b585c-3b5a-45df-90be-51f66a1745d1 | NIC Switch agent | compute-2 | None | :-) | UP | neutron-sriov-nic-agent | | 0dc1dd0e-928d-4827-926f-923c3b645b33 | L3 agent | compute-0 | nova | :-) | UP | neutron-l3-agent | | 167b07dc-76ad-44f2-acd1-ae73f977af8d | NIC Switch agent | compute-4 | None | :-) | UP | neutron-sriov-nic-agent | | 1b8f8bb7-a9cd-4bd7-b429-82eb6a57e5ed | Open vSwitch agent | compute-0 | None | :-) | UP | neutron-openvswitch-agent | | 1d09fdbd-b6bf-468f-9162-21e1984cca40 | L3 agent | compute-1 | nova | :-) | UP | neutron-l3-agent | | 2ac2053c-073c-4d75-bbac-6367937b9b7b | Open vSwitch agent | compute-3 | None | :-) | UP | neutron-openvswitch-agent | | 2b3e4e26-7364-4d55-aac8-9e75d5bd517c | NIC Switch agent | compute-3 | None | :-) | UP | neutron-sriov-nic-agent | | 45a5a758-df01-4d74-b197-f60c023c8342 | Open vSwitch agent | compute-2 | None | :-) | UP | neutron-openvswitch-agent | | 4684d7f7-6606-4888-ab83-c97867f81cc1 | DHCP agent | compute-3 | nova | :-) | UP | neutron-dhcp-agent | | 46ad03ef-5656-4911-a1dc-20dd48f0c0ae | NIC Switch agent | compute-0 | None | :-) | UP | neutron-sriov-nic-agent | | 4c4413b2-72ce-4dd0-a609-d35dca96a33c | Metadata agent | compute-0 | None | :-) | UP | neutron-metadata-agent | | 5593bdb5-8fb0-4589-9417-4eea8200af18 | DHCP agent | compute-4 | nova | :-) | UP | neutron-dhcp-agent | | 58290c99-d3b3-43ac-9d32-84e4197d52f4 | L3 agent | compute-2 | nova | :-) | UP | neutron-l3-agent | | 603e727d-0f28-4010-84ab-31fc799f67c1 | Metadata agent | compute-3 | None | :-) | UP | neutron-metadata-agent | | 639282a0-0f1b-4ef5-8352-a3ea90a0b797 | Open vSwitch agent | compute-4 | None | :-) | UP | neutron-openvswitch-agent | | 6e13479a-565d-44a1-8b54-30280342362d | Metadata agent | compute-2 | None | :-) | UP | neutron-metadata-agent | | 79ac04e7-4f7b-4015-be3c-9ff0a411914b | Open vSwitch agent | compute-1 | None | :-) | UP | neutron-openvswitch-agent | | 844028c3-b522-4e12-83dd-093cb35e7b04 | L3 agent | compute-4 | nova | :-) | UP | neutron-l3-agent | | 864fbc99-cf8b-422f-bb7b-2ac942340629 | NIC Switch agent | compute-1 | None | :-) | UP | neutron-sriov-nic-agent | | a647d5ea-012a-4940-8dc4-4db4cd17b619 | DHCP agent | compute-0 | nova | :-) | UP | neutron-dhcp-agent | | ad20d257-e40c-48f8-ae17-088c6cd64b18 | Metadata agent | compute-1 | None | :-) | UP | neutron-metadata-agent | | caad4860-bf10-48b6-bd2a-ec6fb1b10c2a | DHCP agent | compute-1 | nova | :-) | UP | neutron-dhcp-agent | +--------------------------------------+--------------------+-----------+-------------------+-------+-------+---------------------------+  Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-list' [2019-07-07 04:54:57,648] 423 DEBUG MainThread ssh.expect :: Output: +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | compute-0 | worker | unlocked | enabled | available | | 3 | compute-1 | worker | unlocked | enabled | available | | 4 | compute-2 | worker | unlocked | enabled | available | | 5 | compute-3 | worker | unlocked | enabled | available | | 6 | compute-4 | worker | unlocked | enabled | available | | 7 | controller-1 | controller | unlocked | enabled | available | | 8 | storage-0 | storage | unlocked | enabled | available | | 9 | storage-1 | storage | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ controller-1:~$ [2019-07-07 04:54:57,649] 301 DEBUG MainThread ssh.send :: Send 'echo $?' [2019-07-07 04:54:57,751] 423 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2019-07-07 04:54:57,752] 282 DEBUG MainThread system_helper.get_hosts:: Filtered hosts: ['controller-0', 'compute-0', 'compute-1', 'compute-2', 'compute-3', 'compute-4', 'controller-1', 'storage-0', 'storage-1'] [2019-07-07 04:54:57,752] 1534 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for wcp_113_121 [2019-07-07 04:54:57,752] 466 DEBUG MainThread ssh.exec_cmd:: Executing command... [2019-07-07 04:54:57,753] 301 DEBUG MainThread ssh.send :: Send 'whoami' [2019-07-07 04:54:57,857] 423 DEBUG MainThread ssh.expect :: Output: sysadmin controller-1:~$ [2019-07-07 04:54:57,857] 301 DEBUG MainThread ssh.send :: Send '' [2019-07-07 04:54:57,959] 423 DEBUG MainThread ssh.expect :: Output: controller-1:~$ [2019-07-07 04:54:57,960] 1185 INFO MainThread ssh.connect :: Attempt to connect to compute-3 from 128.224.150.45... [2019-07-07 04:54:57,960] 301 DEBUG MainThread ssh.send :: Send '/usr/bin/ssh -o RSAAuthentication=no -o PubkeyAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null sysadmin@compute-3' [2019-07-07 04:54:58,131] 423 DEBUG MainThread ssh.expect :: Output: Warning: Permanently added 'compute-3,192.168.222.103' (ECDSA) to the list of known hosts. sysadmin@compute-3's password: [2019-07-07 04:54:58,131] 301 DEBUG MainThread ssh.send :: Send 'Li69nux*' [2019-07-07 04:54:58,398] 423 DEBUG MainThread ssh.expect :: Output: Last login: Sat Jul 6 23:06:06 2019 from controller-0 /etc/motd.d/00-header: compute-3:~$ [2019-07-07 04:54:58,712] 165 INFO MainThread host_helper.reboot_hosts:: Rebooting compute-3 [2019-07-07 04:54:58,713] 301 DEBUG MainThread ssh.send :: Send 'sudo reboot -f' [2019-07-07 04:54:58,826] 423 DEBUG MainThread ssh.expect :: Output: Password:  Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-list' [2019-07-07 04:55:40,600] 423 DEBUG MainThread ssh.expect :: Output: +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | compute-0 | worker | unlocked | enabled | available | | 3 | compute-1 | worker | unlocked | enabled | available | | 4 | compute-2 | worker | unlocked | enabled | available | | 5 | compute-3 | worker | unlocked | disabled | offline | | 6 | compute-4 | worker | unlocked | enabled | available | | 7 | controller-1 | controller | unlocked | enabled | available | | 8 | storage-0 | storage | unlocked | enabled | available | | 9 | storage-1 | storage | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ end 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-list' [2019-07-07 05:00:05,427] 423 DEBUG MainThread ssh.expect :: Output: +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | compute-0 | worker | unlocked | enabled | available | | 3 | compute-1 | worker | unlocked | enabled | available | | 4 | compute-2 | worker | unlocked | enabled | available | | 5 | compute-3 | worker | unlocked | enabled | available | | 6 | compute-4 | worker | unlocked | enabled | available | | 7 | controller-1 | controller | unlocked | enabled | available | | 8 | storage-0 | storage | unlocked | enabled | available | | 9 | storage-1 | storage | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list' [2019-07-07 05:00:43,303] 423 DEBUG MainThread ssh.expect :: Output: +----+---------------------+-----------------+-----------------+-------+ | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | +----+---------------------+-----------------+-----------------+-------+ | 4 | compute-1 | QEMU | 192.168.223.77 | up | | 7 | compute-2 | QEMU | 192.168.223.176 | up | | 10 | compute-0 | QEMU | 192.168.223.171 | up | | 13 | compute-3 | QEMU | 192.168.223.237 | up | | 16 | compute-4 | QEMU | 192.168.223.64 | up | +----+---------------------+-----------------+-----------------+-------+ Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne network agent list' [2019-07-07 05:03:19,248] 423 DEBUG MainThread ssh.expect :: Output: +--------------------------------------+--------------------+-----------+-------------------+-------+-------+---------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+--------------------+-----------+-------------------+-------+-------+---------------------------+ | 02266b80-8c4e-40b7-a9fa-72d6695a1d35 | Metadata agent | compute-4 | None | :-) | UP | neutron-metadata-agent | | 0398ee2d-75b8-4493-8f2a-76e9efe13946 | DHCP agent | compute-2 | nova | :-) | UP | neutron-dhcp-agent | | 0488ea35-3833-4e95-8c89-57cb3badefbe | L3 agent | compute-3 | nova | XXX | DOWN | neutron-l3-agent | | 084b585c-3b5a-45df-90be-51f66a1745d1 | NIC Switch agent | compute-2 | None | :-) | UP | neutron-sriov-nic-agent | | 0dc1dd0e-928d-4827-926f-923c3b645b33 | L3 agent | compute-0 | nova | :-) | UP | neutron-l3-agent | | 167b07dc-76ad-44f2-acd1-ae73f977af8d | NIC Switch agent | compute-4 | None | :-) | UP | neutron-sriov-nic-agent | | 1b8f8bb7-a9cd-4bd7-b429-82eb6a57e5ed | Open vSwitch agent | compute-0 | None | :-) | UP | neutron-openvswitch-agent | | 1d09fdbd-b6bf-468f-9162-21e1984cca40 | L3 agent | compute-1 | nova | :-) | UP | neutron-l3-agent | | 2ac2053c-073c-4d75-bbac-6367937b9b7b | Open vSwitch agent | compute-3 | None | :-) | UP | neutron-openvswitch-agent | | 2b3e4e26-7364-4d55-aac8-9e75d5bd517c | NIC Switch agent | compute-3 | None | :-) | UP | neutron-sriov-nic-agent | | 45a5a758-df01-4d74-b197-f60c023c8342 | Open vSwitch agent | compute-2 | None | :-) | UP | neutron-openvswitch-agent | | 4684d7f7-6606-4888-ab83-c97867f81cc1 | DHCP agent | compute-3 | nova | :-) | DOWN | neutron-dhcp-agent | | 46ad03ef-5656-4911-a1dc-20dd48f0c0ae | NIC Switch agent | compute-0 | None | :-) | UP | neutron-sriov-nic-agent | | 4c4413b2-72ce-4dd0-a609-d35dca96a33c | Metadata agent | compute-0 | None | :-) | UP | neutron-metadata-agent | | 5593bdb5-8fb0-4589-9417-4eea8200af18 | DHCP agent | compute-4 | nova | :-) | UP | neutron-dhcp-agent | | 58290c99-d3b3-43ac-9d32-84e4197d52f4 | L3 agent | compute-2 | nova | :-) | UP | neutron-l3-agent | | 603e727d-0f28-4010-84ab-31fc799f67c1 | Metadata agent | compute-3 | None | :-) | UP | neutron-metadata-agent | | 639282a0-0f1b-4ef5-8352-a3ea90a0b797 | Open vSwitch agent | compute-4 | None | :-) | UP | neutron-openvswitch-agent | | 6e13479a-565d-44a1-8b54-30280342362d | Metadata agent | compute-2 | None | :-) | UP | neutron-metadata-agent | | 79ac04e7-4f7b-4015-be3c-9ff0a411914b | Open vSwitch agent | compute-1 | None | :-) | UP | neutron-openvswitch-agent | | 844028c3-b522-4e12-83dd-093cb35e7b04 | L3 agent | compute-4 | nova | :-) | UP | neutron-l3-agent | | 864fbc99-cf8b-422f-bb7b-2ac942340629 | NIC Switch agent | compute-1 | None | :-) | UP | neutron-sriov-nic-agent | | a647d5ea-012a-4940-8dc4-4db4cd17b619 | DHCP agent | compute-0 | nova | :-) | UP | neutron-dhcp-agent | | ad20d257-e40c-48f8-ae17-088c6cd64b18 | Metadata agent | compute-1 | None | :-) | UP | neutron-metadata-agent | | caad4860-bf10-48b6-bd2a-ec6fb1b10c2a | DHCP agent | compute-1 | nova | :-) | UP | neutron-dhcp-agent | +--------------------------------------+--------------------+-----------+-------------------+-------+-------+---------------------------+ controller-1:~$ [2019-07-07 05:03:19,248] 301 DEBUG MainThread ssh.send :: Send 'echo $?' Severity -------- Major Steps to Reproduce ------------------ 1. Verify system health by checking alarm, hosts in available state and network agent list 2. force reboot compute wait for the hosts become available 3. As description says neutron-l3-agent and neutron-dhcp-agent never came up. Expected Behavior ------------------ After force reboot network agent list shows all the binary up. Actual Behavior ---------------- As per description 2 binary’s are not up. Reproducibility --------------- Tried only once in latest load. System Configuration -------------------- storage system Branch/Pull Time/Commit ----------------------- 20190706T013000Z Last Pass --------- Not known Timestamp/Logs -------------- 2019-07-07T04:54:58.000 Test Activity ------------- Regression test
2019-07-08 17:12:46 Ghada Khalil tags stx.networking
2019-07-08 17:13:03 Ghada Khalil starlingx: status New Incomplete
2019-07-08 17:14:40 Numan Waheed tags stx.networking stx.networking stx.retestneeded
2019-07-08 17:21:58 Anujeyan Manokeran attachment added collect logs https://bugs.launchpad.net/starlingx/+bug/1835807/+attachment/5275671/+files/ALL_NODES_20190707.080518.tar
2019-07-08 17:43:31 Ghada Khalil tags stx.networking stx.retestneeded stx.2.0 stx.networking stx.nfv stx.retestneeded
2019-07-08 17:43:54 Ghada Khalil starlingx: importance Undecided High
2019-07-08 17:43:57 Ghada Khalil starlingx: status Incomplete Triaged
2019-07-08 17:44:45 Ghada Khalil starlingx: assignee Kevin Smith (kevin.smith.wrs)
2019-07-08 17:45:11 Numan Waheed tags stx.2.0 stx.networking stx.nfv stx.retestneeded stx.2.0 stx.networking stx.nfv stx.regression stx.retestneeded
2019-07-10 21:47:25 Ghada Khalil starlingx: status Triaged Incomplete
2019-07-16 16:07:38 Ghada Khalil starlingx: status Incomplete Invalid
2019-07-18 15:27:58 Ghada Khalil starlingx: status Invalid Confirmed
2019-07-18 15:31:07 Ghada Khalil starlingx: assignee Kevin Smith (kevin.smith.wrs) Joseph Richard (josephrichard)
2019-07-18 21:18:53 Ghada Khalil summary neutron-l3-agent and neutron-dhcp-agent never recovered after force reboot on compute neutron-l3-agent and neutron-dhcp-agent never recovered after force reboot
2019-07-23 17:47:46 Ghada Khalil tags stx.2.0 stx.networking stx.nfv stx.regression stx.retestneeded stx.2.0 stx.networking stx.regression stx.retestneeded
2019-08-27 14:24:03 Bart Wensley bug added subscriber Bart Wensley
2019-10-09 14:58:50 Yosief Gebremariam attachment added SELECT_NODES_20191009.143843.tar https://bugs.launchpad.net/starlingx/+bug/1835807/+attachment/5295983/+files/SELECT_NODES_20191009.143843.tar
2019-11-18 22:53:40 Ghada Khalil starlingx: status Confirmed Incomplete
2019-12-30 19:58:28 Ghada Khalil tags stx.2.0 stx.networking stx.regression stx.retestneeded stx.3.0 stx.networking stx.regression stx.retestneeded
2019-12-30 19:59:07 Ghada Khalil starlingx: status Incomplete Won't Fix
2019-12-30 19:59:44 Ghada Khalil tags stx.3.0 stx.networking stx.regression stx.retestneeded stx.2.0 stx.networking stx.regression stx.retestneeded
2020-01-30 16:29:50 Yang Liu tags stx.2.0 stx.networking stx.regression stx.retestneeded stx.2.0 stx.networking stx.regression