Description:
------------
The tripleo-ci-centos-8-standalone-validation-common fails the Tempest TestNetworkBasicOps test.
Failure seems to be stochastic in nature as the job passes in an overwhelming number of cases.
Log snip:
---------
2021-07-09 18:12:57.991 690215 ERROR tempest.lib.common.utils.linux.remote_client [-] (TestNetworkBasicOps:test_update_instance_port_admin_state) Executing command on 192.168.24.160 failed. Error: Command 'set -eu -o pipefail; PATH=$PATH:/sbin:/usr/sbin; ping -c1 -w1 -s56 10.100.0.11' failed, exit status: 1, stderr:
stdout:
PING 10.100.0.11 (10.100.0.11): 56 data bytes
--- 10.100.0.11 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
: neutron_tempest_plugin.common.utils.SSHExecCommandFailed: Command 'set -eu -o pipefail; PATH=$PATH:/sbin:/usr/sbin; ping -c1 -w1 -s56 10.100.0.11' failed, exit status: 1, stderr:
2021-07-09 18:12:57.993 690215 WARNING tempest.scenario.manager [-] Failed to check icmp connectivity for IP 10.100.0.11 via a ssh connection from: 192.168.24.160.: neutron_tempest_plugin.common.utils.SSHExecCommandFailed: Command 'set -eu -o pipefail; PATH=$PATH:/sbin:/usr/sbin; ping -c1 -w1 -s56 10.100.0.11' failed, exit status: 1, stderr:
2021-07-09 18:12:57.993 690215 DEBUG tempest.lib.common.utils.test_utils [-] Call connect_remote returns true in 10.245530 seconds call_until_true /usr/lib/python3.6/site-packages/tempest/lib/common/utils/test_utils.py:118
2021-07-09 18:12:59.498 690215 INFO tempest.lib.common.rest_client [req-2003920f-fc1c-4f38-926e-21bd516bf166 ] Request (TestNetworkBasicOps:test_update_instance_port_admin_state): 200 PUT http://192.168.24.3:9696/v2.0/ports/828b5a2b-c370-41ea-91f2-85e453de0e92 1.504s
2021-07-09 18:12:59.499 690215 DEBUG tempest.lib.common.rest_client [req-2003920f-fc1c-4f38-926e-21bd516bf166 ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
Body: {"port": {"admin_state_up": true}}
Response - Headers: {'content-type': 'application/json', 'content-length': '956', 'x-openstack-request-id': 'req-2003920f-fc1c-4f38-926e-21bd516bf166', 'date': 'Fri, 09 Jul 2021 18:12:59 GMT', 'connection': 'close', 'status': '200', 'content-location': 'http://192.168.24.3:9696/v2.0/ports/828b5a2b-c370-41ea-91f2-85e453de0e92'}
Body: b'{"port":{"id":"828b5a2b-c370-41ea-91f2-85e453de0e92","name":"","network_id":"6a34d9bd-9f81-4e84-b947-e1b10b1e8724","tenant_id":"196f33a0d07548a29af391fd935ee5ba","mac_address":"fa:16:3e:b7:0b:a3","admin_state_up":true,"status":"ACTIVE","device_id":"8050d8eb-21c1-4d4d-9c58-de2a078a06b9","device_owner":"compute:nova","fixed_ips":[{"subnet_id":"d7ed15f2-fd0f-4c08-a758-499b18995827","ip_address":"10.100.0.11"}],"allowed_address_pairs":[],"extra_dhcp_opts":[],"security_groups":["49a8d501-16b8-401e-847d-265f8b7b0f4a"],"description":"","binding:vnic_type":"normal","qos_policy_id":null,"qos_network_policy_id":null,"port_security_enabled":true,"dns_name":"","dns_assignment":[{"ip_address":"10.100.0.11","hostname":"host-10-100-0-11","fqdn":"host-10-100-0-11.openstacklocal."}],"ip_allocation":"immediate","tags":[],"created_at":"2021-07-09T18:11:45Z","updated_at":"2021-07-09T18:12:58Z","revision_number":6,"project_id":"196f33a0d07548a29af391fd935ee5ba"}}' _log_request_full /usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py:456
2021-07-09 18:12:59.706 690215 INFO tempest.lib.common.rest_client [req-e0348baa-a8d7-4c93-8a7d-e7d085cbc85f ] Request (TestNetworkBasicOps:test_update_instance_port_admin_state): 200 GET http://192.168.24.3:9696/v2.0/floatingips/54793d76-60d9-4237-921f-832532d0443c 0.206s
2021-07-09 18:12:59.707 690215 DEBUG tempest.lib.common.rest_client [req-e0348baa-a8d7-4c93-8a7d-e7d085cbc85f ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
Body: None
Response - Headers: {'content-type': 'application/json', 'content-length': '883', 'x-openstack-request-id': 'req-e0348baa-a8d7-4c93-8a7d-e7d085cbc85f', 'date': 'Fri, 09 Jul 2021 18:12:59 GMT', 'connection': 'close', 'status': '200', 'content-location': 'http://192.168.24.3:9696/v2.0/floatingips/54793d76-60d9-4237-921f-832532d0443c'}
Body: b'{"floatingip": {"id": "54793d76-60d9-4237-921f-832532d0443c", "tenant_id": "196f33a0d07548a29af391fd935ee5ba", "floating_ip_address": "192.168.24.159", "floating_network_id": "af7455ce-9ca9-431d-a601-f7b5e7293528", "router_id": "8e343b57-9242-4c3e-9b56-9eee1aa899d3", "port_id": "828b5a2b-c370-41ea-91f2-85e453de0e92", "fixed_ip_address": "10.100.0.11", "status": "ACTIVE", "description": "", "qos_policy_id": null, "port_details": {"name": "", "network_id": "6a34d9bd-9f81-4e84-b947-e1b10b1e8724", "mac_address": "fa:16:3e:b7:0b:a3", "admin_state_up": true, "status": "ACTIVE", "device_id": "8050d8eb-21c1-4d4d-9c58-de2a078a06b9", "device_owner": "compute:nova"}, "dns_domain": "", "dns_name": "", "port_forwardings": [], "tags": [], "created_at": "2021-07-09T18:11:54Z", "updated_at": "2021-07-09T18:11:55Z", "revision_number": 1, "project_id": "196f33a0d07548a29af391fd935ee5ba"}}' _log_request_full /usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py:456
2021-07-09 18:12:59.707 690215 INFO tempest.scenario.manager [-] FloatingIP: {'id': '54793d76-60d9-4237-921f-832532d0443c', 'tenant_id': '196f33a0d07548a29af391fd935ee5ba', 'floating_ip_address': '192.168.24.159', 'floating_network_id': 'af7455ce-9ca9-431d-a601-f7b5e7293528', 'router_id': '8e343b57-9242-4c3e-9b56-9eee1aa899d3', 'port_id': '828b5a2b-c370-41ea-91f2-85e453de0e92', 'fixed_ip_address': '10.100.0.11', 'status': 'ACTIVE', 'description': '', 'qos_policy_id': None, 'port_details': {'name': '', 'network_id': '6a34d9bd-9f81-4e84-b947-e1b10b1e8724', 'mac_address': 'fa:16:3e:b7:0b:a3', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '8050d8eb-21c1-4d4d-9c58-de2a078a06b9', 'device_owner': 'compute:nova'}, 'dns_domain': '', 'dns_name': '', 'port_forwardings': [], 'tags': [], 'created_at': '2021-07-09T18:11:54Z', 'updated_at': '2021-07-09T18:11:55Z', 'revision_number': 1, 'project_id': '196f33a0d07548a29af391fd935ee5ba'} is at status: ACTIVE
2021-07-09 18:12:59.708 690215 DEBUG tempest.lib.common.utils.test_utils [-] Call refresh returns true in 0.207918 seconds call_until_true /usr/lib/python3.6/site-packages/tempest/lib/common/utils/test_utils.py:118
2021-07-09 18:12:59.708 690215 DEBUG tempest.scenario.manager [-] checking network connections to IP 192.168.24.159 with user: cirros check_vm_connectivity /usr/lib/python3.6/site-packages/tempest/scenario/manager.py:941
2021-07-09 18:12:59.709 690215 DEBUG tempest.scenario.manager [-] TestNetworkBasicOps:test_update_instance_port_admin_state begins to ping 192.168.24.159 in 120 sec and the expected result is reachable ping_ip_address /usr/lib/python3.6/site-packages/tempest/scenario/manager.py:904
2021-07-09 18:12:59.733 690215 DEBUG tempest.lib.common.utils.test_utils [-] Call ping returns true in 0.023405 seconds call_until_true /usr/lib/python3.6/site-packages/tempest/lib/common/utils/test_utils.py:118
2021-07-09 18:12:59.734 690215 DEBUG tempest.scenario.manager [-] TestNetworkBasicOps:test_update_instance_port_admin_state finishes ping 192.168.24.159 in 120 sec and the ping result is expected ping_ip_address /usr/lib/python3.6/site-packages/tempest/scenario/manager.py:910
2021-07-09 18:12:59.871 690215 INFO tempest.lib.common.rest_client [req-d2415539-ded3-4ce4-8ac0-445ff717e8a5 ] Request (TestNetworkBasicOps:test_update_instance_port_admin_state): 200 POST http://192.168.24.3:8774/v2.1/servers/8050d8eb-21c1-4d4d-9c58-de2a078a06b9/action 0.136s
2021-07-09 18:12:59.872 690215 DEBUG tempest.lib.common.rest_client [req-d2415539-ded3-4ce4-8ac0-445ff717e8a5 ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-OpenStack-Nova-API-Version': '2.1', 'X-Auth-Token': '<omitted>'}
Body: {"os-getConsoleOutput": {}}
Response - Headers: {'date': 'Fri, 09 Jul 2021 18:12:59 GMT', 'server': 'Apache', 'content-length': '53225', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-d2415539-ded3-4ce4-8ac0-445ff717e8a5', 'x-compute-request-id': 'req-d2415539-ded3-4ce4-8ac0-445ff717e8a5', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://192.168.24.3:8774/v2.1/servers/8050d8eb-21c1-4d4d-9c58-de2a078a06b9/action'}
Body: b'{"output": "[ 0.000000] Linux version 5.3.0-26-generic (buildd@lgw01-amd64-039) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #28~18.04.1-Ubuntu SMP Wed Dec 18 16:40:14 UTC 2019 (Ubuntu 5.3.0-26.28~18.04.1-generic 5.3.13)\\n[ 0.000000] Command line: LABEL=cirros-rootfs ro console=tty1 console=ttyS0\\n[ 0.000000] KERNEL supported cpus:\\n[ 0.000000] Intel GenuineIntel\\n[ 0.000000] AMD AuthenticAMD\\n[ 0.000000] Hygon HygonGenuine\\n[ 0.000000] Centaur CentaurHauls\\n[ 0.000000] zhaoxin Shanghai \\n[ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: \'x87 floating point registers\'\\n[ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: \'SSE registers\'\\n[ 0.000000] x86/fpu: Supporting XSAVE feature 0x008: \'MPX bounds registers\'\\n[ 0.000000] x86/fpu: Supporting XSAVE feature 0x010: \'MPX CSR\'\\n[ 0.000000] x86/fpu: Supporting XSAVE feature 0x200: \'Protection Keys User registers\'\\n[ 0.000000] x86/fpu: xstate_offset[3]: 960, xstate_sizes[3]: 64\\n[ 0.000000] x86/fpu: xstate_offset[4]: 1024, xstate_sizes[4]: 64\\n[ 0.000000] x86/fpu: xstate_offset[9]: 2688, xstate_sizes[9]: 8\\n[ 0.000000] x86/fpu: Enabled xstate features 0x21b, context size is 2696 bytes, using \'standard\' format.\\n[ 0.000000] BIOS-provided physical RAM map:\\n[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable\\n[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved\\n[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved\\n[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000007fdafff] usable\\n[ 0.000000] BIOS-e820: [mem 0x0000000007fdb000-0x0000000007ffffff] reserved\\n[ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved\\n[ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved\\n[ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved\\n[ 0.000000] NX (Execute Disable) protection: active\\n[ 0.000000] SMBIOS 2.8 present.\\n[ 0.000000] DMI: RDO OpenStack Compute/RHEL-AV, BIOS 1.14.0-1.el8s 04/01/2014\\n[ 0.000000] last_pfn = 0x7fdb max_arch_pfn = 0x400000000\\n[ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT \\n[ 0.000000] found SMP MP-table at [mem 0x000f5c10-0x000f5c1f]\\n[ 0.000000] check: Scanning 1 areas for low memory corruption\\n[ 0.000000] Using GB pages for direct mapping\\n[ 0.000000] RAMDISK: [mem 0x0798c000-0x07fcafff]\\n[ 0.000000] ACPI: Early table checksum verification disabled\\n[ 0.000000] ACPI: RSDP 0x00000000000F59A0 000014 (v00 BOCHS )\\n[ 0.000000] ACPI: RSDT 0x0000000007FE201B 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001)\\n[ 0.000000] ACPI: FACP 0x0000000007FE1E4B 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)\\n[ 0.000000] ACPI: DSDT 0x0000000007FE0040 001E0B (v01 BOCHS BXPC 00000001 BXPC 00000001)\\n[ 0.000000] ACPI: FACS 0x0000000007FE0000 000040\\n[ 0.000000] ACPI: APIC 0x0000000007FE1F3F 000078 (v01 BOCHS BXPC 00000001 BXPC 00000001)\\n[ 0.000000] ACPI: MCFG 0x0000000007FE1FB7 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)\\n[ 0.000000] ACPI: WAET 0x0000000007FE1FF3 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)\\n[ 0.000000] No NUMA configuration found\\n[ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000007fdafff]\\n[ 0.000000] NODE_DATA(0) allocated [mem 0x07961000-0x0798bfff]\\n[ 0.000000] Zone ranges:\\n[ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff]\\n[ 0.000000] DMA32 [mem 0x0000000001000000-0x0000000007fdafff]\\n[ 0.000000] Normal empty\\n[ 0.000000] Device empty\\n[ 0.000000] Movable zone start for each node\\n[ 0.000000] Early memory node ranges\\n[ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff]\\n[ 0.000000] node 0: [mem 0x0000000000100000-0x0000000007fdafff]\\n[ 0.000000] Zeroed struct page in unavailable ranges: 98 pages\\n[ 0.000000] Initmem setup node 0 _log_request_full /usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py:456
Logs:
-----
https://zuul.opendev.org/t/openstack/build/2280a11e1f894125bdb1b53a0e9f71c8
https://5a035c34399aaf265f61-e6ddc7380eb470b6a5615640c6b2397b.ssl.cf1.rackcdn.com/799738/1/gate/tripleo-ci-centos-8-standalone-validation-common/2280a11/logs/undercloud/home/zuul/tempest/tempest.log
Duplicate of https:/
same cause https:/