60_nova/resources.sh:106:ping_check_public fails intermittently

[Matt Riedemann]

http://logs.openstack.org/12/186112/17/gate/gate-grenade-dsvm/4da364e/logs/grenade.sh.txt.gz#_2015-06-09_22_42_15_929

2015-06-09 22:42:13.960 | --- 172.24.5.1 ping statistics ---
2015-06-09 22:42:13.960 | 1 packets transmitted, 0 received, 100% packet loss, time 0ms
2015-06-09 22:42:13.960 |
2015-06-09 22:42:15.929 | + [[ True = \T\r\u\e ]]
2015-06-09 22:42:15.929 | + die 67 '[Fail] Couldn'\''t ping server'
2015-06-09 22:42:15.929 | + local exitcode=0
2015-06-09 22:42:15.929 | [Call Trace]
2015-06-09 22:42:15.929 | /opt/stack/new/grenade/projects/60_nova/resources.sh:134:verify
2015-06-09 22:42:15.929 | /opt/stack/new/grenade/projects/60_nova/resources.sh:101:verify_noapi
2015-06-09 22:42:15.929 | /opt/stack/new/grenade/projects/60_nova/resources.sh:106:ping_check_public
2015-06-09 22:42:15.929 | /opt/stack/new/grenade/functions:67:die
2015-06-09 22:42:15.931 | [ERROR] /opt/stack/new/grenade/functions:67 [Fail] Couldn't ping server
2015-06-09 22:42:16.933 | 1 die /opt/stack/old/devstack/functions-common
2015-06-09 22:42:16.933 | 67 ping_check_public /opt/stack/new/grenade/functions
2015-06-09 22:42:16.933 | 106 verify_noapi /opt/stack/new/grenade/projects/60_nova/resources.sh
2015-06-09 22:42:16.933 | 101 verify /opt/stack/new/grenade/projects/60_nova/resources.sh
2015-06-09 22:42:16.933 | 134 main /opt/stack/new/grenade/projects/60_nova/resources.sh
2015-06-09 22:42:16.933 | Exit code: 1
2015-06-09 22:42:16.961 | World dumping... see /opt/stack/old/worlddump-2015-06-09-224216.txt for details
2015-06-09 22:42:26.139 | [Call Trace]
2015-06-09 22:42:26.139 | ./grenade.sh:250:resources
2015-06-09 22:42:26.139 | /opt/stack/new/grenade/inc/plugin:82:die
2015-06-09 22:42:26.141 | [ERROR] /opt/stack/new/grenade/inc/plugin:82 Failed to run /opt/stack/new/grenade/projects/60_nova/resources.sh verify

I wonder if there is a race in setting up security groups.

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiW0ZhaWxdIENvdWxkbid0IHBpbmcgc2VydmVyXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6ImN1c3RvbSIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJmcm9tIjoiMjAxNS0wNS0yN1QwMDozMDoxNiswMDowMCIsInRvIjoiMjAxNS0wNi0xMFQwMDozMDoxNiswMDowMCIsInVzZXJfaW50ZXJ2YWwiOiIwIn0sInN0YW1wIjoxNDMzODk2MjUwNTAyfQ==

This hits in nova-network and neutron grenade jobs.

[Matt Riedemann (mriedem)]

Better query:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiZGllXCIgQU5EIG1lc3NhZ2U6XCInW0ZhaWxdIENvdWxkbidcXCcndCBwaW5nIHNlcnZlcidcIiBBTkQgdGFnczpcImdyZW5hZGUuc2gudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJmcm9tIjoiMjAxNS0wNS0yN1QwMDozMDoxNiswMDowMCIsInRvIjoiMjAxNS0wNi0xMFQwMDozMDoxNiswMDowMCIsInVzZXJfaW50ZXJ2YWwiOiIwIn0sInN0YW1wIjoxNDM4MTgyMTY5Mjk5LCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9

[Sean Dague (sdague)]

This grenade bug was last updated over 180 days ago, as grenade
is a fast moving project and we'd like to get the tracker down to
currently actionable bugs, this is getting marked as Invalid. If the
issue still exists, please feel free to reopen it.

[Dr. Jens Harbott (j-harbott)]

New match: http://logs.openstack.org/81/495781/3/check/gate-grenade-dsvm-neutron-dvr-multinode-ubuntu-xenial/7ff7e3c/

[Lee Yarwood (lyarwood)]

https://zuul.opendev.org/t/openstack/build/d867a3f0a61049769a3059c24537d1bb/log/controller/logs/grenade.sh_log.txt#964

2020-10-28 07:29:33.385 | + /opt/stack/new/grenade/functions:ping_check_public:66 : [[ True = \T\r\u\e ]]
2020-10-28 07:29:33.388 | + /opt/stack/new/grenade/functions:ping_check_public:67 : die 67 '[Fail] Couldn'\''t ping server'
2020-10-28 07:29:33.390 | + /opt/stack/new/devstack/functions-common:die:198 : local exitcode=0
2020-10-28 07:29:33.392 | [Call Trace]
2020-10-28 07:29:33.392 | /opt/stack/new/grenade/projects/60_nova/resources.sh:217:create
2020-10-28 07:29:33.392 | /opt/stack/new/grenade/projects/60_nova/resources.sh:137:ping_check_public
2020-10-28 07:29:33.392 | /opt/stack/new/grenade/functions:67:die
2020-10-28 07:29:33.395 | [ERROR] /opt/stack/new/grenade/functions:67 [Fail] Couldn't ping server

I appreciate that this is an incredibly generic bug but it looks like grenade is hitting this again. I've added Nova and Neutron for some additional visibility and review if anyone has time.

[Balazs Gibizer (balazs-gibizer)]

Unfortunately we don't have the log any more. I looked at this as I saw "failed: route add -net "0.0.0.0/0" gw "10.1.0.1"" failures recently but it searching "[Fail] Couldn't ping server" in logstash turns up nothing now. So I guess they are different errors.

[Sylvain Bauza (sylvain-bauza)]

putting it as invalid as we can't really help here, but in case I'm wrong, please punt it again as New.

[Balazs Gibizer (balazs-gibizer)]

It is still happening based on logstash. 5 hits in the last 7 days
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Couldn't%20ping%20server%5C%22

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/grenade/+/874822

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/grenade/+/874822
Committed: https://opendev.org/openstack/grenade/commit/3f9fe2e8fc1fccf0324538274e3b07b3e90b96b9
Submitter: "Zuul (22348)"
Branch: master

commit 3f9fe2e8fc1fccf0324538274e3b07b3e90b96b9
Author: yatinkarel <email address hidden>
Date: Thu Feb 23 12:39:46 2023 +0530

    Bump ping timeout to 60 seconds

    The jobs fails intermittently on ping check
    with current timeout of 30 seconds.

    The testvm can take time to boot and have network
    ready on slow systems and using qemu hypervisor
    and lead to ping timeout.

    With [1] included noticed it taking more than 40 seconds
    to boot testvm, so bumping timeout to 60 seconds will
    help in such cases.

    [1] https://review.opendev.org/c/openstack/grenade/+/874417

    Related-Bug: #1463631
    Related-Bug: #2007357
    Change-Id: Ibdd030e126d508e6ff24cde180c611ada7f24cb3

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/grenade/+/875340

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/grenade/+/875340
Committed: https://opendev.org/openstack/grenade/commit/4102f1b503c3b537fe08b24da235f10d3b2e29bd
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 4102f1b503c3b537fe08b24da235f10d3b2e29bd
Author: yatinkarel <email address hidden>
Date: Thu Feb 23 12:39:46 2023 +0530

    Bump ping timeout to 60 seconds

    The jobs fails intermittently on ping check
    with current timeout of 30 seconds.

    The testvm can take time to boot and have network
    ready on slow systems and using qemu hypervisor
    and lead to ping timeout.

    With [1] included noticed it taking more than 40 seconds
    to boot testvm, so bumping timeout to 60 seconds will
    help in such cases.

    [1] https://review.opendev.org/c/openstack/grenade/+/874417

    Related-Bug: #1463631
    Related-Bug: #2007357
    Change-Id: Ibdd030e126d508e6ff24cde180c611ada7f24cb3
    (cherry picked from commit 3f9fe2e8fc1fccf0324538274e3b07b3e90b96b9)