Grenade jobs fail during verify with timeout

[Dan Smith]

Grenade jobs are occasionally failing with a timeout while running sync:

2023-10-10 12:33:49.688 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:verify_noapi:245 : timeout 30 ssh -v -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i /opt/stack/save/cinder_key.pem cirros@172.24.5.210 sync
2023-10-10 12:33:49.695 | OpenSSH_8.9p1 Ubuntu-3ubuntu0.4, OpenSSL 3.0.2 15 Mar 2022
2023-10-10 12:33:49.695 | debug1: Reading configuration data /etc/ssh/ssh_config
2023-10-10 12:33:49.695 | debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
2023-10-10 12:33:49.695 | debug1: /etc/ssh/ssh_config line 21: Applying options for *
2023-10-10 12:33:49.696 | debug1: Connecting to 172.24.5.210 [172.24.5.210] port 22.
2023-10-10 12:33:49.704 | debug1: fd 3 clearing O_NONBLOCK
2023-10-10 12:33:49.704 | debug1: Connection established.
2023-10-10 12:33:49.704 | debug1: identity file /opt/stack/save/cinder_key.pem type 2
2023-10-10 12:33:49.704 | debug1: identity file /opt/stack/save/cinder_key.pem-cert type -1
2023-10-10 12:33:49.704 | debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.4
2023-10-10 12:33:49.730 | debug1: Remote protocol version 2.0, remote software version dropbear_2020.81
2023-10-10 12:33:49.730 | debug1: compat_banner: no match: dropbear_2020.81
2023-10-10 12:33:49.730 | debug1: Authenticating to 172.24.5.210:22 as 'cirros'
2023-10-10 12:33:49.730 | debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
2023-10-10 12:33:49.730 | debug1: SSH2_MSG_KEXINIT sent
2023-10-10 12:33:49.730 | debug1: SSH2_MSG_KEXINIT received
2023-10-10 12:33:49.730 | debug1: kex: algorithm: curve25519-sha256
2023-10-10 12:33:49.730 | debug1: kex: host key algorithm: ssh-ed25519
2023-10-10 12:33:49.730 | debug1: kex: server->client cipher: <email address hidden> MAC: <implicit> compression: none
2023-10-10 12:33:49.730 | debug1: kex: client->server cipher: <email address hidden> MAC: <implicit> compression: none
2023-10-10 12:33:49.733 | debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
2023-10-10 12:33:49.801 | debug1: SSH2_MSG_KEX_ECDH_REPLY received
2023-10-10 12:33:49.801 | debug1: Server host key: ssh-ed25519 SHA256:xYF3/NWzL2emqPReb7/I97TimxDel9q1Zn9diUoVi3c
2023-10-10 12:33:49.801 | debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
2023-10-10 12:33:49.801 | Warning: Permanently added '172.24.5.210' (ED25519) to the list of known hosts.
2023-10-10 12:33:49.810 | debug1: rekey out after 134217728 blocks
2023-10-10 12:33:49.811 | debug1: SSH2_MSG_NEWKEYS sent
2023-10-10 12:33:49.811 | debug1: expecting SSH2_MSG_NEWKEYS
2023-10-10 12:33:49.811 | debug1: SSH2_MSG_NEWKEYS received
2023-10-10 12:33:49.811 | debug1: rekey in after 134217728 blocks
2023-10-10 12:33:49.811 | debug1: Will attempt key: /opt/stack/save/cinder_key.pem ECDSA SHA256:sVZ90mbZniy5dibywsKv1qPUtSmkxdpmJuRmxPnJWrk explicit
2023-10-10 12:33:49.811 | debug1: SSH2_MSG_EXT_INFO received
2023-10-10 12:33:49.811 | debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,ssh-rsa>
2023-10-10 12:33:49.815 | debug1: SSH2_MSG_SERVICE_ACCEPT received
2023-10-10 12:33:49.819 | debug1: Authentications that can continue: publickey,password
2023-10-10 12:33:49.820 | debug1: Next authentication method: publickey
2023-10-10 12:33:49.820 | debug1: Offering public key: /opt/stack/save/cinder_key.pem ECDSA SHA256:sVZ90mbZniy5dibywsKv1qPUtSmkxdpmJuRmxPnJWrk explicit
2023-10-10 12:33:49.825 | debug1: Server accepts key: /opt/stack/save/cinder_key.pem ECDSA SHA256:sVZ90mbZniy5dibywsKv1qPUtSmkxdpmJuRmxPnJWrk explicit
2023-10-10 12:33:49.873 | Authenticated to 172.24.5.210 ([172.24.5.210]:22) using "publickey".
2023-10-10 12:33:49.874 | debug1: channel 0: new [client-session]
2023-10-10 12:33:49.874 | debug1: Entering interactive session.
2023-10-10 12:33:49.874 | debug1: pledge: network
2023-10-10 12:33:49.877 | debug1: Sending environment.
2023-10-10 12:33:49.878 | debug1: channel 0: setting env LANG = "C.UTF-8"
2023-10-10 12:33:49.878 | debug1: channel 0: setting env LC_ALL = "C.UTF-8"
2023-10-10 12:33:49.878 | debug1: Sending command: sync
2023-10-10 12:34:19.690 | debug1: channel 0: free: client-session, nchannels 1
2023-10-10 12:34:19.690 | Killed by signal 15.
2023-10-10 12:34:19.693 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:verify_noapi:1 : exit_trap
2023-10-10 12:34:19.695 | + /opt/stack/new/grenade/functions:exit_trap:115 : local r=124
2023-10-10 12:34:19.698 | 1 verify_noapi /opt/stack/new/grenade/projects/70_cinder/resources.sh
2023-10-10 12:34:19.698 | 216 verify /opt/stack/new/grenade/projects/70_cinder/resources.sh
2023-10-10 12:34:19.698 | 308 main /opt/stack/new/grenade/projects/70_cinder/resources.sh
2023-10-10 12:34:19.698 | Exit code: 124
2023-10-10 12:34:22.145 | + inc/plugin:resources:123 : die 123 'Failed to run /opt/stack/new/grenade/projects/70_cinder/resources.sh verify pre-upgrade'
2023-10-10 12:34:22.147 | + /opt/stack/new/devstack/functions-common:die:264 : local exitcode=124
2023-10-10 12:34:22.149 | [Call Trace]

If this is close to the edge and that's why it only fails some of the time, we might just need to increase the timeout.