RDO cloud jobs failing with SSH Error: data could not be sent to remote host \"38.145.32.100\". Make sure this host can be reached over ssh

[chandan kumar]

On All branches OpenStack-check and openstack-periodic 24 hours job got failed with giving POST_FAILURE and with following error:
2018-07-16 04:44:34.735807 | primary | TASK [rdo-kolla-build : Build and push images] *********************************
2018-07-16 06:44:45.050329 | [Zuul] Log Stream did not terminate
2018-07-16 06:44:45.051225 | primary | ERROR
2018-07-16 06:44:45.051414 | primary | {
2018-07-16 06:44:45.051473 | primary | "msg": "SSH Error: data could not be sent to remote host \"38.145.32.100\". Make sure this host can be reached over ssh",
2018-07-16 06:44:45.051522 | primary | "unreachable": true

Below is the list of affected jobs:
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-centos-7-queens-containers-build/8aa51e6/job-output.txt.gz#_2018-07-16_06_44_51_326465

https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-pike-upload/fcd86ab/job-output.txt.gz#_2018-07-16_06_27_54_988307

https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike/85a8656/job-output.txt.gz#_2018-07-16_06_15_49_631042

https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-pike/05e6fad/job-output.txt.gz#_2018-07-16_06_28_39_032920

https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/5b7343d/job-output.txt.gz#_2018-07-16_06_27_29_646653

https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-centos-7-pike-containers-build/1f98165/job-output.txt.gz#_2018-07-16_06_38_52_874246

https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-ocata-upload/1427a55/job-output.txt.gz#_2018-07-16_06_27_59_590250

https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata/8773fa5/job-output.txt.gz#_2018-07-16_06_28_05_790110

https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-ocata/f7cf044/job-output.txt.gz#_2018-07-16_06_15_16_845042

https://logs.rdoproject.org/29/582829/1/openstack-check/legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch/8f820b3/job-output.txt.gz#_2018-07-16_06_52_52_524457

https://logs.rdoproject.org/29/582829/1/openstack-check/legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/bb1201a/job-output.txt.gz#_2018-07-16_07_07_53_048663

2018-07-16 07:08:01.963100 | primary | ERROR
2018-07-16 07:08:01.963319 | primary | {
2018-07-16 07:08:01.963387 | primary | "msg": "SSH Error: data could not be sent to remote host \"38.145.33.250\". Make sure this host can be reached over ssh",
2018-07-16 07:08:01.963477 | primary | "unreachable": true
2018-07-16 07:08:01.963524 | primary | }

https://logs.rdoproject.org/29/582829/1/openstack-check/legacy-tripleo-ci-centos-7-container-to-container-upgrades-master/9b8af30/job-output.txt.gz#_2018-07-16_07_05_23_966980

https://logs.rdoproject.org/29/582829/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/2744331/job-output.txt.gz#_2018-07-16_07_04_54_225882

https://logs.rdoproject.org/29/582829/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/4561bc6/job-output.txt.gz#_2018-07-16_07_06_30_345890

https://logs.rdoproject.org/29/582829/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch/f921b67/job-output.txt.gz#_2018-07-16_07_06_45_433857

https://logs.rdoproject.org/29/582829/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch/b8734d4/job-output.txt.gz#_2018-07-16_06_54_26_484999

https://logs.rdoproject.org/29/582829/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/bb72d2d/job-output.txt.gz#_2018-07-16_06_54_25_255885

It might be an infra issue, It need to checked it is blocking most of the openstack-check jobs.

[Sagi (Sergey) Shnaidman (sshnaidm)]

This is failure for connection from zuul executor to nodepool node.

for example:
https://logs.rdoproject.org/29/582829/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/bb72d2d/job-output.txt.gz#_2018-07-16_06_54_25_255885

"msg": "SSH Error: data could not be sent to remote host \"38.145.33.10\". Make sure this host can be reached over ssh",

and 38.145.33.10 is vm host:
https://logs.rdoproject.org/29/582829/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/bb72d2d/zuul-info/inventory.yaml

Seems like network issue in rdo cloud.

[Tristan Cacqueray (tristan-cacqueray)]

It surprising that 7 jobs failed right after these tasks: "Build and push images" or "run the image build script (direct)"
Then the others jobs also failed after a similar tasks:
2 failed after "build-test-packages : Run DLRN"
1 failed after "Prepare the overcloud images for deploy"
1 failed after "Run the package installation script"
1 failed after "modify-image : Run script on image"
1 failed after "modify-image : Close qcow2 image"

Though there are other non related last task:
1 failed after "Add eth2 interface from eth2.conf"
1 failed after "Run the TripleO-CI VXLAN networking script"

[Sagi (Sergey) Shnaidman (sshnaidm)]

Opened a ticket to rdo cloud ops, number #726

[Paul Belanger (pabelanger)]

I don't believe this to be a networking issue, but issue where nodepool is deleting the nodes behind the back of the running jobs. The correct fix here is to maybe move zookeeper in sf.io into its own dedicated server and back it with SSDs. We should also cluster zookeeper here to avoid single point of failure.

[Tristan Cacqueray (tristan-cacqueray)]

We updated the zookeeper settings to:

tickTime=5000
minSessionTimeout=600000
maxSessionTimeout=1800000

Please let us know if the job failure happens again.

[chandan kumar (chkumar246)]

We are still seeing this issue http://logs.openstack.org/23/582823/4/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e548c5a/job-output.txt.gz#_2018-07-18_07_44_08_936878

2018-07-18 06:09:55.151553 | primary | TASK [overcloud-prep-containers : Prepare for the containerized deployment] ****
2018-07-18 06:09:55.199369 | primary | Wednesday 18 July 2018 06:09:55 +0000 (0:00:02.988) 0:00:36.454 ********
2018-07-18 07:44:08.786736 | [Zuul] Log Stream did not terminate
2018-07-18 07:44:08.912815 | primary | ERROR
2018-07-18 07:44:08.935101 | primary | {
2018-07-18 07:44:08.936878 | primary | "msg": "SSH Error: data could not be sent to remote host \"149.202.166.193\". Make sure this host can be reached over ssh",
2018-07-18 07:44:08.941615 | primary | "unreachable": true
2018-07-18 07:44:08.941741 | primary | }
2018-07-18 07:44:10.313197 |

[Sagi (Sergey) Shnaidman (sshnaidm)]

It happens again, quite a lot.
In gates jobs:

http://logs.openstack.org/23/582823/4/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/6232ced/job-output.txt.gz#_2018-07-18_07_49_30_459692

http://logs.openstack.org/23/582823/4/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e548c5a/job-output.txt.gz#_2018-07-18_07_44_08_936878

[chandan kumar (chkumar246)]

Some other jobs:
http://logs.openstack.org/23/582823/4/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/522bae9/job-output.txt.gz#_2018-07-18_07_29_39_888159

http://logs.openstack.org/23/582823/4/gate/tripleo-ci-centos-7-undercloud-oooq/623f192/job-output.txt.gz#_2018-07-18_07_29_06_414853

http://logs.openstack.org/23/582823/4/gate/tripleo-ci-centos-7-scenario004-multinode-oooq-container/a75827e/job-output.txt.gz#_2018-07-18_06_10_49_756078

It all happened around 06:05 to 06:20

[Matt Riedemann (mriedem)]

I'm seeing this in non-triple jobs:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22SSH%20Error%3A%20data%20could%20not%20be%20sent%20to%20remote%20host%5C%22%20AND%20message%3A%5C%22Make%20sure%20this%20host%20can%20be%20reached%20over%20ssh%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

[Matt Riedemann (mriedem)]

Looks like it was a major issue with packethost-us-west-1.

[Sagi (Sergey) Shnaidman (sshnaidm)]

was solved in infra