Error reading SSH protocol banner

[Salvatore Orlando]

This appears similar to bug 1210664 which is now marked as released.

Once parallel testing is enabled (running with the patches for blueprint neutron-parallel-testing), this error appears frequently.
One example here: http://logs.openstack.org/20/57420/40/experimental/check-tempest-dsvm-neutron-isolated-parallel/40bee04/console.html.gz

Please note that the manifestation of this error is similar to bug 1253896, but the error is different, and possibly the root cause as well, as it seems the connection on port 22 is established successfully.

[Nachi Ueno (nati-ueno)]

I found error in the metadata server
http://logs.openstack.org/20/57420/40/experimental/check-tempest-dsvm-neutron-isolated-parallel/40bee04/logs/screen-q-meta.txt.gz#_2013-12-27_23_18_24_134

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/65849

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/65849
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=4fb7ce6bcfd5d7a3f4571cfe594456ec85e5f32e
Submitter: Jenkins
Branch: master

commit 4fb7ce6bcfd5d7a3f4571cfe594456ec85e5f32e
Author: Nachi Ueno <email address hidden>
Date: Thu Jan 9 18:29:34 2014 -0800

    log console_output when test_network_basic_ops failed

    For trouble shooting network connectibility check,
    This commit add console_output log

    Change-Id: I77826970d397b12ee5945a7ed9d68f1de863b9bc
    Related-bug:1265495

[Matt Riedemann (mriedem)]

Can we get an elastic-recheck query patch up to track this one? Doesn't seem like it would be hard to fingerprint.

[Matt Riedemann (mriedem)]

BTW, saw this on stable/havana too:

http://logs.openstack.org/74/70474/1/check/check-tempest-dsvm-neutron/8ffbbfa/console.html

[Matt Riedemann (mriedem)]

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiU1NIRXhjZXB0aW9uOiBFcnJvciByZWFkaW5nIFNTSCBwcm90b2NvbCBiYW5uZXJcIiBBTkQgZmlsZW5hbWU6XCJjb25zb2xlLmh0bWxcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5MTI5MzIxNTc4NX0=

Looks like it blipped on 1/28 but really started spiking on 1/30, it's all check-tempest-dvsm-neutron* jobs.

[Matt Riedemann (mriedem)]

e-r query patch: https://review.openstack.org/#/c/70556/

[Joe Gordon (jogo)]

This is happening for nova-network as well

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiU1NIRXhjZXB0aW9uOiBFcnJvciByZWFkaW5nIFNTSCBwcm90b2NvbCBiYW5uZXJcIiBBTkQgZmlsZW5hbWU6XCJjb25zb2xlLmh0bWxcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5MTI5MzIxNTc4NSwibW9kZSI6InRlcm1zIiwiYW5hbHl6ZV9maWVsZCI6ImJ1aWxkX25hbWUifQ==

[Mark McClain (markmcclain)]

Looking at test failures this looks to be related to kernel faults we have been seeing. For havana/stable we should disable key injection by back porting the changes to devstack stable/havana and possibly nova.

[Salvatore Orlando (salvatore-orlando)]

To confirm that this bug is exacerbated by concurrent operations, out of 60 failures for the full job, 9 were due to this bug.
This means this bug accounts for about 15% of failures in the full job.

[Mark McClain (markmcclain)]

Closing this bug as we have not seen it in the gate in nearly a week. This bug is overly general and we should this occur again we should open more specific bugs instead.