paramiko EOFError in test_security_groups_basic_ops

Bug #1295808 reported by Dolph Mathews
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
neutron
Expired
Low
Unassigned
tempest
Expired
Medium
Unassigned

Bug Description

As seen in http://logs.openstack.org/35/81235/6/check/check-tempest-dsvm-neutron/34d2f0c/console.html

2014-03-20 16:34:48,542 RESP: [404] CaseInsensitiveDict({'date': 'Thu, 20 Mar 2014 16:34:48 GMT', 'content-length': '78', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-73413bde-8be9-445d-8041-a729a5ce7548'})
RESP BODY: {"itemNotFound": {"message": "The resource could not be found.", "code": 404}}
}}}

Traceback (most recent call last):
  File "tempest/test.py", line 121, in wrapper
    return f(self, *func_args, **func_kwargs)
  File "tempest/scenario/test_security_groups_basic_ops.py", line 442, in test_cross_tenant_traffic
    self._verify_mac_addr(self.alt_tenant)
  File "tempest/scenario/test_security_groups_basic_ops.py", line 423, in _verify_mac_addr
    mac_addr = access_point_ssh.get_mac_address()
  File "tempest/common/utils/linux/remote_client.py", line 99, in get_mac_address
    return self.exec_command(cmd)
  File "tempest/common/utils/linux/remote_client.py", line 47, in exec_command
    return self.ssh_client.exec_command(cmd)
  File "tempest/common/ssh.py", line 110, in exec_command
    ssh = self._get_ssh_connection()
  File "tempest/common/ssh.py", line 75, in _get_ssh_connection
    timeout=self.channel_timeout, pkey=self.pkey)
  File "/usr/local/lib/python2.7/dist-packages/paramiko/client.py", line 242, in connect
    t.start_client()
  File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 348, in start_client
    raise e
EOFError

Not a perfect logstash query, but most of these results contain this specific failure: http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRmlsZSBcXFwidGVtcGVzdC9zY2VuYXJpby90ZXN0X3NlY3VyaXR5X2dyb3Vwc19iYXNpY19vcHMucHlcXFwiLCBsaW5lIDQyMywgaW4gX3ZlcmlmeV9tYWNfYWRkclwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiIxNzI4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzk1NDI4NDMzODI1fQ==

This appears to be distinct from a couple other bugs with related titles:

  https://bugs.launchpad.net/swift/+bug/1224001
  https://bugs.launchpad.net/tempest/+bug/1291957

Tags: gate-failure
Revision history for this message
punal patel (punal-patel) wrote :
Revision history for this message
Matt Riedemann (mriedem) wrote :
Revision history for this message
Matt Riedemann (mriedem) wrote :

There are some errors in the neutron server log related to firewalls and nova/neutron event callbacks, but not sure if those are related to this failure.

http://logs.openstack.org/40/105840/1/check/check-tempest-dsvm-neutron/1861848/logs/screen-q-svc.txt.gz?level=TRACE

Revision history for this message
Matt Riedemann (mriedem) wrote :
Revision history for this message
Matt Riedemann (mriedem) wrote :

The FirewallNotFound and nova event error messages show up in too many successful job runs to indicate they are the case of the issue.

Revision history for this message
Matt Riedemann (mriedem) wrote :
Revision history for this message
Matt Riedemann (mriedem) wrote :

The e-r query for this is actually way too open-ended, it's picking up on other paramiko failures that aren't the EOFError.

Revision history for this message
Dolph Mathews (dolph) wrote :

The same failure is possible in other contexts, for example:

tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic[compute,gate,network,smoke]
--------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/test.py", line 128, in wrapper
        return f(self, *func_args, **func_kwargs)
      File "tempest/scenario/test_network_basic_ops.py", line 362, in test_hotplug_nic
        self._check_public_network_connectivity(should_connect=True)
      File "tempest/scenario/test_network_basic_ops.py", line 178, in _check_public_network_connectivity
        self.servers.keys())
      File "tempest/scenario/manager.py", line 1031, in _check_public_network_connectivity
        should_connect=should_connect)
      File "tempest/scenario/manager.py", line 1018, in _check_vm_connectivity
        self.get_remote_client(ip_address, username, private_key)
      File "tempest/scenario/manager.py", line 479, in get_remote_client
        linux_client.validate_authentication()
      File "tempest/common/utils/linux/remote_client.py", line 53, in validate_authentication
        self.ssh_client.test_connection_auth()
      File "tempest/common/ssh.py", line 150, in test_connection_auth
        connection = self._get_ssh_connection()
      File "tempest/common/ssh.py", line 75, in _get_ssh_connection
        timeout=self.channel_timeout, pkey=self.pkey)
      File "/usr/local/lib/python2.7/dist-packages/paramiko/client.py", line 242, in connect
        t.start_client()
      File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 346, in start_client
        raise e
    EOFError

The current e-r query does not account for that.

tags: added: gate-failure
Changed in neutron:
importance: Undecided → Low
Changed in tempest:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Albert (albert-vico) wrote :

Could you elaborate about how to reproduce this?

Revision history for this message
Matt Riedemann (mriedem) wrote :

This isn't hitting anymore on master so marking as incomplete.

Changed in neutron:
status: New → Incomplete
Changed in tempest:
status: Confirmed → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for tempest because there has been no activity for 60 days.]

Changed in tempest:
status: Incomplete → Expired
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for neutron because there has been no activity for 60 days.]

Changed in neutron:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.