fuel_migration failed with EOFError

[Maksym Strukov]

Scenario:
    0. Setup master node
    1. Configuration releases
    2. Bootstrap slave and make snapshot
    3. Revert bootstrapped nodes
    4. Create Fuel Environment
    5. Add nodes to environment
    6. Run network checker
    7. Deploy environment
    8. Run network checker
    9. Migrate Fuel Master to a compute
    10. Check periodically the status of Fuel Master migration process

Actual:
2016-06-16 01:02:48,710 - ERROR __init__.py:66 -- wait_phrase_in_log raised: EOFError()
Traceback: Traceback (most recent call last):
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.fuel_master_migrate/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.fuel_master_migrate/fuelweb_test/helpers/checkers.py", line 229, in wait_phrase_in_log
    "remote node".format(phrase, log_path))
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 98, in wait
    while not predicate():
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.fuel_master_migrate/fuelweb_test/helpers/checkers.py", line 225, in <lambda>
    lambda: not SSHManager().execute(ip=ip, cmd=cmd)['exit_code'],
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.fuel_master_migrate/fuelweb_test/helpers/ssh_manager.py", line 143, in execute
    return remote.execute(cmd)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 338, in execute
    chan, stdin, stderr, stdout = self.execute_async(command)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 358, in execute_async
    chan = self._ssh.get_transport().open_session()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/transport.py", line 702, in open_session
    timeout=timeout)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/transport.py", line 823, in open_channel
    raise e
EOFError

Env: 9.0-mos-490
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.fuel_master_migrate/140/console

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Dina Belova (dbelova)]

Assigning to Fuel Sustaining team, as I don't see any MOS-related issues here.

[Dmitry Klenov (dklenov)]

Targeting to 9.0-updates as bug repro requires Fuel Master migration, which is not a common case.

[Peter Zhurba (pzhurba)]

According provided logs, main case of failure is network driver crash on compute node:

 25148416 bytes (25 MB) copied, 1.08273 s, 23.2 MB/s
 + ssh 10.109.5.5 'dd of=/dev/vda2 bs=1048576'
 + dd if=/dev/vda2 bs=1048576
 Warning: Permanently added '10.109.5.5' (ECDSA) to the list of known hosts.

Last entries in "var/log/fuel-migrate.log" show us starting of data sysncing.
At the same time in var/log/remote/node-2.test.domain.local/kernel.log we can see crash:

 2016-06-16T01:02:32.009420+00:00 warning: [ 2218.006167] ------------[ cut here ]------------
 2016-06-16T01:02:32.009420+00:00 warning: [ 2218.006181] WARNING: CPU: 0 PID: 20461 at /build/linux-A0LYlV/linux-3.13.0/net/core/dev.c:2223 skb_warn_bad_offload+0xcd/0xda()
 2016-06-16T01:02:32.009420+00:00 warning: [ 2218.006185] e1000: caps=(0x0000000200014b89, 0x0000000000000000) len=10493 data_len=10465 gso_size=1480 gso_type=6 ip_summed=3

This is no first similar error occurrence. Previous one is described here

https://bugs.launchpad.net/fuel/+bug/1576276

Manual tests show us that using virtio drivers in dos.py helps.

[Alisa Tselovalnikova (atselovalnikova)]

This bug was disappeared after the commit https://review.openstack.org/#/c/317340/10 had been added.