Fuel migration fails while starting data copying: 'ssh: connect to host 172.16.162.47 port 22: No route to host'

[Artem Panchenko]

Fuel version info (7.0 build #301): http://paste.openstack.org/show/474355/

Fuel migration fails on data sync step, because target VM is inaccessible via SSH from bare-metal master node just after reboot:

+ mv -f /root/fuel-migrate.var /root/fuel-migrate.var.old
+ ifup eth0
Determining if ip address 172.16.162.36 is already in use for device eth0...
+ /etc/init.d/network start
Bringing up loopback interface: ESC[60G[ESC[0;32m OK ESC[0;39m]
Bringing up interface eth0: RTNETLINK answers: File exists
ESC[60G[ESC[0;32m OK ESC[0;39m]
+ /etc/init.d/sshd start
Starting sshd: ESC[60G[ESC[0;32m OK ESC[0;39m]
+ initctl start tty TTY=/dev/tty2
tty (/dev/tty2) start/running, process 1221
+ cecho 'You may login from tty2. Switch by Ctr-Alt-F2'
+ echo 'You may login from tty2. Switch by Ctr-Alt-F2'
+ rsync_data
+ dd if=/dev/sda1
+ ssh 172.16.162.47 'dd of=/dev/vda1'
ssh: connect to host 172.16.162.47 port 22: No route to host

Steps to reproduce:

1. Enable STP on all switch ports attached to master/slave nodes
2. Deploy environment with compute nodes using bare-metal servers
3. Run fuel master node migration to VM, for example: `fuel-migrate 172.16.162.42 --fvm_ram=4096`, where '172.16.162.42' is IP of some compute node

Expected result: Fuel master node is successfully migrated to VM

Actual result: migration fails

Migration fails because script is trying to access target VM via SSH just after network interface is turned on and there no re-tries. When STP is enabled on network switch there is could be some delay between bringing interface on (master node) and switching port to 'forwarding' state. So some re-tries or sleep should be added to fuel-migrate script.

Full fuel-migrate.log is attached. Note that it contains logs from few migrations, see the first one to check original error.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/228362

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/228362
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=ceae8aa63634f83f1195bb1c1732cb661a9b1e8e
Submitter: Jenkins
Branch: master

commit ceae8aa63634f83f1195bb1c1732cb661a9b1e8e
Author: Artem Panchenko <email address hidden>
Date: Mon Sep 28 11:56:30 2015 +0300

    Check VM is reachable before data sync

    Add 'wait_connection' function and make sure that target VM is
    accessible before starting data copying via SSH.

    Change-Id: Id87ba5df5e42924cc2669edba573a11f7b68022e
    Closes-bug: #1500343

[Nastya Urlapova (aurlapova)]

Fuel master migration to VM on 1 Controller, 1 Compute on Neutron/VLAN works on swarm against #482:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "482"
  build_id: "482"
  fuel-nailgun_sha: "ae949905142507f2cb446071783731468f34a572"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "481ed135de2cb5060cac3795428625befdd1d814"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "5724477c83cb0aad179484a8f60eca9e026f9715"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "2e6634994aaf6f9035325bbbbaf4df11561760b9"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "6c6b088a3d52dd0eaf43d59f3a3a149c93a07e7e"

[Nastya Urlapova (aurlapova)]

Also we can verify it on baremetal, as I see from description there is some specific.

[Egor Kotko (ykotko)]

ISO #478