Loopback interface inside 'vrouter' namespace is down after deploy on non-primary controllers

[Dennis Dmitriev]

Scenario:
    - Create HA cluster (Ubuntu/IBP/Neutron+vlan)
    - add 3 controllers + ceph-osd
    - add 2 compute + ceph-osd
I guess ceph-osd roles are not necessary in this case, but I haven't tested another scenario yet.
    - deploy cluster

Actual results: 'lo' interfaces are down inside 'vrouter' namespace on node-2 and node-3.

Cluster roles: http://paste.openstack.org/show/199344/

Interfaces in 'vrouter' namespace on primary controller: http://paste.openstack.org/show/199345/

Interfaces in 'vrouter' namespace on other controllers: http://paste.openstack.org/show/199346/

Services that was started inside the 'vrouter' namespace (for example, dnsmasq and ntpd) are inaccessible from the router:

root@node-3:~# ip netns exec vrouter ntpq -pn
127.0.0.1: timed out, nothing received
***Request timed out

[Dennis Dmitriev (ddmitriev)]

After reboot the controller (node-3) , 'lo' interface started successfully:

http://paste.openstack.org/show/199369/

[Oleksiy Molchanov (omolchanov)]

https://review.openstack.org/#/c/171619/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/171619
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d019dfe8eb6768d00eeece5c9c4fd845e397814f
Submitter: Jenkins
Branch: master

commit d019dfe8eb6768d00eeece5c9c4fd845e397814f
Author: Oleksiy Molchanov <email address hidden>
Date: Wed Apr 8 16:17:48 2015 +0300

    OCF for conntrackd add lo

    Change conntrackd OCF script:
     - Add loopback intreface to conntrackd OCF
     - Change hard-coded interface an bridge names to variables

    Closes-Bug: 1441121
    Change-Id: Ibe7bdbbaa059e3a5ff0620cef56c626858378d69

[Anastasia Palkina (apalkina)]

Verified on ISO #338

"build_id": "2015-04-22_22-54-31", "build_number": "338", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-04-22_22-54-31", "build_number": "338", "api": "1.0", "fuel-library_sha": "9b26cb98e7ceb905fe85b744eee2f808f7e4737b", "nailgun_sha": "4972120d5a5d41efe9be2d10b81329adf3258a78", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "2b311b3b82a1e2df1dc3484a0f37e282273cd988", "astute_sha": "5be3506e6a4f8b7f0a858db2e162dabcc07347df", "fuel-ostf_sha": "dcda5a869d1780efc265b6f6b7b8c14c8324a68b", "release": "6.1", "fuelmain_sha": "2f8be90209e709810674f73abb02bb54a700ce5e"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "9b26cb98e7ceb905fe85b744eee2f808f7e4737b", "nailgun_sha": "4972120d5a5d41efe9be2d10b81329adf3258a78", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "2b311b3b82a1e2df1dc3484a0f37e282273cd988", "astute_sha": "5be3506e6a4f8b7f0a858db2e162dabcc07347df", "fuel-ostf_sha": "dcda5a869d1780efc265b6f6b7b8c14c8324a68b", "release": "6.1", "fuelmain_sha": "2f8be90209e709810674f73abb02bb54a700ce5e"

root@node-3:~# ip netns exec vrouter ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
19: vr-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether d2:01:66:ff:95:28 brd ff:ff:ff:ff:ff:ff
21: vr-mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 12:45:81:62:0e:ce brd ff:ff:ff:ff:ff:ff
27: conntrd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 9e:2e:34:20:61:07 brd ff:ff:ff:ff:ff:ff
40: vr-ns: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 82:33:af:74:61:5a brd ff:ff:ff:ff:ff:ff

root@node-4:~# ip netns exec vrouter ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
17: conntrd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 86:b6:6c:0a:79:41 brd ff:ff:ff:ff:ff:ff
23: vr-ns: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether a6:6d:dd:6f:b3:05 brd ff:ff:ff:ff:ff:ff

root@node-5:~# ip netns exec vrouter ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
17: conntrd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 4a:db:16:31:08:7f brd ff:ff:ff:ff:ff:ff
23: vr-ns: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 7a:9b:ba:21:0c:36 brd ff:ff:ff:ff:ff:ff