[Reduced footprint] After node with virt,compute is rebooted, VM with controller receives new MAC

[Ksenia Svechnikova]

VERSION:
  feature_groups:
    - mirantis
    - advanced
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "260"
  build_id: "260"
  nailgun_sha: "3de0f32fe9e09fbeee8a695d738f31fe2347e55a"
  python-fuelclient_sha: "9643fa07f1290071511066804f962f62fe27b512"
  fuel-agent_sha: "082a47bf014002e515001be05f99040437281a2d"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "53c86cba593ddbac776ce5a3360240274c20738c"
  fuel-library_sha: "e055af9dee6fbaf84cb8e86a9ebff9b25b07792e"
  fuel-ostf_sha: "582a81ccaa1e439a3aec4b8b8f6994735de840f4"
  fuelmain_sha: "994bb9a8a2a3c4ab1f16e57b479d052abe45f921"

Scenario:

  1. Deploy Tun env
  2. Add 3 compute,virt nodes
  3. Add one VMs' config on each compute,virt node
  4. Lauch VMs
  5. Add 3 controllers on KVM
  6. Deploy cluster
  7. Forse reboot virt,compute node

Actual result:

  1. The VM is running:

    root@node-3:~# virsh list --all
     Id Name State
    ----------------------------------------------------
     2 1_vm running

  2. Fuel didn't see it:

    id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
    ---|--------|------------------|---------|--------------|-------------------|---------------|---------------|--------|---------
    13 | ready | Untitled (0d:c3) | 2 | 172.16.40.83 | 52:54:00:63:0d:c3 | controller | | False | 2
    14 | ready | Untitled (43:92) | 2 | 172.16.40.82 | 52:54:00:84:43:92 | controller | | True | 2
    3 | ready | Untitled (8d:a0) | 2 | 172.16.40.73 | 0c:c4:7a:17:8d:a0 | compute, virt | | True | 2
    5 | ready | Untitled (00:e0) | 2 | 172.16.40.76 | 0c:c4:7a:15:00:e0 | compute, virt | | True | 2
    4 | ready | Untitled (01:3c) | 2 | 172.16.40.75 | 0c:c4:7a:15:01:3c | compute, virt | | True | 2
    12 | ready | Untitled (2d:ef) | 2 | 172.16.40.81 | 52:54:00:dc:2d:ef | controller | | True | 2

  3. Services are down

root@node-14:~# nova-manage service list
Binary Host Zone Status State Updated_At
nova-consoleauth node-12.domain.tld internal enabled :-) 2015-09-02 12:26:45
nova-scheduler node-12.domain.tld internal enabled :-) 2015-09-02 12:26:45
nova-conductor node-12.domain.tld internal enabled :-) 2015-09-02 12:26:17
nova-cert node-12.domain.tld internal enabled :-) 2015-09-02 12:26:15
nova-consoleauth node-14.domain.tld internal enabled :-) 2015-09-02 12:26:14
nova-scheduler node-14.domain.tld internal enabled :-) 2015-09-02 12:26:14
nova-conductor node-14.domain.tld internal enabled :-) 2015-09-02 12:26:18
nova-cert node-14.domain.tld internal enabled :-) 2015-09-02 12:26:25
nova-consoleauth node-13.domain.tld internal enabled XXX 2015-09-02 06:28:25
nova-scheduler node-13.domain.tld internal enabled XXX 2015-09-02 06:28:26
nova-conductor node-13.domain.tld internal enabled XXX 2015-09-02 06:28:26
nova-cert node-13.domain.tld internal enabled XXX 2015-09-02 06:28:37
nova-compute node-5.domain.tld nova enabled :-) 2015-09-02 12:26:25
nova-compute node-4.domain.tld nova enabled :-) 2015-09-02 12:26:42
nova-compute node-3.domain.tld nova enabled :-) 2015-09-02 12:26:20

   4. node-13 is stopped in Corosync:

root@node-14:~# crm status
Last updated: Wed Sep 2 12:27:09 2015
Last change: Tue Sep 1 22:27:04 2015
Stack: corosync
Current DC: node-12.domain.tld (12) - partition with quorum
Version: 1.1.12-561c4cf
3 Nodes configured
43 Resources configured

Online: [ node-12.domain.tld node-14.domain.tld ]
OFFLINE: [ node-13.domain.tld ]

 Clone Set: clone_p_vrouter [p_vrouter]
     Started: [ node-12.domain.tld node-14.domain.tld ]
 vip__management (ocf::fuel:ns_IPaddr2): Started node-12.domain.tld
 vip__vrouter_pub (ocf::fuel:ns_IPaddr2): Started node-12.domain.tld
 vip__vrouter (ocf::fuel:ns_IPaddr2): Started node-12.domain.tld
 vip__public (ocf::fuel:ns_IPaddr2): Started node-14.domain.tld
 Master/Slave Set: master_p_conntrackd [p_conntrackd]
     Masters: [ node-12.domain.tld ]
     Slaves: [ node-14.domain.tld ]
 Clone Set: clone_p_haproxy [p_haproxy]
     Started: [ node-12.domain.tld node-14.domain.tld ]
 Clone Set: clone_p_mysql [p_mysql]
     Started: [ node-12.domain.tld node-14.domain.tld ]
 Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
     Masters: [ node-12.domain.tld ]
     Slaves: [ node-14.domain.tld ]
 Clone Set: clone_p_dns [p_dns]
     Started: [ node-12.domain.tld node-14.domain.tld ]
 Clone Set: clone_p_heat-engine [p_heat-engine]
     Started: [ node-12.domain.tld node-14.domain.tld ]
 Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
     Started: [ node-12.domain.tld node-14.domain.tld ]
 Clone Set: clone_p_neutron-dhcp-agent [p_neutron-dhcp-agent]
     Started: [ node-12.domain.tld node-14.domain.tld ]
 Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
     Started: [ node-12.domain.tld node-14.domain.tld ]
 Clone Set: clone_p_neutron-l3-agent [p_neutron-l3-agent]
     Started: [ node-12.domain.tld node-14.domain.tld ]
 Clone Set: clone_p_ntp [p_ntp]
     Started: [ node-12.domain.tld node-14.domain.tld ]
 Clone Set: clone_ping_vip__public [ping_vip__public]
     Started: [ node-12.domain.tld node-14.domain.tld

MACs on VMs' interfaces are updated after compute host was rebooted and the node lose connection

[slava valyavskiy (slava-val-al)]

Hi, I just tested it on lab. Problem occurred cause we have no mac specified for virtual machine. So, to have static mac after the reboot we should define virtual machine using our template, then make `virsh dumpxml > /etc/libirt/autostart/${vm_name}` to have new node's xml with generated macs.

[slava valyavskiy (slava-val-al)]

When I said "we have no mac specified for virtual machine" I meant that we have no mac specified in vm's xml configuration.

[Ivan Ponomarev (ivanzipfer)]

This bug block node reinstallation using partition preservation feature

[Vladimir Kuklin (vkuklin)]

btw, https://review.openstack.org/#/c/219758/

[Ksenia Svechnikova (kdemina)]

Verify the fix at ISO#267
TUN env: 3xcompute,virt node with one controller on each node

Steps:

1. Reboot compute with virtual controller
2. Wait 5-7 min
3. Check the MAC was not changed
4. Check the state of rebooted controller (ssh, fuel node, crm stat, nova)
5. Run OSTF