[Partition preservation] Sometimes VM lose network connectivity after rollback of compute node where they were booted

[Sergey Novikov]

Upstream bug: https://bugs.launchpad.net/nova/+bug/1532890

Sometimes VM lose network connectivity after rollback with partition preservation of compute node where it was booted.

Step to reproduce:

1. Deploy cluster (1 controller + 2 compute, neutron VLAN)
2. Make sure that cluster is operational. For this run OSTF tests
3. Boot VM and assign to it proper security rules for network connectivity
4. Ping VM
5. Enable partition preservation for compute node (exclude partition for OS) where VM is booted:
    - download disk.yaml 'fuel node --node <NODE_ID> --disk --download'
    - edit downloaded disk.yaml. Change flag 'keep_data' on true for partition, exclude partition for OS
    - upload disk.yaml 'fuel node --node <NODE_ID> --disk --upload'
6. Rollback compute node:
    - fuel node --node-id <NODE_ID> --provision
    - fuel node --node-id <NODE_ID> --deploy
7. Ping VM

Expected result:
VM is accessible by ICMP

Actual result:
VM is running and accessible by VNC, but VM is not accessible by ICMP

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2014.2.2-7.0"
  api: "1.0"
  build_number: "44"
  build_id: "2015-07-16_10-46-35"
  nailgun_sha: "bb8b7ce29abf258e4a73d227f78bb07a56bc5199"
  python-fuelclient_sha: "fe14335efc3c800162b72d1e333e5179314e2ff4"
  fuel-agent_sha: "22b566f9161daab93bd3a3324307332e01925bb3"
  astute_sha: "3cc1bb6b2a7cc2b437c6c6ffc6366653b08749bf"
  fuel-library_sha: "36a02ee002f7071470d388089bfa95b14482d5bd"
  fuel-ostf_sha: "5b5ec07f15b1d15e452366e838c4c830010ee795"
  fuelmain_sha: "533d1722016da9774fb406f86d35942e5870ed73"

[Sergey Novikov (snovikov)]

The small research was done: OVS don't create veth pair for connection of br-int and bridge which was created for VM.

For example,

root@node-3:~# ovs-vsctl show
ec7e0e5e-27f1-495f-8132-f75c6143f8ec
    Bridge br-int
        fail_mode: secure
        Port "qvo39a97be3-88"
            tag: 1
            Interface "qvo39a97be3-88"
        Port int-br-prv
            Interface int-br-prv
                type: patch
                options: {peer=phy-br-prv}
        Port br-int
            Interface br-int
                type: internal
        Port "qvob3dbdaaa-24"
            tag: 1
            Interface "qvob3dbdaaa-24"
    Bridge br-prv
        Port phy-br-prv
            Interface phy-br-prv
                type: patch
                options: {peer=int-br-prv}
        Port "p_br-prv-0"
            Interface "p_br-prv-0"
                type: internal
        Port br-prv
            Interface br-prv
                type: internal
    ovs_version: "2.3.1"
root@node-3:~# brctl show
bridge name bridge id STP enabled interfaces
br-fw-admin 8000.640ac0d516d8 no eth0
br-mgmt 8000.64474cf31a9b no eth2
       p_br-prv-0
br-storage 8000.6456e9078afc no eth4
qbr1ceb0108-7e 8000.da23070d75fb no qvb1ceb0108-7e
       tap1ceb0108-7e
qbr39a97be3-88 8000.5aba96dbdc1d no qvb39a97be3-88
       tap39a97be3-88
qbrb3dbdaaa-24 8000.362497c67f72 no qvbb3dbdaaa-24
       tapb3dbdaaa-24

As you can see, veth pair is absent for bridges br-int and qvb1ceb0108-7e. Bridge "qvb1ceb0108-7e" matches inaccessible VM.

[Ivan Ponomarev (ivanzipfer)]

This is neutron bug . It's not a fuel-libruary related problem.

[Oleg Bondarev (obondarev)]

Actually plugging VM ports into br_int is the responsibility of Nova.
However I'd like to understand the problem more.
Is it correct that there were 3 vms on the compute node and only one of them lost connectivity after rollback?

Also is it expected that VMs must keep connectivity after compute node rollback (a bit concerned about provision and rollback steps)? Which component/feature should ensure that? Is it documented anywhere?

[Sergey Novikov (snovikov)]

> Is it correct that there were 3 vms on the compute node and only one of them lost connectivity after rollback?

Correct.

> Also is it expected that VMs must keep connectivity after compute node rollback (a bit concerned about provision and rollback steps)? Which component/feature should ensure that? Is it documented anywhere?

Possibility of VMs preservation after redeployment of compute node is necessary for MOS upgrade. Also It's reason to increase bug's priority

[Oleg Bondarev (obondarev)]

Logs from compute node show that br-int did not exist at the moment nova-compute service started: http://paste.openstack.org/show/412381/

Attempt to add interface to br-int was done by nova-compute at 12:54:59.262 while neutron agent started at 12:54:59.229

Not sure if it's fuel who creates br-int or neutron agent (neutron agent will create br-int if not exists but probably fuel creates it beforehand) but clearly it should be created before starting nova-compute on the node.

Generally, nova-compute should not be started until neutron ovs agent (fuel?) creates br-int.

Not sure what can be done on neutron side. Reassigning to fuel-lib

[Michael Polenchuk (mpolenchuk)]

https://bugs.launchpad.net/neutron/+bug/1328076

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/211595

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/211595
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=398f05b27875d38bc624ba0a60fea777055e2f2c
Submitter: Jenkins
Branch: master

commit 398f05b27875d38bc624ba0a60fea777055e2f2c
Author: Sergey Kolekonov <email address hidden>
Date: Tue Aug 11 17:20:26 2015 +0300

    Forbid nova-compute to run before Neutron services

    - Don't allow nova compute to start before Neutron services are started and
      integration bridge is created, because in case of non-empty intances database
      nova-compute will try to plug instances' interfaces to the integration bridge
    - Added additional dependency to ubuntu::override tweak to handle a case when
      a service's title is defined

    Change-Id: Ic7fbe29048bafb659d7a9f6f2a3c5c2c5167021f
    Closes-bug: #1477475

[Sergey Novikov (snovikov)]

Verified on:

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "164"
  build_id: "2015-08-12_15-16-47"
  nailgun_sha: "fff6bda090fac15c48b27cca7832a70f8e381101"
  python-fuelclient_sha: "26fc025e0fc5791b62e5ed8561a6016bf8a406bc"
  fuel-agent_sha: "57145b1d8804389304cd04322ba0fb3dc9d30327"
  fuel-nailgun-agent_sha: "e01693992d7a0304d926b922b43f3b747c35964c"
  astute_sha: "e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc"
  fuel-library_sha: "22f848670e49d89fc04aaed4d8efd1b07360cbe7"
  fuel-ostf_sha: "58220583f10fa47f12291488ef77854809c68310"
  fuelmain_sha: "67e5214c0dc5d4ba6da4ae651cef9934800459a9"

[Roman Podoliaka (rpodolyaka)]

Re-opening for 8.0 as there is a new duplicate - https://bugs.launchpad.net/fuel/+bug/1531241

At the same time, I don't think it's Critical, or even High FWIW: the steps to reproduce described (as well as the OSTF test case) imply downtime of data plane - I don't think it's a good idea to do that without disabling the compute host in nova-scheduler and migrating the workloads first, or at least shutting off all the instances running on the compute host before re-installation.

Having said that, we somehow need to define the dependency of nova-compute on an OVS bridge creation done by neutron-openvswitch-agent. Given the fact, it's only needed in OVS deployments, it would probably be easier to do that in nova-compute code, rather than in init scripts.

[Timur Nurlygayanov (tnurlygayanov)]

So, it looks like we need to fix fuel tools for nodes redeployment with the same disks or add the information about the correct workflow for this operation to the documentation first. It is not good way to just add more steps to the tests, because test case is just verification that the product works as expected, but it doesn't and this is the reason of failed test.

For MOS 7.0 it works, so, why we don't want to do the same for MOS 8.0 then? (if we don't have better solution yet).

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Roman Podoliaka <email address hidden>
Review: https://review.fuel-infra.org/16201

[Roman Podoliaka (rpodolyaka)]

I checked the docs (https://docs.mirantis.com/openstack/fuel/fuel-7.0/user-guide.html#partition-preservation and https://docs.mirantis.com/openstack/fuel/fuel-7.0/user-guide.html#node-reinstallation) and looks like our approach to this was way too naive and only worked by chance:

1) we don't describe how to put nodes into "maintenance mode": e.g. for compute nodes on triggering node reinstallation we leave VMs hanging in ACTIVE state, although they will be actually shut off during re-provisioning / re-deployment. Neither we disable nova-compute service in nova-scheduler (nova service-disable), thus, in busy clouds we may end up with half-provisioned VMs interrupted by node reinstallation

2) the way our deployment works we run into multiple different problems on re-deployment with preserving of partitions on the compute node:

- we lose all OVS bridges and don't control over the order of OVS vs nova-compute initialization (order of start in upstart is not enough - we need to make sure OVS / neutron agent *completed* initialisation before we attempt to boot VMs, not that they started)

- the way our deployment works leads to a situation when nova-compute is configured/restarted multiple times. The interesting fact is that Ceph tasks are executed at late stage of deployment, thus, it's possible that nova-compute will start with Ceph not configured for ephemeral after re-deployment, although it *did* use Ceph ephemeral before attempt to reinstall the node. In this case we'll mistakenly rewrite VMs XMLs to look for local disks which never existed

and so on. These are just two problems we've run so far, but looks like there are more to come.

[Roman Podoliaka (rpodolyaka)]

In other words, I think we are fighting with symptoms here (just like we fixed the original problem in 7.0), instead of fixing of the root cause.

I suggest we fix the docs and tests instead.

The procedure for compute nodes must be the following (this is from Nova perspective - we'll need to repeat this exercise for other components as well):

1. Disable scheduling of new VMs on this nova-compute instance:

   nova service-disable <host> nova-compute

2. Shut off all the VMs running on the node to be re-installed:

    for VM in $(nova list --host <host>); do
       nova stop $VM;
    done

or alternatively, (live) migrate VMs first (please see https://docs.google.com/document/d/1nZ4QHfOOqyioOUDgnrO3i7NQkUeiD7qyq0jlrIcuuNU/edit for details)

3. Preserve partitions.

4. Start redeployment.

5. Enable nova-compute service:

   nova service-enable <host> nova-compute

If we didn't (live) migrated VMs:

6. Start VMs:

    for VM in $(nova list --host <host> --status SHUTOFF); do
       nova start $VM
    done

Please see the google doc for detailed description of the maintenance mode and VMs migration.

[Roman Podoliaka (rpodolyaka)]

Disk issue I mentioned in #13: https://bugs.launchpad.net/fuel/+bug/1536115

[Dmitriy Kruglov (dkruglov)]

The initial issue is still (periodically) reproduced if the procedure described in comment #14 is followed (stopping VM, w/o migration):
1 Deploy a cluster (3 controllers + 1 compute, neutron VLAN)
2 Create an OS instance and assign to it proper security rules for network connectivity
3 Ensure that the instance is pingable
4 Stop the instance
     $ nova stop <INSTANCE_ID>
5 Disable nova-compute service on the node that hosts the instance
     $ nova service-disable <HOSTNAME> nova-compute
6 Enable partition preservation for compute node (excluding OS partition) that hosts the test instance:
    - download disk.yaml
             $ fuel node --node <NODE_ID> --disk --download
    - edit downloaded disk.yaml. Set 'keep_data' flag to true for partitions, excluding OS partition
    - upload disk.yaml
             $ fuel node --node <NODE_ID> --disk --upload
7 Reinstall the compute node:
    $ fuel node --node-id <NODE_ID> --provision
    $ fuel node --node-id <NODE_ID> --deploy
8 Enable the nova-compute service back
     $ nova service-enable <HOSTNAME> nova-compute
9 Start the instance
     $ nova start <INSTANCE_ID>
10 Ping the instance

Expected result: the instance is accessible via ICMP
Actual result: the instance is up and running, but is not accessible via ICMP

MOS 8.0, build 548.

[Michele Fagan (michelefagan)]

Request to add information. Moved to 9.0

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Roman Podoliaka <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/16201

[Bug Checker Bot (esikachev-l)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

version

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Vadim Rovachev (vrovachev)]

Usually reproduced on patching-ci for 8.0 tests:
https://patching-ci.infra.mirantis.net/view/8.0.swarm/job/8.0.system_test.ubuntu.partition_preservation/7/
https://patching-ci.infra.mirantis.net/view/8.0.swarm/job/8.0.system_test.ubuntu.partition_preservation/8/
https://patching-ci.infra.mirantis.net/view/8.0.swarm/job/8.0.system_test.ubuntu.partition_preservation/9/

Upped status to high for 8.0 milestone

[Roman Podoliaka (rpodolyaka)]

Dmitriy, are you sure you waited for completion of all deployment tasks on the compute node? (the ones that configure OVS and Ceph seem to be executed relatively late)

I gave it another try on 9.0 and still believe the approach described in https://bugs.launchpad.net/fuel/+bug/1477475/comments/13 and https://bugs.launchpad.net/fuel/+bug/1477475/comments/14 is the preferred way, thus, fixing the docs and the way we tackle this to make sure workloads are properly disabled/migrated before doing redeployment of a compute node.

We could use an ugly hack to work around the corner case with an OVS bridge, but, as I mentioned in the comments, there are more similar problems, e.g. the one with Ceph (https://bugs.launchpad.net/fuel/+bug/1536115). I.e. the external orchestration that triggers the redeployment process must disable / enable nova-compute and its workloads (or migrate those).

Docs team, I suggest we update the docs as it's described here - https://bugs.launchpad.net/fuel/+bug/1477475/comments/14

[Vitaly Sedelnik (vsedelnik)]

Removed from 8.0-mu-2 scope as we are not going to have a fix in nova for this issue per above comment