live-migration failure leave the port to BUILD state

Bug #1439857 reported by Mathieu Rohon
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Mathieu Rohon
Juno
Fix Released
Undecided
Unassigned
Kilo
Fix Released
Medium
Unassigned

Bug Description

I've set up a lab where live migration can occur in block mode

It seems that if I leave the default config, block live-migration fails;

I can see that the port is left in BUILD state after the failure, but the VM is still running on the source host.

Changed in neutron:
assignee: nobody → Mathieu Rohon (mathieu-rohon)
Changed in neutron:
importance: Undecided → Medium
tags: added: api
Changed in neutron:
status: New → In Progress
Kyle Mestery (mestery)
tags: added: kilo-rc-potential
Kyle Mestery (mestery)
Changed in neutron:
milestone: none → liberty-1
Changed in neutron:
assignee: Mathieu Rohon (mathieu-rohon) → Kyle Mestery (mestery)
Kyle Mestery (mestery)
Changed in neutron:
assignee: Kyle Mestery (mestery) → nobody
assignee: nobody → Mathieu Rohon (mathieu-rohon)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/163178
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9b53b82ce7dad551ebc0f02ff667d5345fb7e139
Submitter: Jenkins
Branch: master

commit 9b53b82ce7dad551ebc0f02ff667d5345fb7e139
Author: mathieu-rohon <email address hidden>
Date: Sat Mar 7 13:30:49 2015 +0100

    ML2: Change port status only when it's bound to the host

    Currently, nothing prevents the port status to be changed to BUILD
    state when get_device_details() is sent by a host that doesn't own
    the port.
    In some cases the port might stay in BUILD state.
    This could happen during a live-migration, or for multi-hosted ports
    such as HA ports.
    This commit allows the port status modification only if the port
    is bound to the host that is asking for it.

    Closes-Bug: #1439857
    Closes-Bug: #1438040
    Closes-Bug: #1416933

    Change-Id: I9b3673f453abbafaaa4f78542fcfebe8dc93f2bb

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/175385

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/175385
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b339391bcb223c0f03d30f36dea47d13adb12c71
Submitter: Jenkins
Branch: stable/kilo

commit b339391bcb223c0f03d30f36dea47d13adb12c71
Author: mathieu-rohon <email address hidden>
Date: Sat Mar 7 13:30:49 2015 +0100

    ML2: Change port status only when it's bound to the host

    Currently, nothing prevents the port status to be changed to BUILD
    state when get_device_details() is sent by a host that doesn't own
    the port.
    In some cases the port might stay in BUILD state.
    This could happen during a live-migration, or for multi-hosted ports
    such as HA ports.
    This commit allows the port status modification only if the port
    is bound to the host that is asking for it.

    Closes-Bug: #1439857
    Closes-Bug: #1438040
    Closes-Bug: #1416933

    Change-Id: I9b3673f453abbafaaa4f78542fcfebe8dc93f2bb
    (cherry picked from commit 9b53b82ce7dad551ebc0f02ff667d5345fb7e139)

Thierry Carrez (ttx)
tags: removed: kilo-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/179286

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (neutron-pecan)

Fix proposed to branch: neutron-pecan
Review: https://review.openstack.org/185072

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/185900

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)
Download full text (11.7 KiB)

Reviewed: https://review.openstack.org/179286
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b8a09568b5bde8b71ad00468c27b757c98d9c0da
Submitter: Jenkins
Branch: master

commit 7260e0e3fc2ea479e80e0962624aca7fd38a1f60
Author: Henry Gessau <email address hidden>
Date: Mon Apr 27 09:59:21 2015 -0400

    Run radvd as root

    During the refactoring of external process management radvd lost
    its root privileges.

    Closes-bug: 1448813

    Change-Id: I84883fe81684afafac9b024282a03f447c8f825a
    (cherry picked from commit a5e54338770fc074e01fa88dbf909ee1af1b66b2)

commit d37e566dcadf8a540eb5f84b668847fa192393a1
Author: Kevin Benton <email address hidden>
Date: Fri Apr 24 00:35:31 2015 -0700

    Don't resync on DHCP agent setup failure

    There are various cases where the DHCP agent will try to
    create a DHCP port for a network and there will be a failure.
    This has primarily been caused by a lack of available IP addresses
    in the allocation pool. Trying to fix all availability corner cases
    on the server side will be very difficult due to race conditions between
    multiple ports being created, the dhcp_agents_per_network parameter, etc.

    This patch just stops the resync attempt on the agent side if a failure
    is caused by an IP address generation problem. Future updates to the subnet
    will cause another attempt so if the tenant does fix the issue they will
    get DHCP service.

    Change-Id: I0896730126d6dca13fe9284b4d812cfb081b6218
    Closes-Bug: #1447883
    (cherry picked from commit db9ac7e0110a0c2ef1b65213317ee8b7f1053ddc)

commit 38211ae67cb76ade85b08c028b6e88bfc867afc9
Author: Ihar Hrachyshka <email address hidden>
Date: Mon Apr 20 17:06:38 2015 +0200

    tests: confirm that _output_hosts_file does not log too often

    I3ad7864eeb2f959549ed356a1e34fa18804395cc didn't include any regression unit
    tests to validate that the method won't ever log too often again,
    reintroducing performance drop in later patches. It didn't play well
    with stable backports of the fix, where context was lost when doing the
    backport, that left the bug unfixed in stable/juno even though the patch
    was merged there [1].

    The patch adds an explicit note in the code that suggests not to add new
    log messages inside the loop to avoid regression, and a unit test was
    added to capture it.

    Once the test is merged in master, it will be proposed for stable/juno
    inclusion, with additional changes that would fix the regression again.

    Related-Bug: #1414218
    Change-Id: I5d43021932d6a994638c348eda277dd8337cf041
    (cherry picked from commit 3b74095a935f6d2027e6bf04cc4aa21f8a1b46f2)

commit 53b3e751f3c7b32bed48c14742d3dd3a1178d00d
Author: Maru Newby <email address hidden>
Date: Thu Apr 9 17:00:57 2015 +0000

    Double functional testing timeout to 180s

    The increase in ovs testing is resulting in job failure due to
    timeouts in test_killed_monitor_respawns. Giving the test more
    time to complete should reduce the failure rate.

    Change-Id: I2ba9b1eb388bfbbebbd6b0f3edb6d5a5ae0bfead
    Closes-Bug: #1442272
    (c...

Thierry Carrez (ttx)
Changed in neutron:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/qos)

Fix proposed to branch: feature/qos
Review: https://review.openstack.org/196097

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (feature/qos)
Download full text (93.9 KiB)

Reviewed: https://review.openstack.org/196097
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1cfed745d54a6ce9cb3dd4e6f454666d9e6676c2
Submitter: Jenkins
Branch: feature/qos

commit ba7d673d1ddd5bfa5aa1be5b26a59e9a8cd78a9f
Author: Kevin Benton <email address hidden>
Date: Thu Jun 25 18:31:38 2015 -0700

    Remove duplicated call to setup_coreplugin

    The test case for vlan_transparent was calling setup_coreplugin
    before calling the super setUp method which already calls
    setup_coreplugin. This was causing duplicate core plugin fixtures
    which resulted in patching the dhcp periodic check twice.

    Change-Id: Ide4efad42748e799d8e9c815480c8ffa94b27b38
    Partial-Bug: #1468998

commit e64062efa3b793f7c4ce4ab9e62918af4f1bfcc9
Author: Kevin Benton <email address hidden>
Date: Thu Jun 25 18:29:37 2015 -0700

    Remove double mock of dhcp agent periodic check

    The test case for the periodic check was patching a target
    that the core plugin fixture already patched out. This removes
    that and exposes the mock from the fixture so the test case
    can reference it.

    Change-Id: I3adee6a875c497e070db4198567b52aa16b81ce8
    Partial-Bug: #1468998

commit 25ae0429a713143d42f626dd59ed4514ba25820c
Author: Kevin Benton <email address hidden>
Date: Thu Jun 25 18:24:10 2015 -0700

    Remove double fanout mock

    The test_mech_driver was duplicating a fanout mock already setup
    in the setUp routine.

    Change-Id: I5b88dff13113d55c72241d3d5025791a76672ac2
    Partial-Bug: #1468998

commit 993771556332d9b6bbf7eb3f0300cf9d8a2cb464
Author: Kevin Benton <email address hidden>
Date: Thu Jun 25 17:55:16 2015 -0700

    Remove double callback manager mocks

    setup_test_registry_instance() in the base test case class gives
    each test its own registry by mocking out the get_callback_manager.
    The L3 agent test cases were duplicating this.

    Partial-Bug: #1468998
    Change-Id: I7356daa846524611e9f92365939e8ad15d1e1cd8

commit 0be1efad93734f11cd63fb3b7bd2983442ce1268
Author: Kevin Benton <email address hidden>
Date: Thu Jun 25 16:57:30 2015 -0700

    Remove ensure_dirs double-patch

    test_spawn_radvd called mock.patch on ensure_dirs after the
    setup method already patched it out. This causes issues when
    mock.patch.stopall() is called because the mocks are stored
    as a set and are unwound in a non-deterministic fashion.[1]
    So some of the time they will be undone correctly, but others
    will leave a monkey-patched in mock, causing the ensure_dir
    test to fail.

    1. http://bugs.python.org/issue21239

    Closes-Bug: #1467908
    Change-Id: I321b5fed71dc73bd19b5099311c6f43640726cd4

commit 0a2238e34e72c17ca8a75e36b1f56e41a3ece74e
Author: Sukhdev Kapur <email address hidden>
Date: Thu Jun 25 15:11:28 2015 -0700

    Fix tenant-id in Arista ML2 driver to support HA router

    When HA router is created, the framework creates a network and does
    not specify the tenant-id. This casuse Arista ML2 driver to fail.
    This patch sets the tenant-id when it is not passed explicitly by
    by the network_create() call from the HA r...

tags: added: in-feature-qos
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/pecan)

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/196701

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (feature/pecan)

Change abandoned by Kyle Mestery (<email address hidden>) on branch: feature/pecan
Review: https://review.openstack.org/196701
Reason: This is lacking the functional fix [1], so I'll propose a new merge commit which includes that one.

[1] https://review.openstack.org/#/c/196711/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/pecan)

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/196920

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/juno)

Reviewed: https://review.openstack.org/185900
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1e6c2edfe987b67af6049b14ea8138a3e922424c
Submitter: Jenkins
Branch: stable/juno

commit 1e6c2edfe987b67af6049b14ea8138a3e922424c
Author: mathieu-rohon <email address hidden>
Date: Wed May 27 11:26:06 2015 +0200

    ML2: Change port status only when it's bound to the host

    Currently, nothing prevents the port status to be changed to BUILD
    state when get_device_details() is sent by a host that doesn't own
    the port.
    In some cases the port might stay in BUILD state.
    This could happen during a live-migration, or for multi-hosted ports
    such as HA ports.
    This commit allows the port status modification only if the port
    is bound to the host that is asking for it.

    (cherry picked from commit 9b53b82ce7dad551ebc0f02ff667d5345fb7e139)

    Closes-Bug: #1439857
    Closes-Bug: #1438040
    Closes-Bug: #1416933

    Change-Id: I9b3673f453abbafaaa4f78542fcfebe8dc93f2bb

tags: added: in-stable-juno
Thierry Carrez (ttx)
Changed in neutron:
milestone: liberty-1 → 7.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.