Health checks leak floating-ips with Neutron Networking

[Brad Durrow]

Fuel 4.0 HA with Neutron VLAN Networking

{"build_id": "2013-12-27_00-24-14", "ostf_sha": "83ada35fec2664089e07fdc0d34861ae2a4d948a", "build_number": "214", "nailgun_sha": "af1598bcc9faf468d4d9265cc5c51fa8cea53136", "fuelmain_sha": "17eed776b30886851ae0042fa7a30184f5cd8eb6", "astute_sha": "6ce36837882399e0d3bb1ffdb2c3b2d8dcb84b54", "release": "4.0", "fuellib_sha": "eebe07913ee09311c8e7c9231f6785081327dc0e"}

My health checks related to Floating IPs have started failing, here is the debugging I have done so far if anyone feels generous enough to help: http://paste.openstack.org/show/72743/

Looks like I found a health check bug. If I neutron floatingip-delete one of the floating ips; then run the functional health checks a few things happen...

 1) All of the health checks except "Check network connectivity from instance via floating IP" pass. Previously "Check that VM is accessible via floating IP address" would also fail.

 2) The floatingip I deleted is back, but with a new id

 3) Subsequent health checks show both of the checks involving floating IPs fail.

[Vladimir Kuklin (vkuklin)]

Brad, could you please post a diagnostic snapshot as it contains OSTF logs.

[Brad Durrow (l-brad)]

it was faster for me to patch the problem than to submit logs. If you want logs it is pretty easy to replicate the problem just make a neutron deployment that only has a n floating IPs and run the health checks n/2+1 times.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/79591

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/79591
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=14e57ba5474442d00662b3455099cb21a6afa54f
Submitter: Jenkins
Branch: master

commit 14e57ba5474442d00662b3455099cb21a6afa54f
Author: Tatyana Leontovich <email address hidden>
Date: Tue Mar 11 14:23:20 2014 +0000

    Add deletion of floating ip after disassociation

    First problem - we have 2 very simmilar tests
    that differs only by 1 step. In this patch
    this 2 tests merge into 1.

    Second one - during the tests we do not release
    floating ip, we just disassociate it,
    and release only in teardown(
    when all tests from suite were executed).To fix this issue
    add floating ip deleteion as a separate step

    Change-Id: I5cc0a782cdc5914e250ae5176aa8a724b797a934
    Closes-bug: #1288547

[Anastasiia Naboikina (anaboikina)]

Verified on iso
{"build_id": "2014-03-12_13-26-57", "mirantis": "yes", "build_number": "25", "nailgun_sha": "9c02b6aab96814a2311dbb7a7014a1901cddb8f9", "ostf_sha": "41b5c491b11bfad194e6bb17e3fec1ab5fa5a624", "fuelmain_sha": "b4fb821cceeb100241a99b36ff7a5523b19093a7", "astute_sha": "9f0a2a3135ace2e8d9bf7a4c5f661b8326ace285", "release": "5.0", "fuellib_sha": "48ced96753378e49883cdade8957160ef1b29899"}

[Tatyanka (tatyana-leontovich)]

set in progress status according fix should be back-ported in 4.1.1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/83088

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/83088
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=cb71838f1cc572b3223e01a0c24bc694b0b72537
Submitter: Jenkins
Branch: stable/4.1

commit cb71838f1cc572b3223e01a0c24bc694b0b72537
Author: Tatyana Leontovich <email address hidden>
Date: Tue Mar 11 14:23:20 2014 +0000

    4.1.1 Add deletion of floating ip

    First problem - we have 2 very simmilar tests
    that differs only by 1 step. In this patch
    this 2 tests merge into 1.

    Second one - during the tests we do not release
    floating ip, we just disassociate it,
    and release only in teardown(
    when all tests from suite were executed).To fix this issue
    add floating ip deleteion as a separate step

    Change-Id: I5cc0a782cdc5914e250ae5176aa8a724b797a934
    Closes-bug: #1288547

[Anastasiia Naboikina (anaboikina)]

Verified on iso 4.1 265