FloatingIP is at status: ACTIVE. failed to reach status: DOWN

[Jason Dunsmore]

We've been seeing the following intermittent error for the past week in the Heat gate:

Traceback (most recent call last):
  File "tempest/test.py", line 113, in wrapper
    return f(self, *func_args, **func_kwargs)
  File "tempest/scenario/test_network_basic_ops.py", line 404, in test_network_basic_ops
    msg="after disassociate "
  File "tempest/scenario/test_network_basic_ops.py", line 208, in check_public_network_connectivity
    self.check_floating_ip_status(floating_ip, floatingip_status)
  File "tempest/scenario/manager.py", line 897, in check_floating_ip_status
    st=status))
  File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 406, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 493, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 'DOWN' != u'ACTIVE': FloatingIP: <"FloatingIP" addr="172.24.5.57" id="e697b957-68dd-44df-bd79-30e47b7098fc"> is at status: ACTIVE. failed to reach status: DOWN

Full log:
http://logs.openstack.org/51/308051/4/check/gate-grenade-dsvm-heat/7c9f7e6/console.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/313804

[Kevin Benton (kevinbenton)]

This is killing the heat grenade gate because they test VPNaaS resources.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/313804
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=a65310b7bbc1e48382876c8ac4868e60a0d47ba7
Submitter: Jenkins
Branch: master

commit a65310b7bbc1e48382876c8ac4868e60a0d47ba7
Author: Kevin Benton <email address hidden>
Date: Fri May 6 16:43:34 2016 -0700

    Add grenade upgrade hooks to stop agent

    This adds grenade upgrade hooks so the agent can actually
    be stopped during an upgrade.

    This is necessary now that vpnaas runs in a devstack plugin
    because its agent will now no longer be stopped by the neutron
    plugin inside of grenade.

    Change-Id: I5271f0ef4f857eb5650612829d352a821416487a
    Partial-Bug: #1579235

[Jason Dunsmore (jasondunsmore)]

We're still seeing failures in the Heat gate after this merged. Do you have any other ideas? The Heat gate has been seriously affected by this issue for a week and a half now.

Also, we noticed that https://review.openstack.org/#/c/313792/ just recently merged, so it may not be related after all.

Thanks,
Jason

[Thomas Herve (therve)]

Yeah one question is now that the mitaka patch landed, if we need to backport the fix too.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/315254

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/315254
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=8cddec3bf41477ce0cab435808f08a407d516f36
Submitter: Jenkins
Branch: master

commit 8cddec3bf41477ce0cab435808f08a407d516f36
Author: Kevin Benton <email address hidden>
Date: Wed May 11 00:38:40 2016 -0700

    Fix grenade plugin

    Adds settings file is necessary to register the service.
    Fixes undefined reference in shutdown by copy-paste. :/

    Change-Id: I1c2129d581c00719766f80b802031f04b4a07a2a
    Closes-Bug: #1579235

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/316623

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/316624

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/316623
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=1d48dcd5cfdc097f243f164ef1d591f88c4796ba
Submitter: Jenkins
Branch: stable/mitaka

commit 1d48dcd5cfdc097f243f164ef1d591f88c4796ba
Author: Kevin Benton <email address hidden>
Date: Fri May 6 16:43:34 2016 -0700

    Add grenade upgrade hooks to stop agent

    This adds grenade upgrade hooks so the agent can actually
    be stopped during an upgrade.

    This is necessary now that vpnaas runs in a devstack plugin
    because its agent will now no longer be stopped by the neutron
    plugin inside of grenade.

    Change-Id: I5271f0ef4f857eb5650612829d352a821416487a
    Partial-Bug: #1579235
    (cherry picked from commit a65310b7bbc1e48382876c8ac4868e60a0d47ba7)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/316624
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=caab92d6ff122a38386f74e639f3c315e3737325
Submitter: Jenkins
Branch: stable/mitaka

commit caab92d6ff122a38386f74e639f3c315e3737325
Author: Kevin Benton <email address hidden>
Date: Wed May 11 00:38:40 2016 -0700

    Fix grenade plugin

    Adds settings file is necessary to register the service.
    Fixes undefined reference in shutdown by copy-paste. :/

    Change-Id: I1c2129d581c00719766f80b802031f04b4a07a2a
    Closes-Bug: #1579235
    (cherry picked from commit 8cddec3bf41477ce0cab435808f08a407d516f36)

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron-vpnaas 8.1.1 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron-vpnaas 9.0.0.0b1 development milestone.