l3ha with L2pop disabled breaks neutron

[Kevin Carter]

when using l3ha the system will fail to build a vm if L2 population is disabled under most circumstances. To resolve this issue the variable `neutron_l2_population` should be set to "true" by default. The current train of thought was that we'd use L3HA by default however due to current differences in the neutron linux bridge agent it seems that is impossible and will require additional upstream work within neutron. In the near term we should re-enable l2 pop by default and effectively disable the built in L3HA.

This issue was reported in the channel by @Ville Vuorinen (IRC: kysse),
see http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2015-12-01.log.html from 18:47 onwards.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/252100

[Assaf Muller (amuller)]

I read through the IRC logs, there's no info to go on there. There's a log with a trace from nova compute, but I don't have the trace from the Neutron server log. Turning on l2pop will not solve any port binding issues I can guarantee that. There's something else going on in that setup and turning on l2pop somehow masked the issue.

Can you re-run tests with L3 HA, LB and l2pop turned *off*, and upload logs when you run in to issues?

[Sean M. Collins (scollins)]

Here's a patch that they were using to test:

https://review.openstack.org/252574

We may be able to get logs from those failed runs

[Kevin Carter (kevin-carter)]

Here are some of the logs from the failure w/in nova:
http://paste.openstack.org/show/480906/

Here's the trace from the LXB agent:
http://sprunge.us/QcVZ

I can generate / test whatever's needed just let me know what need.

[Assaf Muller (amuller)]

From the LB agent log, it looks like you're hitting:
https://bugs.launchpad.net/neutron/+bug/1470584

[Kevin Carter (kevin-carter)]

I put this together to test in the gate w/ l2 pop disabled + l3ha enabled https://review.openstack.org/#/c/253606/

[Assaf Muller (amuller)]

OK, looking at Kevin's job I noticed that the LB agent is logging the same output as in comment 4. After looking at the code it's clear it's crashing without logging anything. I reported a Neutron bug here: https://bugs.launchpad.net/neutron/+bug/1522966.

Meanwhile, the issue is that with the LB agent, you need to configure either l2pop or VXLAN.vxlan_group. If you have both of them off the LB agent won't start.

[Matt Kassawara (ionosphere80)]

Not a neutron bug. Turns out, openstack-ansible configures the "vxlan_group" option with an empty value. Disabling L2 population causes the Linux bridge agent to use multicast. Without a proper value for "vxlan_group", the Linux bridge agent throws a warning and does not operate. This warning should probably become an error that terminates the agent.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/252100
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=00609ea56e09f5dd0488062b64a03420d296e614
Submitter: Jenkins
Branch: master

commit 00609ea56e09f5dd0488062b64a03420d296e614
Author: Kevin Carter <email address hidden>
Date: Tue Dec 1 17:35:06 2015 -0600

    Fix neutron issue w/ l2pop

    This change resolves an issue where neutron is not able to bind to
    a given port because l2 population is disabled and no vxlan multicast
    group has been defined. To resolve this the `neutron_l2_population`
    variable is being defined and set to "False" in the os_neutron defaults
    and the vxlan multicast group will now contain a default value instead
    of an empty string.

    The change also removes the neutron_l2_population checks in the tasks
    and templates because the variable is now being defined.

    Change-Id: Ic2973626d88781bfc67a4275afcf9feffeb63f36
    Closes-Bug: #1521793
    Co-Authored-by: Ville Vuorinen <email address hidden>
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: liberty
Review: https://review.openstack.org/255624

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/255624
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=742eaa2667746af6a6e5dface57dc10960af148d
Submitter: Jenkins
Branch: liberty

commit 742eaa2667746af6a6e5dface57dc10960af148d
Author: Kevin Carter <email address hidden>
Date: Tue Dec 1 17:35:06 2015 -0600

    Fix neutron issue w/ l2pop

    This change resolves an issue where neutron is not able to bind to
    a given port because l2 population is disabled and no vxlan multicast
    group has been defined. To resolve this the `neutron_l2_population`
    variable is being defined and set to "False" in the os_neutron defaults
    and the vxlan multicast group will now contain a default value instead
    of an empty string.

    The change also removes the neutron_l2_population checks in the tasks
    and templates because the variable is now being defined.

    Change-Id: Ic2973626d88781bfc67a4275afcf9feffeb63f36
    Closes-Bug: #1521793
    Co-Authored-by: Ville Vuorinen <email address hidden>
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit 00609ea56e09f5dd0488062b64a03420d296e614)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 12.0.8 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 12.0.9 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 13.0.0 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 13.0.0 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 12.0.11 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/336128

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/336132

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/336132
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_neutron/commit/?id=18a9491ba5e0a89529c865390574e217bc21c76e
Submitter: Jenkins
Branch: master

commit 18a9491ba5e0a89529c865390574e217bc21c76e
Author: Sean M. Collins <email address hidden>
Date: Thu Jun 30 13:32:52 2016 -0400

    Clarify the default for neutron_vxlan_group

    It used to be an empty string, which triggered some serious bugs,
    and was fixed to have a default.

    Change-Id: Iea03142cf03e56428184dea36bad7673c9980e9c
    Related-Bug: #1521793

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/336128
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.