An issue on using Zun with neutron DVR

[hongbin]

This is an issue reported in IRC:

Gaasmann: I have a problem with DVR. In this setup, neutron will create qrouters on compute nodes when needed. With zun, the qrouter is created on a compute only after a second container is started on the same compute node.
Gaasmann: For destruction, it's all right, the qrouter is destroyed after all containers are destroyed

We need to verify if there is any issue on using Zun with Neutron DVR.

[hongbin (hongbin034)]

Note: pending on reproducing steps for triaging.

[Nicolas Haller (gaasmann)]

neutron_l3_agent logs during first container creation.

2018-11-07 21:19:08.748 16 DEBUG neutron.agent.l3.agent [req-d1a06d7f-f6e6-49fd-aea8-4ef0ba23e5df 9c619bb449ad46e28af10087293e6752 66e45724b357411b97ecdec26eccc831 - - -] Got routers updated
 notification :[u'593978d2-0023-4ea8-a878-f7f2283dd35d'] routers_updated /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/l3/agent.py:435
2018-11-07 21:19:08.750 16 DEBUG neutron.agent.l3.agent [-] Starting router update for 593978d2-0023-4ea8-a878-f7f2283dd35d, action None, priority 0 _process_router_update /var/lib/kolla/ven
v/local/lib/python2.7/site-packages/neutron/agent/l3/agent.py:538
2018-11-07 21:19:09.651 16 WARNING neutron.agent.l3.agent [-] Info for router 593978d2-0023-4ea8-a878-f7f2283dd35d was not found. Performing router cleanup
2018-11-07 21:19:09.658 16 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'list'] create_process /var/lib/koll
a/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:92
2018-11-07 21:19:10.785 16 DEBUG neutron.agent.l3.agent [-] Finished a router update for 593978d2-0023-4ea8-a878-f7f2283dd35d _process_router_update /var/lib/kolla/venv/local/lib/python2.7/s
ite-packages/neutron/agent/l3/agent.py:568

[Gaëtan Trellu (goldyfruit)]

Hi,

I'm facing the same issue, namespace appears only after the second container creation.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/623682

[hongbin (hongbin034)]

Folks,

I can reproduce this issue in master branch and here is the proposed fix: https://review.openstack.org/#/c/623682/ . I will propose a backport to stable branches once it is merged in master. If you have a change, could you try the patch if you like, and let me know if it works for you. Thanks.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/623682
Committed: https://git.openstack.org/cgit/openstack/kuryr-libnetwork/commit/?id=7a1e1a4034cab3d36d41f7cbccf4b3da5578a528
Submitter: Zuul
Branch: master

commit 7a1e1a4034cab3d36d41f7cbccf4b3da5578a528
Author: Hongbin Lu <email address hidden>
Date: Sun Dec 9 00:45:29 2018 +0000

    Don't set binding:host_id before port binding

    This is because the neutron DVR scheduler expects both
    'binding:host_id' and 'device_owner' to be changed as an indicator
    to send notification to agent to create the qrouters namespace.
    Right now, kuryr changes 'binding:host_id' and 'device_owner' in
    separated API requests, which confuses the DVR scheduler and
    cause the missing of the notification.

    Change-Id: Ifb22cc50f284dfd46eb7ba1999cd9c2c0b25a2c6
    Closes-Bug: #1802163

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/624559

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/624560

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/624559
Committed: https://git.openstack.org/cgit/openstack/kuryr-libnetwork/commit/?id=7d24d83a2aa3cd6af958e3c4121171f29f5d5b4a
Submitter: Zuul
Branch: stable/rocky

commit 7d24d83a2aa3cd6af958e3c4121171f29f5d5b4a
Author: Hongbin Lu <email address hidden>
Date: Sun Dec 9 00:45:29 2018 +0000

    Don't set binding:host_id before port binding

    This is because the neutron DVR scheduler expects both
    'binding:host_id' and 'device_owner' to be changed as an indicator
    to send notification to agent to create the qrouters namespace.
    Right now, kuryr changes 'binding:host_id' and 'device_owner' in
    separated API requests, which confuses the DVR scheduler and
    cause the missing of the notification.

    Change-Id: Ifb22cc50f284dfd46eb7ba1999cd9c2c0b25a2c6
    Closes-Bug: #1802163
    (cherry picked from commit 7a1e1a4034cab3d36d41f7cbccf4b3da5578a528)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/624560
Committed: https://git.openstack.org/cgit/openstack/kuryr-libnetwork/commit/?id=5ac3a2a15a64276e4e875e3da199cdf2587e7fb3
Submitter: Zuul
Branch: stable/queens

commit 5ac3a2a15a64276e4e875e3da199cdf2587e7fb3
Author: Hongbin Lu <email address hidden>
Date: Sun Dec 9 00:45:29 2018 +0000

    Don't set binding:host_id before port binding

    This is because the neutron DVR scheduler expects both
    'binding:host_id' and 'device_owner' to be changed as an indicator
    to send notification to agent to create the qrouters namespace.
    Right now, kuryr changes 'binding:host_id' and 'device_owner' in
    separated API requests, which confuses the DVR scheduler and
    cause the missing of the notification.

    Change-Id: Ifb22cc50f284dfd46eb7ba1999cd9c2c0b25a2c6
    Closes-Bug: #1802163
    (cherry picked from commit 7a1e1a4034cab3d36d41f7cbccf4b3da5578a528)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kuryr-libnetwork 3.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kuryr-libnetwork 2.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kuryr-libnetwork queens-eol release.