neutron-dynamic-routing: TypeError: bind() takes 4 positional arguments but 5 were given

[Slawek Kaplonski]

Neutron-dynamic-routing scenario jobs in neutron-tempest-plugin repo are failing all the time since few days.

Example of failure https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_cee/691855/1/check/neutron-tempest-plugin-dynamic-routing/cee3785/testr_results.html.gz

It seems that this happens due to error in service plugin, see neutron server log:

Notify callbacks ['neutron_dynamic_routing.services.bgp.scheduler.bgp_dragent_scheduler.BgpDrAgentSchedulerBase.schedule_bgp_speaker_callback--9223372036854649727'] for bgp_speaker, after_create {{(pid=19640) _notify_loop /usr/local/lib/python3.6/dist-packages/neutron_lib/callbacks/manager.py:193}}
Oct 29 19:11:05.959429 ubuntu-bionic-ovh-bhs1-0012563860 neutron-server[19534]: ERROR neutron_lib.callbacks.manager [None req-c747f1f4-6424-4812-b7e4-cfc5c083fd81 tempest-BgpSpeakerTestJSON-1886159994 tempest-BgpSpeakerTestJSON-1886159994] Error during notification for neutron_dynamic_routing.services.bgp.scheduler.bgp_dragent_scheduler.BgpDrAgentSchedulerBase.schedule_bgp_speaker_callback--9223372036854649727 bgp_speaker, after_create: TypeError: bind() takes 4 positional arguments but 5 were given
Oct 29 19:11:05.959429 ubuntu-bionic-ovh-bhs1-0012563860 neutron-server[19534]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
Oct 29 19:11:05.959429 ubuntu-bionic-ovh-bhs1-0012563860 neutron-server[19534]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.6/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
Oct 29 19:11:05.959429 ubuntu-bionic-ovh-bhs1-0012563860 neutron-server[19534]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs)
Oct 29 19:11:05.959429 ubuntu-bionic-ovh-bhs1-0012563860 neutron-server[19534]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron-dynamic-routing/neutron_dynamic_routing/services/bgp/scheduler/bgp_dragent_scheduler.py", line 202, in schedule_bgp_speaker_callback
Oct 29 19:11:05.959429 ubuntu-bionic-ovh-bhs1-0012563860 neutron-server[19534]: ERROR neutron_lib.callbacks.manager plugin.schedule_bgp_speaker(ctx, payload['bgp_speaker'])
Oct 29 19:11:05.959429 ubuntu-bionic-ovh-bhs1-0012563860 neutron-server[19534]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron-dynamic-routing/neutron_dynamic_routing/db/bgp_dragentscheduler_db.py", line 100, in schedule_bgp_speaker
Oct 29 19:11:05.959429 ubuntu-bionic-ovh-bhs1-0012563860 neutron-server[19534]: ERROR neutron_lib.callbacks.manager created_bgp_speaker)
Oct 29 19:11:05.959429 ubuntu-bionic-ovh-bhs1-0012563860 neutron-server[19534]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/scheduler/base_scheduler.py", line 55, in schedule
Oct 29 19:11:05.959429 ubuntu-bionic-ovh-bhs1-0012563860 neutron-server[19534]: ERROR neutron_lib.callbacks.manager context, chosen_agents, resource['id'], force_scheduling)
Oct 29 19:11:05.959429 ubuntu-bionic-ovh-bhs1-0012563860 neutron-server[19534]: ERROR neutron_lib.callbacks.manager TypeError: bind() takes 4 positional arguments but 5 were given
Oct 29 19:11:05.959429 ubuntu-bionic-ovh-bhs1-0012563860 neutron-server[19534]: ERROR neutron_lib.callbacks.manager

It is caused by https://review.opendev.org/#/c/288271/ which we merged recently and revert of this patch is already proposed in https://review.opendev.org/#/c/691710/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/692372

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692372
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=d5832366707241872d4717045b3d70c0e4cbed8d
Submitter: Zuul
Branch: master

commit d5832366707241872d4717045b3d70c0e4cbed8d
Author: Slawek Kaplonski <email address hidden>
Date: Thu Oct 31 14:38:04 2019 +0100

    Switch neutron-dynamic-routing job to be non-voting temporaray

    This job is now broken due to [1]. Revert for this patch is already
    proposed in [2] but as this block all patches in neutron-tempest-plugin
    repo, and there are in queue patches which should improve stability of
    other neutron scenario jobs, lets not wait until [2] will be merged and
    quickly unblock gates now.

    [1] https://review.opendev.org/#/c/288271/
    [2] https://review.opendev.org/#/c/691710/

    Change-Id: I3951fa12a3b2c840ba2a11f22a298e7926aef649
    Related-Bug: #1850626

[Goutham Pacha Ravi (gouthamr)]

manila has three Zuul jobs [1][2] that set up a bgp speaker to do route advertisements to dynamically created tempest test tenant networks. This bug is observed on all three of those jobs:

Steps to reproduce:

1) public_gateway_ipv6=$(openstack subnet show ipv6-public-subnet -c gateway_ip -f value)
2) neutron --debug bgp-speaker-create --ip-version 6 --local-as 100 bgpspeaker
   neutron --debug bgp-speaker-network-add bgpspeaker $PUBLIC_NETWORK_NAME
   neutron --debug bgp-peer-create --peer-ip $public_gateway_ipv6 --remote-as 200 bgppeer
   neutron --debug bgp-speaker-peer-add bgpspeaker bgppeer

Log trace in neutron server log:

Nov 08 23:20:56.266635 ubuntu-bionic-inap-mtl01-0012732936 neutron-server[14101]: DEBUG neutron.api.v2.base [None req-8e095d65-0b34-48b0-bb63-1e067f0c4969 admin admin] Request body: {'bgp_speaker': {'local_as': '100', 'ip_version': 6, 'name': 'bgpspeaker'}} {{(pid=14476) prepare_request_body /opt/stack/new/neutron/neutron/api/v2/base.py:719}}
Nov 08 23:20:56.267073 ubuntu-bionic-inap-mtl01-0012732936 neutron-server[14101]: DEBUG neutron.api.v2.base [None req-8e095d65-0b34-48b0-bb63-1e067f0c4969 admin admin] Unknown quota resources ['bgp_speaker']. {{(pid=14476) _create /opt/stack/new/neutron/neutron/api/v2/base.py:490}}
Nov 08 23:20:56.286015 ubuntu-bionic-inap-mtl01-0012732936 neutron-server[14101]: DEBUG neutron_lib.callbacks.manager [None req-8e095d65-0b34-48b0-bb63-1e067f0c4969 admin admin] Notify callbacks ['neutron_dynamic_routing.services.bgp.scheduler.bgp_dragent_scheduler.BgpDrAgentSchedulerBase.schedule_bgp_speaker_callback--9223372036854306101'] for bgp_speaker, after_create {{(pid=14476) _notify_loop /usr/local/lib/python3.6/dist-packages/neutron_lib/callbacks/manager.py:193}}
Nov 08 23:20:56.297311 ubuntu-bionic-inap-mtl01-0012732936 neutron-server[14101]: ERROR neutron_lib.callbacks.manager [None req-8e095d65-0b34-48b0-bb63-1e067f0c4969 admin admin] Error during notification for neutron_dynamic_routing.services.bgp.scheduler.bgp_dragent_scheduler.BgpDrAgentSchedulerBase.schedule_bgp_speaker_callback--9223372036854306101 bgp_speaker, after_create: TypeError: bind() takes 4 positional arguments but 5 were given
Nov 08 23:20:56.297311 ubuntu-bionic-inap-mtl01-0012732936 neutron-server[14101]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
Nov 08 23:20:56.297311 ubuntu-bionic-inap-mtl01-0012732936 neutron-server[14101]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.6/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
Nov 08 23:20:56.297311 ubuntu-bionic-inap-mtl01-0012732936 neutron-server[14101]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs)
Nov 08 23:20:56.297311 ubuntu-bionic-inap-mtl01-0012732936 neutron-server[14101]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron-dynamic-routing/neutron_dynamic_routing/services/bgp/scheduler/bgp_dragent_scheduler.py", line 202, in schedule_bgp_speaker_callback
Nov 08 23:20:56.297311 ubuntu-bionic-inap-mtl01-0012732936 neutron-server[14101]: ERROR neutron_lib.callbacks.manager plugin.schedule_bgp_speaker(ctx, payload['bgp_speaker'])
No...

[Dr. Jens Harbott (j-harbott)]

The patch that causes this failure hasn't been reverted, so we likely need to adopt all consumers of the changed code. This isn't easily possible because in tests, neutron is consumed from a published release, so it will have to be a three-step-process. see https://review.opendev.org/693688

[Goutham Pacha Ravi (gouthamr)]

> The patch that causes this failure hasn't been reverted, so we likely need to adopt all consumers > of the changed code. This isn't easily possible because in tests, neutron is consumed from a published release, so it will have to be a three-step-process. see https://review.opendev.org/693688

Thanks, https://review.opendev.org/693688 does fix routing to tenant networks in the manila tests (https://review.opendev.org/#/c/693596/). However, our test jobs are running neutron and neutron-dynamic-routing both from trunk/master. Is this because of a released version of neutron-lib?

[Slawek Kaplonski (slaweq)]

It is addressed with https://review.opendev.org/#/c/693688/1 so should be fixed soon

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/697999

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/701373

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/701373

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697999
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=de2a3f243e2631a61737a937a570e698c490ae2b
Submitter: Zuul
Branch: master

commit de2a3f243e2631a61737a937a570e698c490ae2b
Author: Slawek Kaplonski <email address hidden>
Date: Mon Dec 9 12:04:36 2019 +0000

    Revert "Switch neutron-dynamic-routing job to be non-voting temporaray"

    This reverts commit d5832366707241872d4717045b3d70c0e4cbed8d.

    Change-Id: Icc36f5c3e7c8f94d30efd6445106f7381ef64ddd
    Related-Bug: #1850626