StaticScheduler has not attribute schedule

Bug #1980235 reported by Michal Arbet
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Dr. Jens Harbott

Bug Description

Hi,

When we are trying to use new StaticScheduler proposed in [0], it's failing, please check log error below :

BGP dynamic routing agent 2022-06-29 10:00:49 controller1
2022-06-29 10:07:06.160 25 INFO neutron.wsgi [-] 192.168.205.10 "GET / HTTP/1.1" status: 200 len: 232 time: 0.0052805
2022-06-29 10:07:10.276 34 WARNING neutron.db.agentschedulers_db [req-8d9ed03d-67bd-4685-9697-69ca1e04214a - - - - -] Rescheduling bgp_speaker de024f06-f373-4e5f-8c60-02e2e4e737a9 from agent dd10802c-b5a2-4673-84d2-a26aa09199fc because the agent did not report to the server in the last 150 seconds.
2022-06-29 10:07:10.311 34 ERROR neutron.db.agentschedulers_db [req-8d9ed03d-67bd-4685-9697-69ca1e04214a - - - - -] Exception encountered during bgp_speaker rescheduling.: AttributeError: 'StaticScheduler' object has no attribute 'schedule'
2022-06-29 10:07:10.311 34 ERROR neutron.db.agentschedulers_db Traceback (most recent call last):
2022-06-29 10:07:10.311 34 ERROR neutron.db.agentschedulers_db File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/db/agentschedulers_db.py", line 172, in reschedule_resources_from_down_agents
2022-06-29 10:07:10.311 34 ERROR neutron.db.agentschedulers_db reschedule_resource(context, binding_resource_id)
2022-06-29 10:07:10.311 34 ERROR neutron.db.agentschedulers_db File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron_dynamic_routing/db/bgp_dragentscheduler_db.py", line 168, in reschedule_bgp_speaker
2022-06-29 10:07:10.311 34 ERROR neutron.db.agentschedulers_db self.schedule_bgp_speaker(context, bgp_speaker)
2022-06-29 10:07:10.311 34 ERROR neutron.db.agentschedulers_db File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron_dynamic_routing/db/bgp_dragentscheduler_db.py", line 99, in schedule_bgp_speaker
2022-06-29 10:07:10.311 34 ERROR neutron.db.agentschedulers_db agents = self.bgp_drscheduler.schedule(self, context,
2022-06-29 10:07:10.311 34 ERROR neutron.db.agentschedulers_db AttributeError: 'StaticScheduler' object has no attribute 'schedule'
2022-06-29 10:07:10.311 34 ERROR neutron.db.agentschedulers_db
2022-06-29 10:07:36.434 25 INFO neutron.wsgi [-] 192.168.205.10 "GET / HTTP/1.1" status: 200 len: 232 time: 0.0038111

[0] https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/815265

Could you please fix it ?

Thanks,
Kevko

Changed in neutron:
status: New → Confirmed
tags: added: l3-bgp
Revision history for this message
Michal Arbet (michalarbet) wrote :

Additional info how to reproduce bug :

2 Speakers
2 Dragents

Reproduce :

1.

Configure bgp_drscheduler_driver in neutron.conf (neutron-server) as below :

[DEFAULT]
bgp_drscheduler_driver = neutron_dynamic_routing.services.bgp.scheduler.bgp_dragent_scheduler.StaticScheduler

2.

Add Speaker 1 -> dragent 1
Add Speaker 2 -> dragent 2

3.

Turn off speaker 2 for example.

3. Check neutron server log.

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

How do you "Turn off speaker 2"? Do you talk about stopping the dragent?

Also how are your speakers configured that you need two of them? A standard deployment would have only one speaker scheduled to both agents.

Changed in neutron:
status: Confirmed → New
importance: Undecided → Medium
Revision history for this message
Michal Arbet (michalarbet) wrote :
Download full text (4.2 KiB)

(openstack) bgp speaker list
+--------------------------------------+-------------+----------+------------+
| ID | Name | Local AS | IP Version |
+--------------------------------------+-------------+----------+------------+
| 165c1eac-2bc6-422b-9ff1-ae6c396f482c | bgpspeaker | 123 | 4 |
+--------------------------------------+-------------+----------+------------+
(openstack) bgp dragent list
+--------------------------------------+-------------+-------+-------+
| ID | Host | State | Alive |
+--------------------------------------+-------------+-------+-------+
| 937e9726-1e6a-48cb-abe0-652809cd6bf8 | controller0 | True | :-) |
| dd10802c-b5a2-4673-84d2-a26aa09199fc | controller1 | True | :-) |
+--------------------------------------+-------------+-------+-------+

(openstack) bgp speaker show dragents 165c1eac-2bc6-422b-9ff1-ae6c396f482c
The 'openstack bgp speaker show dragents' CLI is deprecated and will be removed in the future. Use 'openstack bgp dragent list' CLI instead.

(openstack) bgp speaker show dragents de024f06-f373-4e5f-8c60-02e2e4e737a9
The 'openstack bgp speaker show dragents' CLI is deprecated and will be removed in the future. Use 'openstack bgp dragent list' CLI instead.

(openstack) bgp dragent add speaker 937e9726-1e6a-48cb-abe0-652809cd6bf8 165c1eac-2bc6-422b-9ff1-ae6c396f482c
(openstack) bgp dragent add speaker dd10802c-b5a2-4673-84d2-a26aa09199fc 165c1eac-2bc6-422b-9ff1-ae6c396f482c
(openstack) bgp speaker show dragents 165c1eac-2bc6-422b-9ff1-ae6c396f482c
The 'openstack bgp speaker show dragents' CLI is deprecated and will be removed in the future. Use 'openstack bgp dragent list' CLI instead.
+--------------------------------------+-------------+-------+-------+
| ID | Host | State | Alive |
+--------------------------------------+-------------+-------+-------+
| 937e9726-1e6a-48cb-abe0-652809cd6bf8 | controller0 | True | :-) |
| dd10802c-b5a2-4673-84d2-a26aa09199fc | controller1 | True | :-) |
+--------------------------------------+-------------+-------+-------+

So, now I have one speaker scheduled to both controllers as you said this is standard deployment.
Then, if I go to one of controllers and just call docker stop neutron_bgp_dragent. I can see error below :

So, it doesn't matter if you have N speakers or just one. Behaviour is same:

2022-06-29 13:33:48.366 35 WARNING neutron.db.agentschedulers_db [req-be677f3a-c39a-4f43-82a2-cf2184e0412d - - - - -] Rescheduling bgp_speaker 165c1eac-2bc6-422b-9ff1-ae6c396f482c from agent 937e9726-1e6a-48cb-abe0-652809cd6bf8 because the agent did not report to the server in the last 150 seconds.
2022-06-29 13:33:48.419 35 ERROR neutron.db.agentschedulers_db [req-be677f3a-c39a-4f43-82a2-cf2184e0412d - - - - -] Exception encountered during bgp_speaker rescheduling.: AttributeError: 'StaticScheduler' object has no attribute 'schedule'
2022-06-29 13:33:48.419 35 ERROR neutron.db.agentschedulers_db Traceback (most recent call last):
2022-06-29 13:33:48.419 35 ERROR neutron.db.agentschedulers_db File "/var/lib/kolla/venv...

Read more...

Revision history for this message
Michal Arbet (michalarbet) wrote :

And yeah, sorry, when I was talking about turn off speaker , I of course meant dragent.

Revision history for this message
Michal Arbet (michalarbet) wrote :

Any news ?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-dynamic-routing (master)
Changed in neutron:
status: New → In Progress
Changed in neutron:
assignee: nobody → Dr. Jens Harbott (j-harbott)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-dynamic-routing (master)

Reviewed: https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/849178
Committed: https://opendev.org/openstack/neutron-dynamic-routing/commit/012f2358d23ef72518ef9bdef8a39f31b1fb86e2
Submitter: "Zuul (22348)"
Branch: master

commit 012f2358d23ef72518ef9bdef8a39f31b1fb86e2
Author: Dr. Jens Harbott <email address hidden>
Date: Sat Jul 9 09:45:45 2022 +0200

    Fix missing functions in StaticScheduler

    Seems the original approach was too simple, it is causing some
    tracebacks due to missing functions.

    Closes-Bug: 1980235
    Change-Id: I15ef7ec34eeccf4dfcce94b3bca2c853c4c4a609

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-dynamic-routing (stable/xena)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-dynamic-routing (stable/yoga)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-dynamic-routing (stable/wallaby)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-dynamic-routing (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/852109
Committed: https://opendev.org/openstack/neutron-dynamic-routing/commit/75c5b9537e8fa070f7d45e335e0482db7e32a8f1
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 75c5b9537e8fa070f7d45e335e0482db7e32a8f1
Author: Dr. Jens Harbott <email address hidden>
Date: Sat Jul 9 09:45:45 2022 +0200

    Fix missing functions in StaticScheduler

    Seems the original approach was too simple, it is causing some
    tracebacks due to missing functions.

    Closes-Bug: 1980235
    Change-Id: I15ef7ec34eeccf4dfcce94b3bca2c853c4c4a609
    (cherry picked from commit 012f2358d23ef72518ef9bdef8a39f31b1fb86e2)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-dynamic-routing (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/852499
Committed: https://opendev.org/openstack/neutron-dynamic-routing/commit/97853cbcb566b709c23abff822bc613f17d74ec1
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 97853cbcb566b709c23abff822bc613f17d74ec1
Author: Dr. Jens Harbott <email address hidden>
Date: Sat Jul 9 09:45:45 2022 +0200

    Fix missing functions in StaticScheduler

    Seems the original approach was too simple, it is causing some
    tracebacks due to missing functions.

    Closes-Bug: 1980235
    Change-Id: I15ef7ec34eeccf4dfcce94b3bca2c853c4c4a609
    (cherry picked from commit 012f2358d23ef72518ef9bdef8a39f31b1fb86e2)

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-dynamic-routing (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/852501
Committed: https://opendev.org/openstack/neutron-dynamic-routing/commit/5371d7157a53f73f7dc043ba5163852e517674d7
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 5371d7157a53f73f7dc043ba5163852e517674d7
Author: Dr. Jens Harbott <email address hidden>
Date: Sat Jul 9 09:45:45 2022 +0200

    Fix missing functions in StaticScheduler

    Seems the original approach was too simple, it is causing some
    tracebacks due to missing functions.

    Closes-Bug: 1980235
    Change-Id: I15ef7ec34eeccf4dfcce94b3bca2c853c4c4a609
    (cherry picked from commit 012f2358d23ef72518ef9bdef8a39f31b1fb86e2)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron-dynamic-routing 21.0.0.0rc1

This issue was fixed in the openstack/neutron-dynamic-routing 21.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron-dynamic-routing 18.1.1

This issue was fixed in the openstack/neutron-dynamic-routing 18.1.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron-dynamic-routing 19.1.1

This issue was fixed in the openstack/neutron-dynamic-routing 19.1.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.