neutron

OVN central service does not start properly

Bug #1930858 reported by Sunday Mgbogu
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Undecided
Unassigned

Bug Description

This is a Devstack deployment. While re-stacking, Unfortunately OVN central service is failing to start properly.

b/neutron_plugins/ovn_agent:start_ovn:713 : local SCRIPTDIR=/share/ovn/scripts
+ lib/neutron_plugins/ovn_agent:start_ovn:714 : use_new_ovn_repository
+ lib/neutron_plugins/ovn_agent:use_new_ovn_repository:180 : [[ False == \F\a\l\s\e ]]
+ lib/neutron_plugins/ovn_agent:use_new_ovn_repository:181 : return 0
+ lib/neutron_plugins/ovn_agent:start_ovn:718 : is_service_enabled ovn-northd
+ functions-common:is_service_enabled:1960 : return 0
+ lib/neutron_plugins/ovn_agent:start_ovn:719 : [[ False == \T\r\u\e ]]
+ lib/neutron_plugins/ovn_agent:start_ovn:725 : _start_process ovn-central.service
+ lib/neutron_plugins/ovn_agent:_start_process:220 : sudo systemctl daemon-reload
+ lib/neutron_plugins/ovn_agent:_start_process:221 : sudo systemctl enable ovn-central.service
Created symlink /etc/systemd/system/multi-user.target.wants/ovn-central.service → /lib/systemd/system/ovn-central.service.
+ lib/neutron_plugins/ovn_agent:_start_process:222 : sudo systemctl restart ovn-central.service
+ lib/neutron_plugins/ovn_agent:start_ovn:729 : wait_for_sock_file /var/run/openvswitch/ovnnb_db.sock
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:169 : local count=0
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=1
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 1 -gt 5 ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=2
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 2 -gt 5 ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=3
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 3 -gt 5 ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=4
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 4 -gt 5 ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=5
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 5 -gt 5 ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=6
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 6 -gt 5 ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:174 : die 174 'Socket /var/run/openvswitch/ovnnb_db.sock not found'
+ functions-common:die:198 : local exitcode=0
[Call Trace]
./stack.sh:1264:start_ovn_services
/opt/stack/devstack/lib/neutron-legacy:477:start_ovn
/opt/stack/devstack/lib/neutron_plugins/ovn_agent:729:wait_for_sock_file
/opt/stack/devstack/lib/neutron_plugins/ovn_agent:174:die
[ERROR] /opt/stack/devstack/lib/neutron_plugins/ovn_agent:174 Socket /var/run/openvswitch/ovnnb_db.sock not found
Error on exit
ebtables v1.8.4 (nf_tables): table `broute' is incompatible, use 'nft' tool.

See original description
Tags: ovn
Sunday Mgbogu (digitalsimboja)
description: updated
Hongbin Lu (hongbin.lu)
tags: added: ovn
Revision history for this message
Hongbin Lu (hongbin.lu) wrote :

Hi @Sunday,

Which branch of devstack you were using? Are you using master branch or a stable branch? In addition, it would be helpful to provide the devstack config you were using.

Revision history for this message
Sunday Mgbogu (digitalsimboja) wrote : Re: [Bug 1930858] Re: OVN central service does not start properly
Download full text (4.7 KiB)

Hello!
I am on Master branch and here is my local.conf
http://paste.openstack.org/show/806408/

On Mon, Jun 7, 2021 at 8:05 AM Hongbin Lu <email address hidden>
wrote:

> Hi @Sunday,
>
> Which branch of devstack you were using? Are you using master branch or
> a stable branch? In addition, it would be helpful to provide the
> devstack config you were using.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1930858
>
> Title:
> OVN central service does not start properly
>
> Status in neutron:
> New
>
> Bug description:
> This is a Devstack deployment. While re-stacking, Unfortunately OVN
> central service is failing to start properly.
>
>
> b/neutron_plugins/ovn_agent:start_ovn:713 : local
> SCRIPTDIR=/share/ovn/scripts
> + lib/neutron_plugins/ovn_agent:start_ovn:714 : use_new_ovn_repository
> + lib/neutron_plugins/ovn_agent:use_new_ovn_repository:180 : [[ False
> == \F\a\l\s\e ]]
> + lib/neutron_plugins/ovn_agent:use_new_ovn_repository:181 : return 0
> + lib/neutron_plugins/ovn_agent:start_ovn:718 : is_service_enabled
> ovn-northd
> + functions-common:is_service_enabled:1960 : return 0
> + lib/neutron_plugins/ovn_agent:start_ovn:719 : [[ False == \T\r\u\e ]]
> + lib/neutron_plugins/ovn_agent:start_ovn:725 : _start_process
> ovn-central.service
> + lib/neutron_plugins/ovn_agent:_start_process:220 : sudo systemctl
> daemon-reload
> + lib/neutron_plugins/ovn_agent:_start_process:221 : sudo systemctl
> enable ovn-central.service
> Created symlink
> /etc/systemd/system/multi-user.target.wants/ovn-central.service →
> /lib/systemd/system/ovn-central.service.
> + lib/neutron_plugins/ovn_agent:_start_process:222 : sudo systemctl
> restart ovn-central.service
> + lib/neutron_plugins/ovn_agent:start_ovn:729 : wait_for_sock_file
> /var/run/openvswitch/ovnnb_db.sock
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:169 : local count=0
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S
> /var/run/openvswitch/ovnnb_db.sock ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 1 -gt 5
> ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S
> /var/run/openvswitch/ovnnb_db.sock ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=2
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 2 -gt 5
> ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S
> /var/run/openvswitch/ovnnb_db.sock ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=3
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 3 -gt 5
> ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S
> /var/run/openvswitch/ovnnb_db.sock ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : slee...

Read more...

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hi Sunday:

Try to:
1) Unstack
2) Delete /var/run/ovn directory
3) Stack again

Do you have the logs of ovn-central.service? Seems that this service is not running correctly and could be useful to see what happened.

Regards.

Revision history for this message
Sunday Mgbogu (digitalsimboja) wrote :
Download full text (4.8 KiB)

Actually, what you proposed is what I did to get going anytime am stuck
right there.
Now I don't have the log but because I followed your above fix.
Thanks

On Tue, Jun 8, 2021, 3:30 PM Rodolfo Alonso <email address hidden>
wrote:

> Hi Sunday:
>
> Try to:
> 1) Unstack
> 2) Delete /var/run/ovn directory
> 3) Stack again
>
>
> Do you have the logs of ovn-central.service? Seems that this service is
> not running correctly and could be useful to see what happened.
>
> Regards.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1930858
>
> Title:
> OVN central service does not start properly
>
> Status in neutron:
> New
>
> Bug description:
> This is a Devstack deployment. While re-stacking, Unfortunately OVN
> central service is failing to start properly.
>
>
> b/neutron_plugins/ovn_agent:start_ovn:713 : local
> SCRIPTDIR=/share/ovn/scripts
> + lib/neutron_plugins/ovn_agent:start_ovn:714 : use_new_ovn_repository
> + lib/neutron_plugins/ovn_agent:use_new_ovn_repository:180 : [[ False
> == \F\a\l\s\e ]]
> + lib/neutron_plugins/ovn_agent:use_new_ovn_repository:181 : return 0
> + lib/neutron_plugins/ovn_agent:start_ovn:718 : is_service_enabled
> ovn-northd
> + functions-common:is_service_enabled:1960 : return 0
> + lib/neutron_plugins/ovn_agent:start_ovn:719 : [[ False == \T\r\u\e ]]
> + lib/neutron_plugins/ovn_agent:start_ovn:725 : _start_process
> ovn-central.service
> + lib/neutron_plugins/ovn_agent:_start_process:220 : sudo systemctl
> daemon-reload
> + lib/neutron_plugins/ovn_agent:_start_process:221 : sudo systemctl
> enable ovn-central.service
> Created symlink
> /etc/systemd/system/multi-user.target.wants/ovn-central.service →
> /lib/systemd/system/ovn-central.service.
> + lib/neutron_plugins/ovn_agent:_start_process:222 : sudo systemctl
> restart ovn-central.service
> + lib/neutron_plugins/ovn_agent:start_ovn:729 : wait_for_sock_file
> /var/run/openvswitch/ovnnb_db.sock
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:169 : local count=0
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S
> /var/run/openvswitch/ovnnb_db.sock ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 1 -gt 5
> ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S
> /var/run/openvswitch/ovnnb_db.sock ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=2
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 2 -gt 5
> ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S
> /var/run/openvswitch/ovnnb_db.sock ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=3
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 3 -gt 5
> ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[...

Read more...

Revision history for this message
natraj (pnatraj) wrote :

The issue that i am facing may be very similar to this isue that is looged here and i am updating the same.

Apologies if this issue is not the same.

Please i am attaching all the ovn logs here.

Revision history for this message
natraj (pnatraj) wrote :
Download full text (10.0 KiB)

from ovn-controller.log

2021-06-19T06:42:13.041Z|00006|jsonrpc|WARN|unix:/var/run/openvswitch/db.sock: receive error: Connection reset by peer
2021-06-19T06:42:13.041Z|00007|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection dropped (Connection reset by peer)
2021-06-19T06:42:14.042Z|00008|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2021-06-19T06:42:14.042Z|00009|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2021-06-19T06:42:14.311Z|00010|jsonrpc|WARN|unix:/var/run/openvswitch/db.sock: receive error: Connection reset by peer
2021-06-19T06:42:14.311Z|00011|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection dropped (Connection reset by peer)
2021-06-19T06:42:14.311Z|00012|reconnect|INFO|unix:/var/run/openvswitch/db.sock: waiting 2 seconds before reconnect
2021-06-19T06:42:16.313Z|00013|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2021-06-19T06:42:16.313Z|00014|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2021-06-19T06:42:16.316Z|00015|reconnect|INFO|tcp:192.168.29.33:6642: connecting...
2021-06-19T06:42:16.316Z|00016|reconnect|INFO|tcp:192.168.29.33:6642: connection attempt failed (Connection refused)
2021-06-19T06:42:17.317Z|00017|reconnect|INFO|tcp:192.168.29.33:6642: connecting...
2021-06-19T06:42:17.317Z|00018|reconnect|INFO|tcp:192.168.29.33:6642: connection attempt failed (Connection refused)
2021-06-19T06:42:17.317Z|00019|reconnect|INFO|tcp:192.168.29.33:6642: waiting 2 seconds before reconnect
2021-06-19T06:42:19.319Z|00020|reconnect|INFO|tcp:192.168.29.33:6642: connecting...
2021-06-19T06:42:19.319Z|00021|reconnect|INFO|tcp:192.168.29.33:6642: connection attempt failed (Connection refused)
2021-06-19T06:42:19.319Z|00022|reconnect|INFO|tcp:192.168.29.33:6642: waiting 4 seconds before reconnect
2021-06-19T06:42:23.324Z|00023|reconnect|INFO|tcp:192.168.29.33:6642: connecting...
2021-06-19T06:42:23.324Z|00024|reconnect|INFO|tcp:192.168.29.33:6642: connection attempt failed (Connection refused)
2021-06-19T06:42:23.324Z|00025|reconnect|INFO|tcp:192.168.29.33:6642: continuing to reconnect in the background but suppressing further logging

from /var/log/ovn/ovn-controller-vtep.log

2021-06-18T17:24:20.723Z|00001|vlog|INFO|opened log file /var/log/ovn/ovn-controller-vtep.log
2021-06-18T17:24:20.724Z|00002|reconnect|INFO|/var/run/openvswitch/db.sock: connecting...
2021-06-18T17:24:20.724Z|00003|reconnect|INFO|/var/run/openvswitch/db.sock: connection attempt failed (Address family not supported by protocol)
2021-06-18T17:24:21.724Z|00004|reconnect|INFO|/var/run/openvswitch/db.sock: connecting...
2021-06-18T17:24:21.724Z|00005|reconnect|INFO|/var/run/openvswitch/db.sock: connection attempt failed (Address family not supported by protocol)
2021-06-18T17:24:21.724Z|00006|reconnect|INFO|/var/run/openvswitch/db.sock: waiting 2 seconds before reconnect
2021-06-18T17:24:23.726Z|00007|reconnect|INFO|/var/run/openvswitch/db.sock: connecting...
2021-06-18T17:24:23.726Z|00008|reconnect|INFO|/var/run/openvswitch/db.sock: connection attempt failed (Address family not supported by protocol)
2021-06-18T17:24:23.726Z|00009|reconnect|INFO|/var/run/openvswitch/db.sock: waiti...

Revision history for this message
Brian Haley (brian-haley) wrote :

I believe the initial issue was fixed with commit 71c99655479174750bcedfe458328328a1596766 (https://review.opendev.org/c/openstack/devstack/+/861915) in the devstack tree which fixed issues with not finding the db sock on startup.

The issue described in comment #6 seems different, a new bug should be opened for that if it's still happening.

Changed in neutron:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.