OVN central service does not start properly

Bug #1930858 reported by Sunday Mgbogu
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Undecided
Unassigned

Bug Description

This is a Devstack deployment. While re-stacking, Unfortunately OVN central service is failing to start properly.

b/neutron_plugins/ovn_agent:start_ovn:713 : local SCRIPTDIR=/share/ovn/scripts
+ lib/neutron_plugins/ovn_agent:start_ovn:714 : use_new_ovn_repository
+ lib/neutron_plugins/ovn_agent:use_new_ovn_repository:180 : [[ False == \F\a\l\s\e ]]
+ lib/neutron_plugins/ovn_agent:use_new_ovn_repository:181 : return 0
+ lib/neutron_plugins/ovn_agent:start_ovn:718 : is_service_enabled ovn-northd
+ functions-common:is_service_enabled:1960 : return 0
+ lib/neutron_plugins/ovn_agent:start_ovn:719 : [[ False == \T\r\u\e ]]
+ lib/neutron_plugins/ovn_agent:start_ovn:725 : _start_process ovn-central.service
+ lib/neutron_plugins/ovn_agent:_start_process:220 : sudo systemctl daemon-reload
+ lib/neutron_plugins/ovn_agent:_start_process:221 : sudo systemctl enable ovn-central.service
Created symlink /etc/systemd/system/multi-user.target.wants/ovn-central.service → /lib/systemd/system/ovn-central.service.
+ lib/neutron_plugins/ovn_agent:_start_process:222 : sudo systemctl restart ovn-central.service
+ lib/neutron_plugins/ovn_agent:start_ovn:729 : wait_for_sock_file /var/run/openvswitch/ovnnb_db.sock
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:169 : local count=0
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=1
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 1 -gt 5 ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=2
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 2 -gt 5 ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=3
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 3 -gt 5 ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=4
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 4 -gt 5 ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=5
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 5 -gt 5 ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=6
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 6 -gt 5 ']'
+ lib/neutron_plugins/ovn_agent:wait_for_sock_file:174 : die 174 'Socket /var/run/openvswitch/ovnnb_db.sock not found'
+ functions-common:die:198 : local exitcode=0
[Call Trace]
./stack.sh:1264:start_ovn_services
/opt/stack/devstack/lib/neutron-legacy:477:start_ovn
/opt/stack/devstack/lib/neutron_plugins/ovn_agent:729:wait_for_sock_file
/opt/stack/devstack/lib/neutron_plugins/ovn_agent:174:die
[ERROR] /opt/stack/devstack/lib/neutron_plugins/ovn_agent:174 Socket /var/run/openvswitch/ovnnb_db.sock not found
Error on exit
ebtables v1.8.4 (nf_tables): table `broute' is incompatible, use 'nft' tool.

Tags: ovn Edit Tag help
description: updated
Hongbin Lu (hongbin.lu)
tags: added: ovn
Revision history for this message
Hongbin Lu (hongbin.lu) wrote :

Hi @Sunday,

Which branch of devstack you were using? Are you using master branch or a stable branch? In addition, it would be helpful to provide the devstack config you were using.

Revision history for this message
Sunday Mgbogu (digitalsimboja) wrote : Re: [Bug 1930858] Re: OVN central service does not start properly
Download full text (4.7 KiB)

Hello!
I am on Master branch and here is my local.conf
http://paste.openstack.org/show/806408/

On Mon, Jun 7, 2021 at 8:05 AM Hongbin Lu <email address hidden>
wrote:

> Hi @Sunday,
>
> Which branch of devstack you were using? Are you using master branch or
> a stable branch? In addition, it would be helpful to provide the
> devstack config you were using.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1930858
>
> Title:
> OVN central service does not start properly
>
> Status in neutron:
> New
>
> Bug description:
> This is a Devstack deployment. While re-stacking, Unfortunately OVN
> central service is failing to start properly.
>
>
> b/neutron_plugins/ovn_agent:start_ovn:713 : local
> SCRIPTDIR=/share/ovn/scripts
> + lib/neutron_plugins/ovn_agent:start_ovn:714 : use_new_ovn_repository
> + lib/neutron_plugins/ovn_agent:use_new_ovn_repository:180 : [[ False
> == \F\a\l\s\e ]]
> + lib/neutron_plugins/ovn_agent:use_new_ovn_repository:181 : return 0
> + lib/neutron_plugins/ovn_agent:start_ovn:718 : is_service_enabled
> ovn-northd
> + functions-common:is_service_enabled:1960 : return 0
> + lib/neutron_plugins/ovn_agent:start_ovn:719 : [[ False == \T\r\u\e ]]
> + lib/neutron_plugins/ovn_agent:start_ovn:725 : _start_process
> ovn-central.service
> + lib/neutron_plugins/ovn_agent:_start_process:220 : sudo systemctl
> daemon-reload
> + lib/neutron_plugins/ovn_agent:_start_process:221 : sudo systemctl
> enable ovn-central.service
> Created symlink
> /etc/systemd/system/multi-user.target.wants/ovn-central.service →
> /lib/systemd/system/ovn-central.service.
> + lib/neutron_plugins/ovn_agent:_start_process:222 : sudo systemctl
> restart ovn-central.service
> + lib/neutron_plugins/ovn_agent:start_ovn:729 : wait_for_sock_file
> /var/run/openvswitch/ovnnb_db.sock
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:169 : local count=0
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S
> /var/run/openvswitch/ovnnb_db.sock ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 1 -gt 5
> ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S
> /var/run/openvswitch/ovnnb_db.sock ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=2
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 2 -gt 5
> ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S
> /var/run/openvswitch/ovnnb_db.sock ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=3
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 3 -gt 5
> ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S
> /var/run/openvswitch/ovnnb_db.sock ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : slee...

Read more...

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hi Sunday:

Try to:
1) Unstack
2) Delete /var/run/ovn directory
3) Stack again

Do you have the logs of ovn-central.service? Seems that this service is not running correctly and could be useful to see what happened.

Regards.

Revision history for this message
Sunday Mgbogu (digitalsimboja) wrote :
Download full text (4.8 KiB)

Actually, what you proposed is what I did to get going anytime am stuck
right there.
Now I don't have the log but because I followed your above fix.
Thanks

On Tue, Jun 8, 2021, 3:30 PM Rodolfo Alonso <email address hidden>
wrote:

> Hi Sunday:
>
> Try to:
> 1) Unstack
> 2) Delete /var/run/ovn directory
> 3) Stack again
>
>
> Do you have the logs of ovn-central.service? Seems that this service is
> not running correctly and could be useful to see what happened.
>
> Regards.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1930858
>
> Title:
> OVN central service does not start properly
>
> Status in neutron:
> New
>
> Bug description:
> This is a Devstack deployment. While re-stacking, Unfortunately OVN
> central service is failing to start properly.
>
>
> b/neutron_plugins/ovn_agent:start_ovn:713 : local
> SCRIPTDIR=/share/ovn/scripts
> + lib/neutron_plugins/ovn_agent:start_ovn:714 : use_new_ovn_repository
> + lib/neutron_plugins/ovn_agent:use_new_ovn_repository:180 : [[ False
> == \F\a\l\s\e ]]
> + lib/neutron_plugins/ovn_agent:use_new_ovn_repository:181 : return 0
> + lib/neutron_plugins/ovn_agent:start_ovn:718 : is_service_enabled
> ovn-northd
> + functions-common:is_service_enabled:1960 : return 0
> + lib/neutron_plugins/ovn_agent:start_ovn:719 : [[ False == \T\r\u\e ]]
> + lib/neutron_plugins/ovn_agent:start_ovn:725 : _start_process
> ovn-central.service
> + lib/neutron_plugins/ovn_agent:_start_process:220 : sudo systemctl
> daemon-reload
> + lib/neutron_plugins/ovn_agent:_start_process:221 : sudo systemctl
> enable ovn-central.service
> Created symlink
> /etc/systemd/system/multi-user.target.wants/ovn-central.service →
> /lib/systemd/system/ovn-central.service.
> + lib/neutron_plugins/ovn_agent:_start_process:222 : sudo systemctl
> restart ovn-central.service
> + lib/neutron_plugins/ovn_agent:start_ovn:729 : wait_for_sock_file
> /var/run/openvswitch/ovnnb_db.sock
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:169 : local count=0
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S
> /var/run/openvswitch/ovnnb_db.sock ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 1 -gt 5
> ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S
> /var/run/openvswitch/ovnnb_db.sock ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=2
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 2 -gt 5
> ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[' '!' -S
> /var/run/openvswitch/ovnnb_db.sock ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:171 : sleep 1
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:172 : count=3
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : '[' 3 -gt 5
> ']'
> + lib/neutron_plugins/ovn_agent:wait_for_sock_file:170 : '[...

Read more...

Revision history for this message
natraj (pnatraj) wrote :

The issue that i am facing may be very similar to this isue that is looged here and i am updating the same.

Apologies if this issue is not the same.

Please i am attaching all the ovn logs here.

Revision history for this message
natraj (pnatraj) wrote :
Download full text (10.0 KiB)

from ovn-controller.log

2021-06-19T06:42:13.041Z|00006|jsonrpc|WARN|unix:/var/run/openvswitch/db.sock: receive error: Connection reset by peer
2021-06-19T06:42:13.041Z|00007|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection dropped (Connection reset by peer)
2021-06-19T06:42:14.042Z|00008|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2021-06-19T06:42:14.042Z|00009|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2021-06-19T06:42:14.311Z|00010|jsonrpc|WARN|unix:/var/run/openvswitch/db.sock: receive error: Connection reset by peer
2021-06-19T06:42:14.311Z|00011|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection dropped (Connection reset by peer)
2021-06-19T06:42:14.311Z|00012|reconnect|INFO|unix:/var/run/openvswitch/db.sock: waiting 2 seconds before reconnect
2021-06-19T06:42:16.313Z|00013|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2021-06-19T06:42:16.313Z|00014|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2021-06-19T06:42:16.316Z|00015|reconnect|INFO|tcp:192.168.29.33:6642: connecting...
2021-06-19T06:42:16.316Z|00016|reconnect|INFO|tcp:192.168.29.33:6642: connection attempt failed (Connection refused)
2021-06-19T06:42:17.317Z|00017|reconnect|INFO|tcp:192.168.29.33:6642: connecting...
2021-06-19T06:42:17.317Z|00018|reconnect|INFO|tcp:192.168.29.33:6642: connection attempt failed (Connection refused)
2021-06-19T06:42:17.317Z|00019|reconnect|INFO|tcp:192.168.29.33:6642: waiting 2 seconds before reconnect
2021-06-19T06:42:19.319Z|00020|reconnect|INFO|tcp:192.168.29.33:6642: connecting...
2021-06-19T06:42:19.319Z|00021|reconnect|INFO|tcp:192.168.29.33:6642: connection attempt failed (Connection refused)
2021-06-19T06:42:19.319Z|00022|reconnect|INFO|tcp:192.168.29.33:6642: waiting 4 seconds before reconnect
2021-06-19T06:42:23.324Z|00023|reconnect|INFO|tcp:192.168.29.33:6642: connecting...
2021-06-19T06:42:23.324Z|00024|reconnect|INFO|tcp:192.168.29.33:6642: connection attempt failed (Connection refused)
2021-06-19T06:42:23.324Z|00025|reconnect|INFO|tcp:192.168.29.33:6642: continuing to reconnect in the background but suppressing further logging

from /var/log/ovn/ovn-controller-vtep.log

2021-06-18T17:24:20.723Z|00001|vlog|INFO|opened log file /var/log/ovn/ovn-controller-vtep.log
2021-06-18T17:24:20.724Z|00002|reconnect|INFO|/var/run/openvswitch/db.sock: connecting...
2021-06-18T17:24:20.724Z|00003|reconnect|INFO|/var/run/openvswitch/db.sock: connection attempt failed (Address family not supported by protocol)
2021-06-18T17:24:21.724Z|00004|reconnect|INFO|/var/run/openvswitch/db.sock: connecting...
2021-06-18T17:24:21.724Z|00005|reconnect|INFO|/var/run/openvswitch/db.sock: connection attempt failed (Address family not supported by protocol)
2021-06-18T17:24:21.724Z|00006|reconnect|INFO|/var/run/openvswitch/db.sock: waiting 2 seconds before reconnect
2021-06-18T17:24:23.726Z|00007|reconnect|INFO|/var/run/openvswitch/db.sock: connecting...
2021-06-18T17:24:23.726Z|00008|reconnect|INFO|/var/run/openvswitch/db.sock: connection attempt failed (Address family not supported by protocol)
2021-06-18T17:24:23.726Z|00009|reconnect|INFO|/var/run/openvswitch/db.sock: waiti...

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers