| 2022-06-30 21:37:42 |
melanie witt |
description |
The is the same as bug 1942697 [1] and I'm opening a new bug per comment #2.
I've seen this in the gate occasionally, ovn fails to start because a socket file it expects is not present. Here's an example trace [2]:
[...]
2022-06-10 06:16:20.567 | + lib/neutron_plugins/ovn_agent:start_ovn:709 : _start_process ovn-central.service
2022-06-10 06:16:20.569 | + lib/neutron_plugins/ovn_agent:_start_process:237 : sudo systemctl daemon-reload
2022-06-10 06:16:21.393 | + lib/neutron_plugins/ovn_agent:_start_process:238 : sudo systemctl enable ovn-central.service
2022-06-10 06:16:23.834 | + lib/neutron_plugins/ovn_agent:_start_process:239 : sudo systemctl restart ovn-central.service
2022-06-10 06:16:24.814 | + lib/neutron_plugins/ovn_agent:start_ovn:714 : wait_for_sock_file /var/run/openvswitch/ovnnb_db.sock
2022-06-10 06:16:24.816 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:186 : local count=0
2022-06-10 06:16:24.818 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
2022-06-10 06:16:24.821 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 : sleep 1
2022-06-10 06:16:26.152 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 : count=1
2022-06-10 06:16:26.156 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 : '[' 1 -gt 5 ']'
2022-06-10 06:16:26.160 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
2022-06-10 06:16:26.163 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 : sleep 1
2022-06-10 06:16:27.170 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 : count=2
2022-06-10 06:16:27.174 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 : '[' 2 -gt 5 ']'
2022-06-10 06:16:27.178 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
2022-06-10 06:16:27.181 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 : sleep 1
2022-06-10 06:16:28.188 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 : count=3
2022-06-10 06:16:28.192 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 : '[' 3 -gt 5 ']'
2022-06-10 06:16:28.196 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
2022-06-10 06:16:28.200 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 : sleep 1
2022-06-10 06:16:29.207 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 : count=4
2022-06-10 06:16:29.211 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 : '[' 4 -gt 5 ']'
2022-06-10 06:16:29.215 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
2022-06-10 06:16:29.218 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 : sleep 1
2022-06-10 06:16:30.225 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 : count=5
2022-06-10 06:16:30.229 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 : '[' 5 -gt 5 ']'
2022-06-10 06:16:30.233 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
2022-06-10 06:16:30.236 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 : sleep 1
2022-06-10 06:16:31.243 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 : count=6
2022-06-10 06:16:31.247 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 : '[' 6 -gt 5 ']'
2022-06-10 06:16:31.251 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:191 : die 191 'Socket /var/run/openvswitch/ovnnb_db.sock not found'
2022-06-10 06:16:31.255 | + functions-common:die:264 : local exitcode=0
2022-06-10 06:16:31.259 | [Call Trace]
2022-06-10 06:16:31.259 | ./stack.sh:1277:start_ovn_services
2022-06-10 06:16:31.260 | /opt/stack/devstack/lib/neutron-legacy:512:start_ovn
2022-06-10 06:16:31.260 | /opt/stack/devstack/lib/neutron_plugins/ovn_agent:714:wait_for_sock_file
2022-06-10 06:16:31.260 | /opt/stack/devstack/lib/neutron_plugins/ovn_agent:191:die
2022-06-10 06:16:31.266 | [ERROR] /opt/stack/devstack/lib/neutron_plugins/ovn_agent:191 Socket /var/run/openvswitch/ovnnb_db.sock not found
2022-06-10 06:16:32.272 | exit_trap: cleaning up child processes
2022-06-10 06:16:32.275 | Error on exit
opensearch query:
11 hits in the last 30 days, check and gate queues, all failures
https://opensearch.logs.openstack.org/_dashboards/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(build_change,build_status,build_queue),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:lucene,query:'message:%20%22%5BERROR%5D%22%20AND%20message:%22Socket%20%2Fvar%2Frun%2Fopenvswitch%2Fovnnb_db.sock%20not%20found%22%20AND%20tags:devstack'),sort:!())
[1] https://bugs.launchpad.net/devstack/+bug/1942697
[2] https://zuul.opendev.org/t/openstack/build/435a3650d30c40668ae06a3de7efb70f/log/controller/logs/devstacklog.txt#13448 |
This is the same as bug 1942697 [1] and I'm opening a new bug per comment #2.
I've seen this in the gate occasionally, ovn fails to start because a socket file it expects is not present. Here's an example trace [2]:
[...]
2022-06-10 06:16:20.567 | + lib/neutron_plugins/ovn_agent:start_ovn:709 : _start_process ovn-central.service
2022-06-10 06:16:20.569 | + lib/neutron_plugins/ovn_agent:_start_process:237 : sudo systemctl daemon-reload
2022-06-10 06:16:21.393 | + lib/neutron_plugins/ovn_agent:_start_process:238 : sudo systemctl enable ovn-central.service
2022-06-10 06:16:23.834 | + lib/neutron_plugins/ovn_agent:_start_process:239 : sudo systemctl restart ovn-central.service
2022-06-10 06:16:24.814 | + lib/neutron_plugins/ovn_agent:start_ovn:714 : wait_for_sock_file /var/run/openvswitch/ovnnb_db.sock
2022-06-10 06:16:24.816 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:186 : local count=0
2022-06-10 06:16:24.818 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
2022-06-10 06:16:24.821 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 : sleep 1
2022-06-10 06:16:26.152 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 : count=1
2022-06-10 06:16:26.156 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 : '[' 1 -gt 5 ']'
2022-06-10 06:16:26.160 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
2022-06-10 06:16:26.163 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 : sleep 1
2022-06-10 06:16:27.170 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 : count=2
2022-06-10 06:16:27.174 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 : '[' 2 -gt 5 ']'
2022-06-10 06:16:27.178 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
2022-06-10 06:16:27.181 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 : sleep 1
2022-06-10 06:16:28.188 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 : count=3
2022-06-10 06:16:28.192 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 : '[' 3 -gt 5 ']'
2022-06-10 06:16:28.196 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
2022-06-10 06:16:28.200 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 : sleep 1
2022-06-10 06:16:29.207 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 : count=4
2022-06-10 06:16:29.211 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 : '[' 4 -gt 5 ']'
2022-06-10 06:16:29.215 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
2022-06-10 06:16:29.218 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 : sleep 1
2022-06-10 06:16:30.225 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 : count=5
2022-06-10 06:16:30.229 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 : '[' 5 -gt 5 ']'
2022-06-10 06:16:30.233 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
2022-06-10 06:16:30.236 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 : sleep 1
2022-06-10 06:16:31.243 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 : count=6
2022-06-10 06:16:31.247 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 : '[' 6 -gt 5 ']'
2022-06-10 06:16:31.251 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:191 : die 191 'Socket /var/run/openvswitch/ovnnb_db.sock not found'
2022-06-10 06:16:31.255 | + functions-common:die:264 : local exitcode=0
2022-06-10 06:16:31.259 | [Call Trace]
2022-06-10 06:16:31.259 | ./stack.sh:1277:start_ovn_services
2022-06-10 06:16:31.260 | /opt/stack/devstack/lib/neutron-legacy:512:start_ovn
2022-06-10 06:16:31.260 | /opt/stack/devstack/lib/neutron_plugins/ovn_agent:714:wait_for_sock_file
2022-06-10 06:16:31.260 | /opt/stack/devstack/lib/neutron_plugins/ovn_agent:191:die
2022-06-10 06:16:31.266 | [ERROR] /opt/stack/devstack/lib/neutron_plugins/ovn_agent:191 Socket /var/run/openvswitch/ovnnb_db.sock not found
2022-06-10 06:16:32.272 | exit_trap: cleaning up child processes
2022-06-10 06:16:32.275 | Error on exit
opensearch query:
11 hits in the last 30 days, check and gate queues, all failures
https://opensearch.logs.openstack.org/_dashboards/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(build_change,build_status,build_queue),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:lucene,query:'message:%20%22%5BERROR%5D%22%20AND%20message:%22Socket%20%2Fvar%2Frun%2Fopenvswitch%2Fovnnb_db.sock%20not%20found%22%20AND%20tags:devstack'),sort:!())
[1] https://bugs.launchpad.net/devstack/+bug/1942697
[2] https://zuul.opendev.org/t/openstack/build/435a3650d30c40668ae06a3de7efb70f/log/controller/logs/devstacklog.txt#13448 |
|
