| 2022-12-12 18:32:20 |
Thales Elero Cervi |
description |
Brief Description
-----------------
Recent installation tests of stx-openstack on Debian failed due to the neutron-ovs-agent-controller pod. It fails to initialize (container neutron-ovs-agent-init) and is broken on Init:CrashLoopBackOff state, breaking then the application apply process.
Severity
--------
Critical: stx-openstack can not be applied.
Steps to Reproduce
------------------
* Upload stx-openstack (Debian stx)
* Apply stx-openstack
Expected Behavior
------------------
stx-openstack should apply successfully
Actual Behavior
----------------
stx-openstack apply fails
Reproducibility
---------------
Reproducible
System Configuration
--------------------
AIO-SX
Branch/Pull Time/Commit
-----------------------
master:
* starlingx/master/debian/monolithic/20221206T070000Z
Last Pass
---------
N/Aq
Timestamp/Logs
--------------
$ kubectl -n openstack describe pod neutron-ovs-agent-controller-0-937646f6-sp9h9 | tail -n 5
Normal Started 37m kubelet Started container neutron-openvswitch-agent-kernel-modules
Normal Created 37m (x4 over 37m) kubelet Created container neutron-ovs-agent-init
Normal Started 37m (x4 over 37m) kubelet Started container neutron-ovs-agent-init
Normal Pulled 34m (x6 over 37m) kubelet Container image "registry.local:9001/docker.io/starlingx/stx-neutron:master-centos-stable-latest" already present on machine
Warning BackOff 4m50s (x152 over 37m) kubelet Back-off restarting failed container
$ kubectl -n openstack get pods | grep neutron-ovs-agent-controller
neutron-ovs-agent-controller-0-937646f6-sp9h9 0/1 Init:CrashLoopBackOff 12 (92s ago) 40m
neutron-ovs-agent-controller-1-cab72f56-kn822 0/1 Init:CrashLoopBackOff 12 (77s ago) 40m
$ kubectl -n openstack logs -f pod/neutron-ovs-agent-controller-0-937646f6-sp9h9 -c neutron-ovs-agent-init
+ OVS_SOCKET=/run/openvswitch/db.sock
+ chown neutron: /run/openvswitch/db.sock
++ cat /run/openvswitch/ovs-vswitchd.pid
+ OVS_PID=35
+ OVS_CTL=/run/openvswitch/ovs-vswitchd.35.ctl
+ chown neutron: /run/openvswitch/ovs-vswitchd.35.ctl
+ DPDK_CONFIG_FILE=/tmp/dpdk.conf
+ DPDK_CONFIG=
+ DPDK_ENABLED=false
+ '[' -f /tmp/dpdk.conf ']'
++ sed 's/[{}"]//g' /tmp/auto_bridge_add
++ tr , '\n'
+ for bmap in '`sed '\''s/[{}"]//g'\'' /tmp/auto_bridge_add | tr "," "\n"`'
+ bridge=br-phy0
+ iface=ens785f0
+ ovs-vsctl --no-wait --may-exist add-br br-phy0
+ '[' -n ens785f0 ']'
+ '[' ens785f0 '!=' null ']'
+ ovs-vsctl --no-wait --may-exist add-port br-phy0 ens785f0
++ get_dpdk_config_value .enabled
++ values=.enabled
++ filter=
+++ echo .enabled
+++ jq -r
/tmp/neutron-openvswitch-agent-init.sh: line 18: jq: command not found
++ value=
++ [[ '' == \n\u\l\l ]]
++ echo ''
+ [[ '' != \t\r\u\e ]]
+ ip link set dev ens785f0 up
+ tunnel_types=vxlan
+ [[ -n vxlan ]]
+ tunnel_interface=docker0
+ '[' -z docker0 ']'
+ [[ false == \t\r\u\e ]]
+ [[ -n vxlan ]]
++ get_ip_address_from_interface docker0
++ local interface=docker0
+++ ip -4 -o addr s docker0
+++ awk '{ print $4; exit }'
+++ awk -F / '{print $1}'
Device "docker0" does not exist.
++ local ip=
++ '[' -z '' ']'
++ exit 1
+ LOCAL_IP=
Test Activity
-------------
Developer Testing
Workaround
----------
N/A |
Brief Description
-----------------
Recent installation tests of stx-openstack on Debian failed due to the neutron-ovs-agent-controller pod. It fails to initialize (container neutron-ovs-agent-init) and is broken on Init:CrashLoopBackOff state, breaking then the application apply process.
This behavior was not previously reproduced on virtual deployments.
Severity
--------
Critical: stx-openstack can not be applied.
Steps to Reproduce
------------------
* Upload stx-openstack (Debian stx)
* Apply stx-openstack
Expected Behavior
------------------
stx-openstack should apply successfully
Actual Behavior
----------------
stx-openstack apply fails
Reproducibility
---------------
Reproducible
System Configuration
--------------------
AIO-SX
Branch/Pull Time/Commit
-----------------------
master:
* starlingx/master/debian/monolithic/20221206T070000Z
Last Pass
---------
N/A
Timestamp/Logs
--------------
$ kubectl -n openstack describe pod neutron-ovs-agent-controller-0-937646f6-sp9h9 | tail -n 5
Normal Started 37m kubelet Started container neutron-openvswitch-agent-kernel-modules
Normal Created 37m (x4 over 37m) kubelet Created container neutron-ovs-agent-init
Normal Started 37m (x4 over 37m) kubelet Started container neutron-ovs-agent-init
Normal Pulled 34m (x6 over 37m) kubelet Container image "registry.local:9001/docker.io/starlingx/stx-neutron:master-centos-stable-latest" already present on machine
Warning BackOff 4m50s (x152 over 37m) kubelet Back-off restarting failed container
$ kubectl -n openstack get pods | grep neutron-ovs-agent-controller
neutron-ovs-agent-controller-0-937646f6-sp9h9 0/1 Init:CrashLoopBackOff 12 (92s ago) 40m
neutron-ovs-agent-controller-1-cab72f56-kn822 0/1 Init:CrashLoopBackOff 12 (77s ago) 40m
$ kubectl -n openstack logs -f pod/neutron-ovs-agent-controller-0-937646f6-sp9h9 -c neutron-ovs-agent-init
+ OVS_SOCKET=/run/openvswitch/db.sock
+ chown neutron: /run/openvswitch/db.sock
++ cat /run/openvswitch/ovs-vswitchd.pid
+ OVS_PID=35
+ OVS_CTL=/run/openvswitch/ovs-vswitchd.35.ctl
+ chown neutron: /run/openvswitch/ovs-vswitchd.35.ctl
+ DPDK_CONFIG_FILE=/tmp/dpdk.conf
+ DPDK_CONFIG=
+ DPDK_ENABLED=false
+ '[' -f /tmp/dpdk.conf ']'
++ sed 's/[{}"]//g' /tmp/auto_bridge_add
++ tr , '\n'
+ for bmap in '`sed '\''s/[{}"]//g'\'' /tmp/auto_bridge_add | tr "," "\n"`'
+ bridge=br-phy0
+ iface=ens785f0
+ ovs-vsctl --no-wait --may-exist add-br br-phy0
+ '[' -n ens785f0 ']'
+ '[' ens785f0 '!=' null ']'
+ ovs-vsctl --no-wait --may-exist add-port br-phy0 ens785f0
++ get_dpdk_config_value .enabled
++ values=.enabled
++ filter=
+++ echo .enabled
+++ jq -r
/tmp/neutron-openvswitch-agent-init.sh: line 18: jq: command not found
++ value=
++ [[ '' == \n\u\l\l ]]
++ echo ''
+ [[ '' != \t\r\u\e ]]
+ ip link set dev ens785f0 up
+ tunnel_types=vxlan
+ [[ -n vxlan ]]
+ tunnel_interface=docker0
+ '[' -z docker0 ']'
+ [[ false == \t\r\u\e ]]
+ [[ -n vxlan ]]
++ get_ip_address_from_interface docker0
++ local interface=docker0
+++ ip -4 -o addr s docker0
+++ awk '{ print $4; exit }'
+++ awk -F / '{print $1}'
Device "docker0" does not exist.
++ local ip=
++ '[' -z '' ']'
++ exit 1
+ LOCAL_IP=
Test Activity
-------------
Developer Testing
Workaround
----------
N/A |
|
