neutron_openvswitch_agent container restarting

Bug #1753511 reported by Brendan Kelly
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Undecided
Unassigned

Bug Description

Neutron openvswitch agent constantly restarting.

Docker image:
1c6b90d1b835 kolla/ubuntu-source-neutron-openvswitch-agent:pike

Docker logs:
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/neutron/neutron.conf
INFO:__main__:Copying /var/lib/kolla/config_files/neutron.conf to /etc/neutron/neutron.conf
INFO:__main__:Setting permission for /etc/neutron/neutron.conf
INFO:__main__:Deleting /etc/neutron/plugins/ml2/ml2_conf.ini
INFO:__main__:Copying /var/lib/kolla/config_files/ml2_conf.ini to /etc/neutron/plugins/ml2/ml2_conf.ini
INFO:__main__:Setting permission for /etc/neutron/plugins/ml2/ml2_conf.ini
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/log/kolla/neutron
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-dhcp-agent.log
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-l3-agent.log
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-server.log
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-metadata-agent.log
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-netns-cleanup.log
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-openvswitch-agent.log
Running command: 'neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini'
Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
Traceback (most recent call last):
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/queue.py", line 118, in switch
    self.greenlet.switch(value)
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/ryu/lib/hub.py", line 65, in _launch
    raise e
SystemExit: 1

ml2_conf.ini:
root@kolla:/etc/kolla/neutron-openvswitch-agent# cat ml2_conf.ini
[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vxlan
mechanism_drivers = openvswitch,l2population
extension_drivers = port_security

[ml2_type_vlan]
network_vlan_ranges =

[ml2_type_flat]
flat_networks = physnet1

[ml2_type_vxlan]
vni_ranges = 1:1000
vxlan_group = 239.1.1.1

[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

[agent]
tunnel_types = vxlan
l2_population = true
arp_responder = true

[ovs]
bridge_mappings = physnet1:dpdk_bridge
datapath_type = netdev
ovsdb_connection = tcp:127.0.0.1:6640
local_ip = 192.168.5.228

globals.yml:
root@kolla:/etc/kolla# cat globals.yml
---
# You can use this file to override _any_ variable throughout Kolla.
# Additional options can be found in the
# 'kolla-ansible/ansible/group_vars/all.yml' file. Default value of all the
# commented parameters are shown here, To override the default value uncomment
# the parameter and change its value.

###############
# Kolla options
###############
# Valid options are [ COPY_ONCE, COPY_ALWAYS ]
#config_strategy: "COPY_ALWAYS"

# Valid options are ['centos', 'debian', 'oraclelinux', 'rhel', 'ubuntu']
kolla_base_distro: "ubuntu"

# Valid options are [ binary, source ]
kolla_install_type: "source"

# Valid option is Docker repository tag
openstack_release: "pike"

# Location of configuration overrides
#node_custom_config: "/etc/kolla/config"

# This should be a VIP, an unused IP on your network that will float between
# the hosts running keepalived for high-availability. If you want to run an
# All-In-One without haproxy and keepalived, you can set enable_haproxy to no
# in "OpenStack options" section, and set this value to the IP of your
# 'network_interface' as set in the Networking section below.
kolla_internal_vip_address: "192.168.5.230"

# This is the DNS name that maps to the kolla_internal_vip_address VIP. By
# default it is the same as kolla_internal_vip_address.
#kolla_internal_fqdn: "{{ kolla_internal_vip_address }}"

# This should be a VIP, an unused IP on your network that will float between
# the hosts running keepalived for high-availability. It defaults to the
# kolla_internal_vip_address, allowing internal and external communication to
# share the same address. Specify a kolla_external_vip_address to separate
# internal and external requests between two VIPs.
#kolla_external_vip_address: "{{ kolla_internal_vip_address }}"

# The Public address used to communicate with OpenStack as set in the public_url
# for the endpoints that will be created. This DNS name should map to
# kolla_external_vip_address.
#kolla_external_fqdn: "{{ kolla_external_vip_address }}"

################
# Docker options
################
# Below is an example of a private repository with authentication. Note the
# Docker registry password can also be set in the passwords.yml file.

#docker_registry: "172.16.0.10:4000"
#docker_namespace: "companyname"
#docker_registry_username: "sam"
#docker_registry_password: "correcthorsebatterystaple"

###################
# Messaging options
###################
# Below is an example of an separate backend that provides brokerless
# messaging for oslo.messaging RPC communications

#om_rpc_transport: "amqp"
#om_rpc_user: "{{ qdrouterd_user }}"
#om_rpc_password: "{{ qdrouterd_password }}"
#om_rpc_port: "{{ qdrouterd_port }}"
#om_rpc_group: "qdrouterd"

##############################
# Neutron - Networking Options
##############################
# This interface is what all your api services will be bound to by default.
# Additionally, all vxlan/tunnel and storage network traffic will go over this
# interface by default. This interface must contain an IPv4 address.
# It is possible for hosts to have non-matching names of interfaces - these can
# be set in an inventory file per host or per group or stored separately, see
# http://docs.ansible.com/ansible/intro_inventory.html
# Yet another way to workaround the naming problem is to create a bond for the
# interface on all hosts and give the bond name here. Similar strategy can be
# followed for other types of interfaces.
network_interface: "enp2s0f1"

# These can be adjusted for even more customization. The default is the same as
# the 'network_interface'. These interfaces must contain an IPv4 address.
#kolla_external_vip_interface: "{{ network_interface }}"
#api_interface: "{{ network_interface }}"
#storage_interface: "{{ network_interface }}"
#cluster_interface: "{{ network_interface }}"
#tunnel_interface: "{{ network_interface }}"
#tunnel_interface: "dpdk_bridge"
#dns_interface: "{{ network_interface }}"

# This is the raw interface given to neutron as its external network port. Even
# though an IP address can exist on this interface, it will be unusable in most
# configurations. It is recommended this interface not be configured with any IP
# addresses for that reason.
neutron_external_interface: "enp2s0f2"

# Valid options are [ openvswitch, linuxbridge, vmware_nsxv, vmware_dvs, opendaylight ]

#neutron_plugin_agent: "openvswitch"
neutron_bridge_name: "dpdk_bridge"

####################
# keepalived options
####################
# Arbitrary unique number from 0..255
#keepalived_virtual_router_id: "51"

#############
# TLS options
#############
# To provide encryption and authentication on the kolla_external_vip_interface,
# TLS can be enabled. When TLS is enabled, certificates must be provided to
# allow clients to perform authentication.
#kolla_enable_tls_external: "no"
#kolla_external_fqdn_cert: "{{ node_config_directory }}/certificates/haproxy.pem"

##############
# OpenDaylight
##############
#enable_opendaylight_qos: "no"
#enable_opendaylight_l3: "yes"

###################
# OpenStack options
###################
# Use these options to set the various log levels across all OpenStack projects
# Valid options are [ True, False ]
#openstack_logging_debug: "False"

# Valid options are [ none, novnc, spice, rdp ]
nova_console: "novnc"

# OpenStack services can be enabled or disabled with these options
#enable_aodh: "no"
#enable_barbican: "no"
#enable_blazar: "no"
#enable_ceilometer: "no"
#enable_central_logging: "no"
#enable_ceph: "yes"
#enable_ceph_mds: "no"
#enable_ceph_rgw: "no"
#enable_ceph_nfs: "no"
#enable_chrony: "no"
#enable_cinder: "yes"
#enable_cinder_backup: "yes"
#enable_cinder_backend_hnas_iscsi: "no"
#enable_cinder_backend_hnas_nfs: "no"
#enable_cinder_backend_iscsi: "no"
#enable_cinder_backend_lvm: "no"
#enable_cinder_backend_nfs: "no"
#enable_cloudkitty: "no"
#enable_collectd: "no"
#enable_congress: "no"
#enable_designate: "no"
#enable_destroy_images: "no"
#enable_etcd: "no"
#enable_fluentd: "yes"
#enable_freezer: "no"
#enable_gnocchi: "no"
#enable_grafana: "no"
#enable_haproxy: "yes"
enable_heat: "yes"
enable_horizon: "yes"
#enable_horizon_cloudkitty: "{{ enable_cloudkitty | bool }}"
#enable_horizon_designate: "{{ enable_designate | bool }}"
#enable_horizon_freezer: "{{ enable_freezer | bool }}"
#enable_horizon_ironic: "{{ enable_ironic | bool }}"
#enable_horizon_karbor: "{{ enable_karbor | bool }}"
#enable_horizon_magnum: "{{ enable_magnum | bool }}"
#enable_horizon_manila: "{{ enable_manila | bool }}"
#enable_horizon_mistral: "{{ enable_mistral | bool }}"
#enable_horizon_murano: "{{ enable_murano | bool }}"
#enable_horizon_neutron_lbaas: "{{ enable_neutron_lbaas | bool }}"
#enable_horizon_sahara: "{{ enable_sahara | bool }}"
#enable_horizon_searchlight: "{{ enable_searchlight | bool }}"
#enable_horizon_senlin: "{{ enable_senlin | bool }}"
#enable_horizon_solum: "{{ enable_solum | bool }}"
#enable_horizon_tacker: "{{ enable_tacker | bool }}"
#enable_horizon_trove: "{{ enable_trove | bool }}"
#enable_horizon_watcher: "{{ enable_watcher | bool }}"
#enable_horizon_zun: "{{ enable_zun | bool }}"
#enable_hyperv: "no"
#enable_influxdb: "no"
#enable_ironic: "no"
#enable_ironic_pxe_uefi: "no"
#enable_karbor: "no"
#enable_kuryr: "no"
#enable_magnum: "no"
#enable_manila: "no"
#enable_manila_backend_generic: "no"
#enable_manila_backend_hnas: "no"
#enable_manila_backend_cephfs_native: "no"
#enable_manila_backend_cephfs_nfs: "no"
#enable_mistral: "no"
#enable_mongodb: "no"
#enable_murano: "no"
#enable_multipathd: "no"
#enable_neutron_bgp_dragent: "no"
#enable_neutron_dvr: "no"
#enable_neutron_lbaas: "no"
#enable_neutron_fwaas: "no"
#enable_neutron_qos: "no"
#enable_neutron_agent_ha: "no"
#enable_neutron_vpnaas: "no"
#enable_neutron_sriov: "no"
#enable_neutron_sfc: "no"
#enable_nova_fake: "no"
#enable_nova_serialconsole_proxy: "no"
#enable_octavia: "no"
#enable_opendaylight: "no"
enable_openvswitch: "yes"
ovs_datapath: "netdev"
enable_ovs_dpdk: "yes"
enable_neutron_provider_networks: "yes"
#enable_osprofiler: "no"
#enable_panko: "no"
#enable_qdrouterd: "no"
#enable_rally: "no"
#enable_redis: "no"
#enable_sahara: "no"
#enable_searchlight: "no"
#enable_senlin: "no"
#enable_skydive: "no"
#enable_solum: "no"
#enable_swift: "no"
#enable_telegraf: "no"
#enable_tacker: "no"
#enable_tempest: "no"
#enable_trove: "no"
#enable_vitrage: "no"
#enable_vmtp: "no"
#enable_watcher: "no"
#enable_zun: "no"

##############
# Ceph options
##############
# Ceph can be setup with a caching to improve performance. To use the cache you
# must provide separate disks than those for the OSDs
#ceph_enable_cache: "no"

# Set to no if using external Ceph without cephx.
#external_ceph_cephx_enabled: "yes"

# Ceph is not able to determine the size of a cache pool automatically,
# so the configuration on the absolute size is required here, otherwise the flush/evict will not work.
#ceph_target_max_bytes: ""
#ceph_target_max_objects: ""

# Valid options are [ forward, none, writeback ]
#ceph_cache_mode: "writeback"

# A requirement for using the erasure-coded pools is you must setup a cache tier
# Valid options are [ erasure, replicated ]
#ceph_pool_type: "replicated"

# Integrate ceph rados object gateway with openstack keystone
#enable_ceph_rgw_keystone: "no"

# Set the pgs and pgps for pool
#ceph_pool_pg_num: 128
#ceph_pool_pgp_num: 128

#############################
# Keystone - Identity Options
#############################

# Valid options are [ uuid, fernet ]
#keystone_token_provider: 'uuid'

# Interval to rotate fernet keys by (in seconds). Must be an interval of
# 60(1 min), 120(2 min), 180(3 min), 240(4 min), 300(5 min), 360(6 min),
# 600(10 min), 720(12 min), 900(15 min), 1200(20 min), 1800(30 min),
# 3600(1 hour), 7200(2 hour), 10800(3 hour), 14400(4 hour), 21600(6 hour),
# 28800(8 hour), 43200(12 hour), 86400(1 day), 604800(1 week).
#fernet_token_expiry: 86400

########################
# Glance - Image Options
########################
# Configure image backend.
#glance_backend_file: "yes"
#glance_backend_ceph: "no"
#glance_backend_vmware: "no"
#glance_backend_swift: "no"

##################
# Barbican options
##################
# Valid options are [ simple_crypto, p11_crypto ]
#barbican_crypto_plugin: "simple_crypto"
#barbican_library_path: "/usr/lib/libCryptoki2_64.so"

################
## Panko options
################
# Valid options are [ mongodb, mysql ]
#panko_database_type: "mysql"

#################
# Gnocchi options
#################
# Valid options are [ file, ceph ]
#gnocchi_backend_storage: "{{ 'ceph' if enable_ceph|bool else 'file' }}"

################################
# Cinder - Block Storage Options
################################
# Enable / disable Cinder backends
#cinder_backend_ceph: "{{ enable_ceph }}"
#cinder_backend_vmwarevc_vmdk: "no"
#cinder_volume_group: "cinder-volumes"

# Valid options are [ nfs, swift, ceph ]
#cinder_backup_driver: "ceph"
#cinder_backup_share: ""
#cinder_backup_mount_options_nfs: ""

###################
# Designate options
###################
# Valid options are [ bind9 ]
#designate_backend: "bind9"
#designate_ns_record: "sample.openstack.org"

########################
# Nova - Compute Options
########################
#nova_backend_ceph: "{{ enable_ceph }}"

# Valid options are [ qemu, kvm, vmware, xenapi ]
#nova_compute_virt_type: "kvm"

# The number of fake driver per compute node
#num_nova_fake_per_node: 5

#################
# Hyper-V options
#################
# Hyper-V can be used as hypervisor
#hyperv_username: "user"
#hyperv_password: "password"
#vswitch_name: "vswitch"
# URL from which Nova Hyper-V MSI is downloaded
#nova_msi_url: "https://www.cloudbase.it/downloads/HyperVNovaCompute_Beta.msi"

#############################
# Horizon - Dashboard Options
#############################
#horizon_backend_database: "{{ enable_murano | bool }}"

#############################
# Ironic options
#############################
#ironic_dnsmasq_dhcp_range:

######################################
# Manila - Shared File Systems Options
######################################
# HNAS backend configuration
#hnas_ip:
#hnas_user:
#hnas_password:
#hnas_evs_id:
#hnas_evs_ip:
#hnas_file_system_name:

################################
# Swift - Object Storage Options
################################
# Swift expects block devices to be available for storage. Two types of storage
# are supported: 1 - storage device with a special partition name and filesystem
# label, 2 - unpartitioned disk with a filesystem. The label of this filesystem
# is used to detect the disk which Swift will be using.

# Swift support two matching modes, valid options are [ prefix, strict ]
#swift_devices_match_mode: "strict"

# This parameter defines matching pattern: if "strict" mode was selected,
# for swift_devices_match_mode then swift_device_name should specify the name of
# the special swift partition for example: "KOLLA_SWIFT_DATA", if "prefix" mode was
# selected then swift_devices_name should specify a pattern which would match to
# filesystems' labels prepared for swift.
#swift_devices_name: "KOLLA_SWIFT_DATA"

################################################
# Tempest - The OpenStack Integration Test Suite
################################################
# following value must be set when enable tempest
tempest_image_id:
tempest_flavor_ref_id:
tempest_public_network_id:
tempest_floating_network_name:

# tempest_image_alt_id: "{{ tempest_image_id }}"
# tempest_flavor_ref_alt_id: "{{ tempest_flavor_ref_id }}"

###################################
# VMware - OpenStack VMware support
###################################
#vmware_vcenter_host_ip:
#vmware_vcenter_host_username:
#vmware_vcenter_host_password:
#vmware_datastore_name:
#vmware_vcenter_name:
#vmware_vcenter_cluster_name:

#######################################
# XenAPI - Support XenAPI for XenServer
#######################################
# XenAPI driver use HIMN(Host Internal Management Network)
# to communicate with XenServer host.
#xenserver_himn_ip:
#xenserver_username:
#xenserver_connect_protocol:

Revision history for this message
Raj Gaire (rajgaire) wrote :

I had the same issue with kolla/centos-binary-neutron-openvswitch-agent:pike. I am also using multinode deployment with default parameters in global.yml (except for the network ips) and openstack_release: "pike".

Revision history for this message
Bonga (bongad) wrote :

same here im using ocata though any fix ?

Revision history for this message
Raymond Flagg (raymond.flagg) wrote :

I'm seeing the same issue on rocky, does anyone have a fix for this?

Revision history for this message
Eric Miller (erickmiller) wrote :

Configuration: Kolla-Ansible 7.0.0 release on CentOS 7.5

I see the same issue with Rocky after a restart of all nodes - during testing failures.

On all compute nodes...

Running:
docker logs neutron_openvswitch_agent

Returns:
+ echo 'Running command: '\''neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/
Running command: 'neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plu
+ exec neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml
/var/lib/kolla/venv/lib/python2.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package ease use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binar
  """)
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/queue.py", line 118, in switch
    self.greenlet.switch(value)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/ryu/lib/hub.py", line 64, in _launch
    raise e
SystemExit: 1

Revision history for this message
Eric Miller (erickmiller) wrote :

In case this helps anyone - restarting the neutron_server container on all controllers, and then restarting the neutron_openvswitch_agent container on all compute nodes, solved the issue.

Some other services, such as nova_compute, had the same issue. Restarting all nova services on all controllers, and then nova services on all compute nodes, solved the issue.

Eric

Changed in kolla:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.