ERROR neutron.plugins.ml2.managers [-] No type driver for tenant network_type: vxlan.

[Jose Gaitan]

A new installation of Openstack-Ansible completes succesfully. Unfortunately, upon testing services, we noticed Horizon is not working and displays an error message: "Something went wrong!".

Upon further investigation we see an error in the journalctl -xe logs from the neutron LXC container:

"ERROR neutron.plugins.ml2.managers [-] No type driver for tenant network_type: vxlan. Service terminated!"

Deployment Environment:
-3x infra nodes
-2x compute nodes
-Debian 11 - fresh install on all nodes
-Clean install of OpenStack Ansible

Can further advise be provided on how to overcome this, either manually or by changing user variables before redeploying the playbooks?

See attachment with relevant configuration files content.

Thank you.

[Dmitriy Rabotyagov (noonedeadpunk)]

Hi, Roger.

Can you kindly specify what version of OpenStack-Ansible are you running? Also I assume you're reffering to the admin part of the interface? As user part seems to work properly at least on master and stable/zed branches.

Also, in attached config, we've spotted that `tenant_network_types` is defined incorrectly for OVN driver.

I assume, that's due to defining vxlan type in openstack_user_config, while OVN does have geneve instead. So in order to get neutron working, you can either to change type of tunnel network from vxlan to geneve or define neutron_provider_networks like mentioned here: https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html#openstack-ansible-user-variables

[Dmitriy Rabotyagov (noonedeadpunk)]

I've also pushed change to close issue in horizon: https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/870801

[Jose Gaitan (vchjgaitan)]

Hello Dimitry,

I appreciate your prompt response. We deployed OpenStack-Ansible from the master branch (26.1.0.dev45). It is also worth noting that not only the admin part of the interface is not working, when we issue the "openstack network agent list" command from the utility_container it returns the following error:

===
HttpException: 503: Server Error for url: http://10.10.36.9:9696/v2.0/agents, 503 Service Unavailable: No server is available to handle this request.
===

Isn't OVN the default neutron driver in Zed? According to the documentation, that seems to be the case, hence we wanted to leave everything as default to avoid conflicts.

We have added the vxlan type_drivers in the /etc/neutron/plugins/ml2/ml2_conf.ini file and that has not help. In fact it brings new errors upon rebooting the neutron container:

The updated ml2_conf.ini is as follows:
===
[ml2]
type_drivers = geneve,vxlan,vlan,flat
tenant_network_types = vxlan,flat,vlan
mechanism_drivers = ovn
extension_drivers = port_security
# ML2 flat networks

[ml2_type_flat]
flat_networks = flat
# ML2 VLAN networks

[ml2_type_vlan]
network_vlan_ranges = vlan:101:200,vlan:301:400
# ML2 VXLAN networks

[ml2_type_vxlan]
vxlan_group = 239.1.1.1
vni_ranges = 1:1000

[ml2_type_geneve]
vni_ranges =
max_header_size = 38
# Security groups

[securitygroup]
enable_security_group = True
enable_ipset = True
===

After updating the ml2_conf.ini file and rebooting the container, we get the following error:

===
2023-01-16 22:07:16.429 84 ERROR neutron File "/openstack/venvs/neutron-26.1.0.dev45/lib/python3.9/site-packages/ovs/socket_util.py", line 221, in inet_parse_active
2023-01-16 22:07:16.429 84 ERROR neutron raise ValueError("%s: bad peer name format" % target)
2023-01-16 22:07:16.429 84 ERROR neutron ValueError: :6642: bad peer name format
2023-01-16 22:07:16.429 84 ERROR neutron
Jan 16 22:07:16 inf1-mia-neutron-server-container-cee43d59 systemd[1]: neutron-server.service: Main process exited, code=exited, status=1/FAILURE
===

I also think is worth noting that all neutron backends in HAProxy are down which seems the entire infrastructure is crippled by this. Keepalived has correctly set the VIP and all other services except neutron, are reachable as it is down:

===
>>> neutron_server-back
..tainer-2e7be461 1 DOWN L4CON 1 0 0 0 0 0 0 0
..tainer-7c4ce35a 1 DOWN L4CON 1 0 0 0 0 0 0 0
..tainer-9e17c650 1 DOWN L4CON 1 0 0 0 0 0 0 0
BACKEND 0 DOWN 0 0 0 0 0 0 820 0
===

I hope this helps narrow down the culprit and provides guidance on what settings should be updated and how, if needed.

Thank you,

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/870801
Committed: https://opendev.org/openstack/openstack-ansible-os_horizon/commit/e61dab9a05be476c442ec667be81d7bbd774777f
Submitter: "Zuul (22348)"
Branch: master

commit e61dab9a05be476c442ec667be81d7bbd774777f
Author: Dmitriy Rabotyagov <email address hidden>
Date: Tue Jan 17 13:53:50 2023 +0100

    Allow to override supported_provider_types

    Supported ML2 provided types depends on the ML2 driver
    and we should make it configurable in order to reflect dropdown list
    that appears for admin panel while creating a network.

    Closes-Bug: #2002897
    Change-Id: Iceedf6af9559d48c28e0ee782a44f9ceb480119d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/870779

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/870779
Committed: https://opendev.org/openstack/openstack-ansible-os_horizon/commit/f7514573f51b0f75b5a336cdeaa3d368894377be
Submitter: "Zuul (22348)"
Branch: stable/zed

commit f7514573f51b0f75b5a336cdeaa3d368894377be
Author: Dmitriy Rabotyagov <email address hidden>
Date: Tue Jan 17 13:53:50 2023 +0100

    Allow to override supported_provider_types

    Supported ML2 provided types depends on the ML2 driver
    and we should make it configurable in order to reflect dropdown list
    that appears for admin panel while creating a network.

    Closes-Bug: #2002897
    Change-Id: Iceedf6af9559d48c28e0ee782a44f9ceb480119d
    (cherry picked from commit e61dab9a05be476c442ec667be81d7bbd774777f)

[jmhal (jmarcelo-alencar)]

I have the exact same problem. Any idea how to fix it?

Adding this:

neutron_plugin_type: ml2.ovn
neutron_plugin_base:
  - ovn-router
neutron_ml2_drivers_type: "vxlan,vlan,local,geneve,flat"

to /etc/openstack_deploy/user_variables.yml would solve the issue?

[Dmitriy Rabotyagov (noonedeadpunk)]

Hi,

While there's a way to have vxlan networks in ovn, this path is not recommended, as there's way lower limitation on amount of networks that can be spawned when vxlan is used. Also, it can't be used with geneve.

So my suggestion would be to remove vxlan out of neutron_ml2_drivers_type and leave vlan, geneve and flat, as it's recommend way to use geneve for tenant networks with OVN.

[jmhal (jmarcelo-alencar)]

Thank you Dmitriy.

I am trying to replicate this test environment: https://docs.openstack.org/openstack-ansible/latest/user/test/example.html#. I have the same scenario, two machines, one network interface each. The only difference is that the machines have more RAM. I also create the VLANs with the same CIDR e VLAN IDs. When I run the playbooks, I am getting the exact ERROR that Roger Rivera describes above, but I am using Ubuntu 22.04 instead of Debian.

I find odd that the file /etc/openstack_deploy/openstack_user_config.yml provided does not mention OpenvSwitch:

    - network:
        container_bridge: "br-vxlan"
        container_type: "veth"
        container_interface: "eth10"
        ip_from_q: "tunnel"
        type: "vxlan"
        range: "1:1000"
        net_name: "vxlan"
        group_binds:
          - neutron_linuxbridge_agent
    - network:
        container_bridge: "br-vlan"
        container_type: "veth"
        container_interface: "eth12"
        host_bind_override: "eth12"
        type: "flat"
        net_name: "flat"
        group_binds:
          - neutron_linuxbridge_agent

All the provider_networks are bond to the linuxbridge agent. Even so, the neutron_server container has OpenvSwitch in the configuration files. Since I just want the example environment up and running so that I can gradually improve it, should I remove br-vxlan and associated provider networks? Also, reinforce "neutron_ml2_drivers_type: "vlan,local,geneve,flat" in /etc/openstack_deploy/user_variables.yml ?

[Jose Gaitan (vchjgaitan)]

Hello jmhal,

We opted to move over to the new OVN provider. This solved our issues and left the deprecated LinuxBrdige driver outside of the equation. Also, VXLAN was replaced with Geneve. Relevant configuration files were adjusted as follows:

# user_variables.yml
# The new default neutron ML2 driver is OVN:
neutron_plugin_type: ml2.ovn
neutron_plugin_base:
  - neutron.services.ovn_l3.plugin.OVNL3RouterPlugin
neutron_ml2_drivers_type: "vlan,local,geneve"

# openstack_user_config.yml
...
    - network:
        container_bridge: br-vxlan
        container_type: veth
        container_interface: eth10
        ip_from_q: tunnel
        type: geneve
        range: '1:1000'
        net_name: geneve
        group_binds:
          - neutron_ovn_controller
    - network:
        container_bridge: "br-provider"
        container_type: "veth"
        type: "vlan"
        range: "101:200"
        net_name: "vlan"
        network_interface: "br-vlan"
        group_binds:
          - neutron_ovn_controller
...

# /etc/openstack_deploy/env.d/neutron.yml
component_skel:
  neutron_ovn_controller:
    belongs_to:
      - neutron_all
  neutron_ovn_northd:
    belongs_to:
      - neutron_all

container_skel:
  neutron_agents_container:
    contains: {}
  neutron_ovn_northd_container:
    belongs_to:
      - network_containers
    contains:
      - neutron_ovn_northd

# /etc/openstack_deploy/env.d/nova.yml
container_skel:
  nova_compute_container:
    belongs_to:
      - compute_containers
      - kvm-compute_containers
      - lxd-compute_containers
      - qemu-compute_containers
    contains:
      - neutron_ovn_controller
      - nova_compute
    properties:
      is_metal: true

# /etc/openstack_deploy/group_vars/network_hosts
openstack_host_specific_kernel_modules:
  - name: "openvswitch"
    pattern: "CONFIG_OPENVSWITCH"

I hope this helps. But it ultimately depends on your network and environment requirements.

Regards,

Roger

[jmhal (jmarcelo-alencar)]

Hello Roger Rivera,

Thank you very much, Horizon is working. First, I did as you explained, but I still got the following error:

ERROR neutron.plugins.ml2.managers [-] No type driver for tenant network_type: flat. Service terminated!

So I manually edit /etc/neutron/plugins/ml2/ml2_conf.ini on the infra1_neutron container, from this:

tenant_network_types = geneve,flat,vlan

To this:

tenant_network_types = geneve,vlan

I believe this happens because I kept the following provider network:

- network:
        container_bridge: "br-vlan"
        container_type: "veth"
        container_interface: "eth12"
        host_bind_override: "eth12"
        type: "flat"
        net_name: "flat"
        group_binds:
          - neutron_ovn_controller

Later, I will test a new deployment without it. But for now it is working.

Best Regards.

[Dmitriy Rabotyagov (noonedeadpunk)]

Yes, we have indeed switched default but we're short in time to clean up and re-write surrounding documentation.
But since lxb was moved to experimental we could not use it as default anymore.

[Jose Gaitan (vchjgaitan)]

Hello,

The Jinja expression responsible for creating the 'supported_provider_types' in Horizon is generating a string and not the expected Python list.

Upon attempting to create a provider network on Horizon, it displays an error. The following error is logged by apache2:

“Undefined provider network types are found: … ”

Further inspecting the /etc/horizon/local_settings.py file we found the following INI-style comma-separated string:

'supported_provider_types': "vlan,flat,geneve",

Wouldn't the correct expected value be a Python list, like this:

'supported_provider_types': ['vlan', 'flat', 'geneve'],

Where /etc/neutron/plugins/ml2/ml2_conf.ini contains:

[ml2]
type_drivers = vlan,flat,geneve

A suggestion could be to convert the INI-style key-value separated values of the ML2 plugin type_drivers to a Python list. It seems the Jinja template responsible to generate the python list is located at templates/horizon_local_settings.py.j2 from openstack-ansible-os_horizon and contains:

'supported_provider_types': {{ horizon_network_provider_types | to_json }}

Do we need to fix this template so that the correct Python list is passed onto the /etc/horizon/local_settings.py file?

Tested environments:
-Ubuntu 22
-Openstack Zed and Antelope clusters.

[Dmitriy Rabotyagov (noonedeadpunk)]

This is known issue, which has been already merged upstream and will be included in the next release.
Patch for the issue is below:
https://review.opendev.org/q/Ida72c712153dcda4cd06e0959f98ade4fee8dfbd

We expect new versions to be live during next week

[Jose Gaitan (vchjgaitan)]

Hi Dmitriy,

That's great news. Thank you for the update.