OVN deployment is failing when used with OVS 2.10

[Numan Siddique]

When we deploy tripleo with latest OVS 2.10, we see the below error

/var/lib/docker-puppet/docker-puppet.sh --net host --volume /etc/hosts:/etc/hosts:ro 192.168.24.1:8787/rhosp14/openstack-nova-compute:2018-08-20.1",
        "2018-08-23 06:31:27,357 ERROR: 19154 -- Failed running docker-puppet.py for ovn_controller",

puppet apply for ovn_controller is failing with the below error

***
+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron,vs_config,exec /etc/config.pp
PMD: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
PMD: net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)
PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4)
PMD: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
PMD: net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)
PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4)
Warning: Facter: Could not retrieve fact='nic_alias', resolution='<anonymous>': Could not execute '/usr/bin/os-net-config -i': command not found
Warning: Facter: Could not retrieve fact='nic_alias', resolution='<anonymous>': Could not execute '/usr/bin/os-net-config -i': command not found
Warning: Undefined variable 'deploy_config_name';
   (file & line not available)
Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
Notice: Scope(Class[Ovn::Controller]): Hi Numan How are you : ovn remote = tcp:172.17.1.27:6642 : encap ip = 172.17.2.19
Warning: This method is deprecated, please use the stdlib validate_legacy function,
                    with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/ovn/manifests/controller.pp", 62]:["/etc/config.pp", 2]
   (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation')
Notice: Compiled catalog for controller-0 in environment production in 0.70 seconds
Error: Failed to apply catalog: undefined method `chomp' for nil:NilClass

***

[Numan Siddique (numansiddique)]

The reason for the failure is because of the warning messages which "ovs-vsctl" displays when puppet vswitch executes the command "/usr/bin/ovs-vsctl list Open_vSwitch .".

***
PMD: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
PMD: net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)
PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4)
PMD: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
PMD: net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)
PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4)
***

Because of these warning messages, the code here - https://github.com/openstack/puppet-vswitch/blob/master/lib/puppet/provider/vs_config/ovs.rb#L41 is Nil.

Ideally ovs-vsctl shouldn't display these warning messages. As a workaround we need to mount "/usr/lib64" to the ovn_controller container during puppet confi step.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-vswitch 9.3.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-vswitch 10.1.0 release.

[David Casier (david-casier)]

(undercloud) [stack@dir-tripleo ~]$ docker pull tripleomaster/centos-binary-ovn-controller:current-tripleo
(undercloud) [stack@dir-tripleo ~]$ docker run -it --rm --entrypoint sh 2a608ca05b95
()[root@f2638fae50eb /]$ rpm -qa|grep puppet-vswitch
puppet-vswitch-10.2.1-0.20190116224210.98fbd0d.el7.noarch
()[root@f2638fae50eb /]$ ovs-vsctl show > /dev/null
PMD: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
PMD: net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)
PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4)
ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-vswitch queens-eol release.