vpp_papi.vpp_papi.VPPIOError seen on bringing up devstack

[suresh]

I have ubuntu running 20.04 with devstack cloned from stable/victoria. I see that devstack installation failed with below Traceback. I am unable below a instance as port binding is failing.

Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent Traceback (most recent call last):
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent File "/usr/local/bin/vpp-agent", line 10, in <module>
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent sys.exit(main())
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent File "/opt/stack/networking-vpp/networking_vpp/agent/server.py", line 3701, in main
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent vppf = VPPForwarder(physnets,
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent File "/opt/stack/networking-vpp/networking_vpp/agent/server.py", line 313, in __init__
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent self.vpp = vpp.VPPInterface(LOG, vpp_cmd_queue_len)
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent File "/opt/stack/networking-vpp/networking_vpp/agent/vpp.py", line 124, in getinstance
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent instances[cls] = cls(*args, **kwargs) # type: ignore
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent File "/opt/stack/networking-vpp/networking_vpp/agent/vpp.py", line 178, in __init__
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent self._vpp.connect("python-VPPInterface", **args)
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent File "/usr/lib/python3/dist-packages/vpp_papi/vpp_papi.py", line 516, in connect
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent return self.connect_internal(name, msg_handler, chroot_prefix, rx_qlen,
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent File "/usr/lib/python3/dist-packages/vpp_papi/vpp_papi.py", line 488, in connect_internal
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent raise VPPIOError(2, 'Connect failed')
Oct 07 00:28:33 gna200405 vpp-agent[28635]: ERROR vpp_agent vpp_papi.vpp_papi.VPPIOError: [Errno 2] Connect failed

stack@xxxx:~/devstack$ grep "^[^#;]" local.conf
[[local|localrc]]
ADMIN_PASSWORD=gnalab
DATABASE_PASSWORD=gnalab
RABBIT_PASSWORD=gnalab
SERVICE_PASSWORD=gnalab
HOST_IP=10.195.234.68
NR_HUGEPAGES=2500
disable_service cinder c-sch c-api c-vol
disable_service tempest
disable_service n-net
enable_service q-svc q-dhcp q-l3 q-meta
disable_service q-agt
enable_plugin networking-vpp https://opendev.org/x/networking-vpp
Q_PLUGIN=ml2
Q_USE_SECGROUP=True
Q_ML2_PLUGIN_MECHANISM_DRIVERS=vpp
Q_ML2_PLUGIN_TYPE_DRIVERS=vlan,flat,vxlan
Q_ML2_TENANT_NETWORK_TYPE=vxlan
Q_ML2_PLUGIN_VXLAN_TYPE_OPTIONS=(vni_ranges=10000:12000)
ML2_VLAN_RANGES=physnet:100:200
MECH_VPP_PHYSNETLIST=physnet:TenGigabitEthernet2/0/1
VPP_STARTUP_CONFIG=/opt/stack/startup.conf
[[post-config|$NOVA_CONF]]
[DEFAULT]
reserved_huge_pages=node:0,size:2048,count:64
[[post-config|/$Q_PLUGIN_CONF_FILE]]
[vxlan]
l2_population = True
enable_vxlan = True
local_ip = 172.16.1.3
LOGFILE=$DEST/logs/stack.sh.log
LOGDAYS=2

stack@xxx:~/devstack$ sudo vppctl show interface
              Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count
BondEthernet0 6 down 9000/0/0/0
TenGigabitEthernet2/0/0 1 up 9000/0/0/0 tx packets 203
                                                                    tx bytes 25172
TenGigabitEthernet2/0/1 2 up 9000/0/0/0
TenGigabitEthernet6/0/0 3 up 9000/0/0/0 tx packets 203
                                                                    tx bytes 25172
TenGigabitEthernet6/0/1 4 up 9000/0/0/0
local0 0 down 0/0/0/0
loop0 5 up 9000/0/0/0

cat /etc/vpp/startup.conf
unix {
  nodaemon
  log /tmp/vpp.log
  full-coredump
  startup-config /etc/vpp-startup.conf
  cli-listen /run/vpp/cli.sock
  gid vpp
}

api-trace {
  on
}

api-segment {
  gid vpp
}

socksvr {
  default
}

dpdk {
    dev default {
    num-rx-queues 1
    num-tx-queues 2
    }
    uio-driver vfio-pci
    dev 0000:02:00.0
    dev 0000:02:00.1
    dev 0000:06:00.0
    dev 0000:06:00.1
# socket-mem 128
}