VM can't send packet through vhostuser port due to missing numa settings in domain xml
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Invalid
|
High
|
Chris Friesen |
Bug Description
Title
-----
VM can't send packet through vhostuser port
Brief Description
-----------------
VM can't send packet through vhostuser port. During the booting up VM, VM will contact DHCP server to get an IP address. However packet can't go through vhostuser port, VM can't get the IP address.
Severity
--------
Critical
Steps to Reproduce
------------------
1. Install Standard 2+2 on 4 bare metals
Guide: https:/
2. Using the scripts from the guide to create network, subnet and router.
3. Download cirros image on controller-0
wget http://
4. Upload cirros image
openstack image create --container-format bare --disk-format qcow2 --file cirros-
5. Create VM
openstack server create --image cirros --flavor m1.tiny --network public-net0 vm1
6. Check which host vm1 is running
openstack server show vm1
7. On the host where vm1 is running, log in the vm1 and check the IP
sudo virsh list
sudo virsh console $num_vm1
login vm1: cirros, cubswin:)
sudo ifconfig
Expected Behavior
------------------
An IP address has been assigned to interface eth0
Actual Behavior
----------------
No IP address assigned to interface eth0
System Configuration
-------
System mode: Standard 2+2 on bare metals
Reproducibility
---------------
100%
Branch/Pull Time/Commit
-------
master as of 20190305T060000Z
Timestamp/Logs
--------------
compute-
Id Name State
-------
18 instance-00000037 running
21 cirros-ovs-dpdk running
22 instance-0000003d running
compute-
Connected to domain instance-0000003d
Escape character is ^]
Sending discover...
Sending discover...
Usage: /sbin/cirros-dhcpc <up|down>
No lease, failing
WARN: /etc/rc3.
cirros-ds 'net' up at 181.29
checking http://
failed 1/20: up 181.30. request failed
failed 2/20: up 183.45. request failed
failed 3/20: up 185.45. request failed
failed 4/20: up 187.46. request failed
failed 5/20: up 189.46. request failed
failed 6/20: up 191.47. request failed
failed 7/20: up 193.47. request failed
failed 8/20: up 195.48. request failed
failed 9/20: up 197.48. request failed
failed 10/20: up 199.49. request failed
failed 11/20: up 201.49. request failed
failed 12/20: up 203.49. request failed
failed 13/20: up 205.50. request failed
failed 14/20: up 207.50. request failed
failed 15/20: up 209.51. request failed
failed 16/20: up 211.51. request failed
failed 17/20: up 213.52. request failed
failed 18/20: up 215.52. request failed
failed 19/20: up 217.53. request failed
failed 20/20: up 219.53. request failed
failed to read iid from metadata. tried 20
no results found for mode=net. up 221.54. searched: nocloud configdrive ec2
failed to get instance-id of datasource
Starting dropbear sshd: generating rsa key... generating dsa key... OK
=== system information ===
Platform: OpenStack Foundation OpenStack Nova
Container: none
Arch: x86_64
CPU(s): 1 @ 2693.508 MHz
Cores/Sockets/
Virt-type:
RAM Size: 491MB
Disks:
NAME MAJ:MIN SIZE LABEL MOUNTPOINT
vda 253:0 1073741824
vda1 253:1 1061061120 cirros-rootfs /
=== sshd host keys ===
-----BEGIN SSH HOST KEY KEYS-----
ssh-rsa AAAAB3NzaC1yc2E
ssh-dss AAAAB3NzaC1kc3M
-----END SSH HOST KEY KEYS-----
=== network info ===
if-info: lo,up,127.
if-info: eth0,up,
=== datasource: None None ===
=== cirros: current=0.3.4 uptime=221.75 ===
route: fscanf
=== pinging gateway failed, debugging connection ===
############ debug start ##############
### /etc/init.d/sshd start
Starting dropbear sshd: OK
route: fscanf
### ifconfig -a
eth0 Link encap:Ethernet HWaddr FA:16:3E:54:33:CA
inet6 addr: fe80::f816:
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:0 (0.0 B) TX bytes:1132 (1.1 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:12 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:1020 (1020.0 B) TX bytes:1020 (1020.0 B)
### route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
route: fscanf
### cat /etc/resolv.conf
cat: can't open '/etc/resolv.conf': No such file or directory
### gateway not found
/sbin/cirros-
### pinging nameservers
### uname -a
Linux cirros 3.2.0-80-virtual #116-Ubuntu SMP Mon Mar 23 17:28:52 UTC 2015 x86_64 GNU/Linux
### lsmod
Module Size Used by Not tainted
nls_iso8859_1 12713 0
nls_cp437 16991 0
vfat 17585 0
fat 61512 1 vfat
isofs 40259 0
ip_tables 27473 0
x_tables 29891 1 ip_tables
pcnet32 42119 0
8139cp 27360 0
ne2k_pci 13691 0
8390 18856 1 ne2k_pci
e1000 108589 0
acpiphp 24231 0
### dmesg | tail
[ 0.970394] acpiphp: Slot [30] registered
[ 0.970405] acpiphp: Slot [31] registered
[ 0.979866] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[ 0.979868] e1000: Copyright (c) 1999-2006 Intel Corporation.
[ 0.988386] ne2k-pci.c:v1.03 9/22/2003 D. Becker/P. Gortmaker
[ 0.994725] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004)
[ 0.999723] pcnet32: pcnet32.c:v1.35 21.Apr.2008 <email address hidden>
[ 1.006790] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 1.364080] Refined TSC clocksource calibration: 2693.563 MHz.
[ 11.344086] eth0: no IPv6 routers present
### tail -n 25 /var/log/messages
Mar 16 17:47:28 cirros kern.info kernel: [ 0.000000] Centaur CentaurHauls
Mar 16 17:47:28 cirros kern.info kernel: [ 0.000000] BIOS-provided physical RAM map:
Mar 16 17:47:28 cirros kern.info kernel: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
Mar 16 17:47:28 cirros kern.info kernel: [ 0.900022] usb 1-1: new full-speed USB device number 2 using uhci_hcd
Mar 16 17:47:28 cirros kern.info kernel: [ 0.927882] EXT3-fs (vda1): using internal journal
Mar 16 17:47:28 cirros kern.info kernel: [ 0.970055] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Mar 16 17:47:28 cirros kern.info kernel: [ 0.970116] acpiphp: Slot [3] registered
Mar 16 17:47:28 cirros kern.info kernel: [ 0.970127] acpiphp: Slot [4] registered
Mar 16 17:47:28 cirros kern.info kernel: [ 0.970138] acpiphp: Slot [5] registered
Mar 16 17:47:28 cirros kern.info kernel: [ 0.970148] acpiphp: Slot [6] registered
Mar 16 17:47:28 cirros kern.info kernel: [ 0.970158] acpiphp: Slot [7] registered
Mar 16 17:47:28 cirros kern.info kernel: [ 0.970168] acpiphp: Slot [8] registered
Mar 16 17:47:28 cirros kern.info kernel: [ 0.970178] acpiphp: Slot [9] registered
Mar 16 17:47:28 cirros kern.info kernel: [ 0.970188] acpiphp: Slot [10] registered
Mar 16 17:47:28 cirros kern.info kernel: [ 0.970198] acpiphp: Slot [11] registered
Mar 16 17:47:28 cirros kern.info kernel: [ 0.970209] acpiphp: Slot [12] registered
Mar 16 17:47:28 cirros kern.info kernel: [ 0.979866] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
Mar 16 17:47:28 cirros kern.info kernel: [ 0.979868] e1000: Copyright (c) 1999-2006 Intel Corporation.
Mar 16 17:47:28 cirros kern.info kernel: [ 0.988386] ne2k-pci.c:v1.03 9/22/2003 D. Becker/P. Gortmaker
Mar 16 17:47:28 cirros kern.info kernel: [ 0.994725] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004)
Mar 16 17:47:28 cirros kern.info kernel: [ 0.999723] pcnet32: pcnet32.c:v1.35 21.Apr.2008 <email address hidden>
Mar 16 17:47:28 cirros kern.info kernel: [ 1.006790] ip_tables: (C) 2000-2006 Netfilter Core Team
Mar 16 17:47:29 cirros kern.info kernel: [ 1.364080] Refined TSC clocksource calibration: 2693.563 MHz.
Mar 16 17:47:39 cirros kern.debug kernel: [ 11.344086] eth0: no IPv6 routers present
Mar 16 17:51:09 cirros authpriv.info dropbear[301]: Running in background
############ debug end ##############
____ ____ ____
/ __/ __ ____ ____ / __ \/ __/
/ /__ / // __// __// /_/ /\ \
\___//_//_/ /_/ \____/___/
http://
login as 'cirros' user. default password: 'cubswin:)'. use 'sudo' for root.
cirros login: cirros
Password:
$ ifconfig
eth0 Link encap:Ethernet HWaddr FA:16:3E:54:33:CA
inet6 addr: fe80::f816:
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:0 (0.0 B) TX bytes:1132 (1.1 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:12 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:1020 (1020.0 B) TX bytes:1020 (1020.0 B)
$
summary: |
- vhostuser port on br-int doesn't work + Lacking numa related sections in domain xml files |
description: | updated |
summary: |
- Lacking numa related sections in domain xml files + VM can't send packet through vhostuser port |
tags: | added: stx.networking |
summary: |
- VM can't send packet through vhostuser port + VM can't send packet through vhostuser port due to missing numa settings + in VM xml |
summary: |
VM can't send packet through vhostuser port due to missing numa settings - in VM xml + in domain xml |
Changed in starlingx: | |
importance: | Undecided → High |
Changed in starlingx: | |
assignee: | nobody → Chris Friesen (cbf123) |
Changed in starlingx: | |
status: | New → In Progress |
tags: | added: stx.2.0 |
We manually assigned correct IP to the vm1 and ping DHCP server in vm1. Ping fails and no packet can be captured by tcpdump.
1. Check the IP assigned to the VM:
openstack server show vm1
2. Manually assign IP to VM:
sudo ifconfig 192.168.101.108/24 up
3. Check DHCP namespace exist and DHCP server address
sudo ip netns
sudo ip netns exec $dhcp_namespace ifconfig
3. tcpdump in dhcp namespace:
sudo ip netns exec $dhcp_namespace tcpdump –i $tap_device
4. Ping DHCP server in VM:
ping 192.168.101.2
Based on above analysis, the vhostuser port doesn't work.