"ip link delete" returns "RTNETLINK answers: Operation not supported" - OpenStack Juno is completely broken in Trusty + Linux 3.19.

[Thiago Martins]

Guys,

 I just upgraded my Trusty with "linux-generic-lts-vivid" that is available in proposed repository.

 OpenStack is completely broken now. I'm using it with VXLAN + OpenvSwitch. All Tenants are offline.

 I'm seeing the following error at Network Node (Neutron L3 Router):

---
==> /var/log/neutron/ovs-cleanup.log <==
2015-05-07 13:30:57.384 881 TRACE neutron File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 86, in _execute
2015-05-07 13:30:57.384 881 TRACE neutron log_fail_as_error=log_fail_as_error)
2015-05-07 13:30:57.384 881 TRACE neutron File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 84, in execute
2015-05-07 13:30:57.384 881 TRACE neutron raise RuntimeError(m)
2015-05-07 13:30:57.384 881 TRACE neutron RuntimeError:
2015-05-07 13:30:57.384 881 TRACE neutron Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'delete', 'tap3da8f4fe-55']
2015-05-07 13:30:57.384 881 TRACE neutron Exit code: 2
2015-05-07 13:30:57.384 881 TRACE neutron Stdout: ''
2015-05-07 13:30:57.384 881 TRACE neutron Stderr: 'RTNETLINK answers: Operation not supported\n'
2015-05-07 13:30:57.384 881 TRACE neutron
---

It also doesn't work manually:

---
root@network-1:~# ip link delete tap3da8f4fe-55
RTNETLINK answers: Operation not supported
---

dmesg shows this:

---
[ 152.307822] Modules linked in: vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch libcrc32c snd_hda_codec_generic iosf_mbi snd_hda_intel crct10dif_pclmul crc32_pclmul snd_hda_controller ghash_clmulni_intel snd_hda_codec aesni_intel ppdev snd_hwdep snd_pcm aes_x86_64 qxl lrw snd_timer gf128mul snd ttm glue_helper drm_kms_helper soundcore i2c_piix4 ablk_helper drm serio_raw cryptd pvpanic 8250_fintek parport_pc mac_hid lp parport psmouse pata_acpi floppy
[ 152.307893] CPU: 0 PID: 1067 Comm: ovs-vswitchd Not tainted 3.19.0-16-generic #16~14.04.2-Ubuntu
[ 152.307901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[ 152.307906] task: ffff880036aee220 ti: ffff880079eb4000 task.ti: ffff880079eb4000
[ 152.307912] RIP: 0010:[<ffffffffc05fd901>] [<ffffffffc05fd901>] ovs_vport_del+0x41/0x60 [openvswitch]
[ 152.307923] RSP: 0018:ffff880079eb79a8 EFLAGS: 00010296
[ 152.307929] RAX: 8000000000000025 RBX: ffff880079f04e40 RCX: 00000001810000ec
[ 152.307934] RDX: 00000001810000ed RSI: ffffea0000d860c0 RDI: 0000000040000000
[ 152.307938] RBP: ffff880079eb79b8 R08: ffff8800361834a0 R09: 00000001810000ec
[ 152.307943] R10: ffffea0000d860c0 R11: ffffffff813a7f65 R12: ffff880079f04e40
[ 152.307948] R13: ffff880079e97100 R14: 00000000ffffffea R15: ffff880079e30180
[ 152.307955] FS: 00007fc5f96a5980(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
[ 152.307961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 152.307966] CR2: 00007f32854bd740 CR3: 000000007a032000 CR4: 00000000000406f0
[ 152.307972] Stack:
[ 152.307975] ffff880079eb7a30 ffff880079eb7a30 ffff880079eb79c8 ffffffffc05f6380
[ 152.307984] ffff880079eb79f8 ffffffffc05f6455 ffffffffc05ff268 ffffffffc0602040
[ 152.307992] ffff88007a700814 ffff880079e97000 ffff880079eb7aa8 ffffffff816db0e5
[ 152.308003] Call Trace:
[ 152.308012] [<ffffffffc05f6380>] ovs_dp_detach_port+0x30/0x40 [openvswitch]
[ 152.308019] [<ffffffffc05f6455>] ovs_vport_cmd_del+0xc5/0x110 [openvswitch]
[ 152.308034] [<ffffffff816db0e5>] genl_family_rcv_msg+0x1a5/0x3d0
[ 152.308040] [<ffffffff816db310>] ? genl_family_rcv_msg+0x3d0/0x3d0
[ 152.308046] [<ffffffff816db3a1>] genl_rcv_msg+0x91/0xd0
[ 152.308051] [<ffffffff816d9191>] netlink_rcv_skb+0xc1/0xe0
[ 152.308056] [<ffffffff816d983c>] genl_rcv+0x2c/0x40
[ 152.308060] [<ffffffff816d8816>] netlink_unicast+0xf6/0x200
[ 152.308065] [<ffffffff816d8c33>] netlink_sendmsg+0x313/0x690
[ 152.308078] [<ffffffff81350fe0>] ? aa_sk_perm.isra.4+0x70/0x150
[ 152.308090] [<ffffffff8168e34c>] do_sock_sendmsg+0x8c/0x100
[ 152.308096] [<ffffffff8168dd0e>] ? copy_msghdr_from_user+0x15e/0x1f0
[ 152.308103] [<ffffffff8168e963>] ___sys_sendmsg+0x313/0x320
[ 152.308112] [<ffffffff8120713c>] ? destroy_inode+0x3c/0x70
[ 152.308118] [<ffffffff81207287>] ? evict+0x117/0x190
[ 152.308122] [<ffffffff8120230f>] ? dentry_free+0x5f/0xa0
[ 152.308127] [<ffffffff81202c64>] ? __dentry_kill+0x154/0x1f0
[ 152.308132] [<ffffffff81202da5>] ? dput+0xa5/0x170
[ 152.308138] [<ffffffff8120bdc4>] ? mntput+0x24/0x40
[ 152.308143] [<ffffffff8168f052>] __sys_sendmsg+0x42/0x80
[ 152.308148] [<ffffffff8168f0a2>] SyS_sendmsg+0x12/0x20
[ 152.308156] [<ffffffff817b520d>] system_call_fastpath+0x16/0x1b
[ 152.308208] Code: 8b 57 30 48 85 c0 48 89 02 74 04 48 89 50 08 48 b8 00 02 20 00 00 00 ad de 48 89 df 48 89 43 30 48 8b 43 48 ff 50 10 48 8b 43 48 <48> 8b 78 40 e8 d6 55 af c0 48 83 c4 08 5b 5d c3 66 66 66 66 66
[ 152.308375] RIP [<ffffffffc05fd901>] ovs_vport_del+0x41/0x60 [openvswitch]
[ 152.308433] RSP <ffff880079eb79a8>
[ 152.308504] ---[ end trace cc9561be41fd191a ]---
[ 217.847530] init: neutron-ovs-cleanup pre-start process (871) terminated with status 1
[ 217.913947] init: wait-for-state (neutron-dhcp-agentneutron-ovs-cleanup) main process (862) killed by TERM signal
---

Falling back to Linux 3.16.

Info:

root@network-1:~# cat /proc/version_signature > version.log
root@network-1:~# cat version.log
Ubuntu 3.19.0-16.16~14.04.2-generic 3.19.3

Thanks,
Thiago
---
AlsaVersion: Advanced Linux Sound Architecture Driver Version k3.19.0-16-generic.
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.10
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/controlC0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Card0.Amixer.info: Error: [Errno 2] No such file or directory
Card0.Amixer.values: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
HibernationDevice: RESUME=/dev/mapper/vg01-swap_1
InstallationDate: Installed on 2015-04-09 (27 days ago)
InstallationMedia: Ubuntu-Server 14.04.2 LTS "Trusty Tahr" - Release amd64 (20150218.1)
IwConfig: Error: [Errno 2] No such file or directory
Lsusb:
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
Package: linux (not installed)
ProcFB: 0 qxldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.19.0-16-generic root=/dev/mapper/vg01-root ro
ProcVersionSignature: Ubuntu 3.19.0-16.16~14.04.2-generic 3.19.3
RelatedPackageVersions:
 linux-restricted-modules-3.19.0-16-generic N/A
 linux-backports-modules-3.19.0-16-generic N/A
 linux-firmware 1.127.11
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty
Uname: Linux 3.19.0-16-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 01/01/2011
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2011:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-trusty:cvnBochs:ct1:cvr:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-trusty
dmi.sys.vendor: QEMU

[Thiago Martins (martinx)]

lspci-vnvn.log

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1452868

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Thiago Martins (martinx)]

apport information

[Thiago Martins (martinx)]

apport information

[Thiago Martins (martinx)]

apport information

[Thiago Martins (martinx)]

apport information

[Thiago Martins (martinx)]

apport information

[Thiago Martins (martinx)]

apport information

[Thiago Martins (martinx)]

apport information

[Thiago Martins (martinx)]

apport information

[Thiago Martins (martinx)]

apport information

[Thiago Martins (martinx)]

apport information

[Thiago Martins (martinx)]

apport information

[Thiago Martins (martinx)]

apport information

[Thiago Martins (martinx)]

apport information

[Thiago Martins (martinx)]

apport information

[Chris J Arges (arges)]

I strongly suspect:
commit fa2d8ff4e3522b4e05f590575d3eb8087f3a8cdc
which is in Ubuntu-3.19.0-17.17~19.

Can you test with this version:
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+sourcepub/5038795/+listing-archive-extra

To see if it resolves this issue?

Thanks,
--chris

[Thiago Martins (martinx)]

Hello Chris,

 After a complete power cycle, Juno with Trusty + Linux 3.19 is working. Tenants have connectivity again.

 So far, the only error that I'm still seeing, is the following:

---
==> /var/log/neutron/ovs-cleanup.log <==
2015-05-07 13:30:57.384 881 TRACE neutron Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'delete', 'tap3da8f4fe-55']
2015-05-07 13:30:57.384 881 TRACE neutron Exit code: 2
2015-05-07 13:30:57.384 881 TRACE neutron Stdout: ''
2015-05-07 13:30:57.384 881 TRACE neutron Stderr: 'RTNETLINK answers: Operation not supported\n'
---

 I'll clean up everything, bridges and Namespaces, to see if the problem persists. I saw this problem in a lab as well, I'll reset the databases and tenants to start over again, with Linux 3.19 from the beginning.

 If I do not manage to reproduce it until next Monday, I'll close this BUG as "not reproducible".

 Sorry about the buzz!

Best,
Thiago

[Thiago Martins (martinx)]

I am unable to replicate this problem. Closing it.