Activity log for bug #1349768

Date Who What changed Old value New value Message
2014-07-29 10:06:41 Tero Marttila bug added bug
2014-07-29 10:08:07 Tero Marttila bug task added linux (Ubuntu)
2014-07-29 10:08:17 Tero Marttila bug task deleted biosdevname (Ubuntu)
2014-07-29 10:09:42 Tero Marttila description I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs loadbalancer and dnsmasq server for pxebooting servers. After updating linux-image 3.13.0-30.55 -> 3.13.0-32.57 I noticed that dnsmasq-tftp stopped working. pxeboot clients would hang on the "Loading ..../linux" TFTP transfer, with the transfer stalling roughly ~1000 blocks into the transfer: 10:30:51.011728 IP 10.1.1.2.43540 > 10.1.12.1.49165: UDP, length 1412 10:30:51.011924 IP 10.1.12.1.49165 > 10.1.1.2.43540: UDP, length 4 10:30:51.012012 IP 10.1.1.2.43540 > 10.1.12.1.49165: UDP, length 1412 10:30:51.012183 IP 10.1.12.1.49165 > 10.1.1.2.43540: UDP, length 4 stracing dnsmasq I noticed something very odd: sendto() on the socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start persistently returning EPERM in mid-transfer, even when dnsmasq continued to periodically retry: select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 1 (in [17], left {0, 249834}) recvfrom(17, "\0\4\3\352", 4096, 0, NULL, NULL) = 4 lseek(16, 1410816, SEEK_SET) = 1410816 read(16, "\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v"..., 1408) = 1408 sendto(17, "\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = 1412 select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 1 (in [17], left {0, 249839}) recvfrom(17, "\0\4\3\353", 4096, 0, NULL, NULL) = 4 lseek(16, 1412224, SEEK_SET) = 1412224 read(16, "*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277\221\\\307\372"..., 1408) = 1408 sendto(17, "\0\3\3\354*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = -1 EPERM (Operation not permitted) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) lseek(16, 1412224, SEEK_SET) = 1412224 read(16, "*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277\221\\\307\372"..., 1408) = 1408 sendto(17, "\0\3\3\354*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = -1 EPERM (Operation not permitted) This was with all iptables rules unloaded (so no OUTPUT -j DENY) and apparmor profiles torn down. I also noticed the following dmesgs appearing at roughly similar times to the tftp transfers getting stuck (although not coinciding exactly with the stall): [70325.516724] IPv6 header not found The error pointed to ipvs (which I am using on the same host as an IPv4 NAT loadbalancer): http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614 I then tore down the ipvs rules (service keepalived stop) and unloaded the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself - the stalled dnsmasq-tftp transfer resumed! This seems to be reproducible, i.e. modprobing ip_vs and starting keepalived will cause dnsmasq-tftp to stall again, and stopping/unloading will resume. This seems to happen reproducibly on boot with -32 and -30. This does NOT seem to happen with 3.13.0-29 which I was using up until now. I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs loadbalancer and dnsmasq server for pxebooting servers. After updating linux-image 3.13.0-29.53 -> 3.13.0-32.57 I noticed that dnsmasq-tftp stopped working. pxeboot clients would hang on the "Loading ..../linux" TFTP transfer, with the transfer stalling roughly ~1000 blocks into the transfer: 10:30:51.011728 IP 10.1.1.2.43540 > 10.1.12.1.49165: UDP, length 1412 10:30:51.011924 IP 10.1.12.1.49165 > 10.1.1.2.43540: UDP, length 4 10:30:51.012012 IP 10.1.1.2.43540 > 10.1.12.1.49165: UDP, length 1412 10:30:51.012183 IP 10.1.12.1.49165 > 10.1.1.2.43540: UDP, length 4 stracing dnsmasq I noticed something very odd: sendto() on the socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start persistently returning EPERM in mid-transfer, even when dnsmasq continued to periodically retry: select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 1 (in [17], left {0, 249834}) recvfrom(17, "\0\4\3\352", 4096, 0, NULL, NULL) = 4 lseek(16, 1410816, SEEK_SET) = 1410816 read(16, "\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v"..., 1408) = 1408 sendto(17, "\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = 1412 select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 1 (in [17], left {0, 249839}) recvfrom(17, "\0\4\3\353", 4096, 0, NULL, NULL) = 4 lseek(16, 1412224, SEEK_SET) = 1412224 read(16, "*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277\221\\\307\372"..., 1408) = 1408 sendto(17, "\0\3\3\354*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = -1 EPERM (Operation not permitted) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) lseek(16, 1412224, SEEK_SET) = 1412224 read(16, "*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277\221\\\307\372"..., 1408) = 1408 sendto(17, "\0\3\3\354*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = -1 EPERM (Operation not permitted) This was with all iptables rules unloaded (so no OUTPUT -j DENY) and apparmor profiles torn down. I also noticed the following dmesgs appearing at roughly similar times to the tftp transfers getting stuck (although not coinciding exactly with the stall): [70325.516724] IPv6 header not found The error pointed to ipvs (which I am using on the same host as an IPv4 NAT loadbalancer): http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614 I then tore down the ipvs rules (service keepalived stop) and unloaded the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself - the stalled dnsmasq-tftp transfer resumed! This seems to be reproducible, i.e. modprobing ip_vs and starting keepalived will cause dnsmasq-tftp to stall again, and stopping/unloading will resume. This seems to happen reproducibly on boot with -32 and -30. This does NOT seem to happen with 3.13.0-29 which I was using up until now.
2014-07-29 10:30:11 Brad Figg linux (Ubuntu): status New Incomplete
2014-07-29 11:04:53 Tero Marttila tags apport-collected trusty
2014-07-29 11:04:54 Tero Marttila description I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs loadbalancer and dnsmasq server for pxebooting servers. After updating linux-image 3.13.0-29.53 -> 3.13.0-32.57 I noticed that dnsmasq-tftp stopped working. pxeboot clients would hang on the "Loading ..../linux" TFTP transfer, with the transfer stalling roughly ~1000 blocks into the transfer: 10:30:51.011728 IP 10.1.1.2.43540 > 10.1.12.1.49165: UDP, length 1412 10:30:51.011924 IP 10.1.12.1.49165 > 10.1.1.2.43540: UDP, length 4 10:30:51.012012 IP 10.1.1.2.43540 > 10.1.12.1.49165: UDP, length 1412 10:30:51.012183 IP 10.1.12.1.49165 > 10.1.1.2.43540: UDP, length 4 stracing dnsmasq I noticed something very odd: sendto() on the socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start persistently returning EPERM in mid-transfer, even when dnsmasq continued to periodically retry: select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 1 (in [17], left {0, 249834}) recvfrom(17, "\0\4\3\352", 4096, 0, NULL, NULL) = 4 lseek(16, 1410816, SEEK_SET) = 1410816 read(16, "\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v"..., 1408) = 1408 sendto(17, "\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = 1412 select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 1 (in [17], left {0, 249839}) recvfrom(17, "\0\4\3\353", 4096, 0, NULL, NULL) = 4 lseek(16, 1412224, SEEK_SET) = 1412224 read(16, "*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277\221\\\307\372"..., 1408) = 1408 sendto(17, "\0\3\3\354*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = -1 EPERM (Operation not permitted) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) lseek(16, 1412224, SEEK_SET) = 1412224 read(16, "*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277\221\\\307\372"..., 1408) = 1408 sendto(17, "\0\3\3\354*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = -1 EPERM (Operation not permitted) This was with all iptables rules unloaded (so no OUTPUT -j DENY) and apparmor profiles torn down. I also noticed the following dmesgs appearing at roughly similar times to the tftp transfers getting stuck (although not coinciding exactly with the stall): [70325.516724] IPv6 header not found The error pointed to ipvs (which I am using on the same host as an IPv4 NAT loadbalancer): http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614 I then tore down the ipvs rules (service keepalived stop) and unloaded the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself - the stalled dnsmasq-tftp transfer resumed! This seems to be reproducible, i.e. modprobing ip_vs and starting keepalived will cause dnsmasq-tftp to stall again, and stopping/unloading will resume. This seems to happen reproducibly on boot with -32 and -30. This does NOT seem to happen with 3.13.0-29 which I was using up until now. I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs loadbalancer and dnsmasq server for pxebooting servers. After updating linux-image 3.13.0-29.53 -> 3.13.0-32.57 I noticed that dnsmasq-tftp stopped working. pxeboot clients would hang on the "Loading ..../linux" TFTP transfer, with the transfer stalling roughly ~1000 blocks into the transfer: 10:30:51.011728 IP 10.1.1.2.43540 > 10.1.12.1.49165: UDP, length 1412 10:30:51.011924 IP 10.1.12.1.49165 > 10.1.1.2.43540: UDP, length 4 10:30:51.012012 IP 10.1.1.2.43540 > 10.1.12.1.49165: UDP, length 1412 10:30:51.012183 IP 10.1.12.1.49165 > 10.1.1.2.43540: UDP, length 4 stracing dnsmasq I noticed something very odd: sendto() on the socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start persistently returning EPERM in mid-transfer, even when dnsmasq continued to periodically retry: select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 1 (in [17], left {0, 249834}) recvfrom(17, "\0\4\3\352", 4096, 0, NULL, NULL) = 4 lseek(16, 1410816, SEEK_SET) = 1410816 read(16, "\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v"..., 1408) = 1408 sendto(17, "\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = 1412 select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 1 (in [17], left {0, 249839}) recvfrom(17, "\0\4\3\353", 4096, 0, NULL, NULL) = 4 lseek(16, 1412224, SEEK_SET) = 1412224 read(16, "*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277\221\\\307\372"..., 1408) = 1408 sendto(17, "\0\3\3\354*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = -1 EPERM (Operation not permitted) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) lseek(16, 1412224, SEEK_SET) = 1412224 read(16, "*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277\221\\\307\372"..., 1408) = 1408 sendto(17, "\0\3\3\354*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = -1 EPERM (Operation not permitted) This was with all iptables rules unloaded (so no OUTPUT -j DENY) and apparmor profiles torn down. I also noticed the following dmesgs appearing at roughly similar times to the tftp transfers getting stuck (although not coinciding exactly with the stall): [70325.516724] IPv6 header not found The error pointed to ipvs (which I am using on the same host as an IPv4 NAT loadbalancer): http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614 I then tore down the ipvs rules (service keepalived stop) and unloaded the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself - the stalled dnsmasq-tftp transfer resumed! This seems to be reproducible, i.e. modprobing ip_vs and starting keepalived will cause dnsmasq-tftp to stall again, and stopping/unloading will resume. This seems to happen reproducibly on boot with -32 and -30. This does NOT seem to happen with 3.13.0-29 which I was using up until now. --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Jul 29 13:43 seq crw-rw---- 1 root audio 116, 33 Jul 29 13:43 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.2 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/catcp2-swap InstallationDate: Installed on 2014-06-03 (56 days ago) InstallationMedia: Ubuntu-Server 14.04 LTS "Trusty Tahr" - Release amd64 (20140416.2) MachineType: Dell Inc. PowerEdge R410 Package: linux-image-3.13.0-32-generic 3.13.0-32.57 PackageArchitecture: amd64 PciMultimedia: ProcEnviron: TERM=xterm PATH=(custom, no user) LANG=C.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-32-generic root=/dev/mapper/hostname-root ro console=ttyS1,115200n8 console=tty0 nomdmonddf nomdmonisw ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4 RelatedPackageVersions: linux-restricted-modules-3.13.0-32-generic N/A linux-backports-modules-3.13.0-32-generic N/A linux-firmware 1.127.5 RfKill: Error: [Errno 2] No such file or directory Tags: trusty Uname: Linux 3.13.0-32-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 07/30/2013 dmi.bios.vendor: Dell Inc. dmi.bios.version: 1.12.0 dmi.board.name: 01V648 dmi.board.vendor: Dell Inc. dmi.board.version: A03 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr1.12.0:bd07/30/2013:svnDellInc.:pnPowerEdgeR410:pvr:rvnDellInc.:rn01V648:rvrA03:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R410 dmi.sys.vendor: Dell Inc.
2014-07-29 11:04:55 Tero Marttila attachment added BootDmesg.txt https://bugs.launchpad.net/bugs/1349768/+attachment/4165024/+files/BootDmesg.txt
2014-07-29 11:04:57 Tero Marttila attachment added CurrentDmesg.txt https://bugs.launchpad.net/bugs/1349768/+attachment/4165025/+files/CurrentDmesg.txt
2014-07-29 11:04:59 Tero Marttila attachment added Dependencies.txt https://bugs.launchpad.net/bugs/1349768/+attachment/4165026/+files/Dependencies.txt
2014-07-29 11:05:01 Tero Marttila attachment added IwConfig.txt https://bugs.launchpad.net/bugs/1349768/+attachment/4165027/+files/IwConfig.txt
2014-07-29 11:05:04 Tero Marttila attachment added Lspci.txt https://bugs.launchpad.net/bugs/1349768/+attachment/4165028/+files/Lspci.txt
2014-07-29 11:05:05 Tero Marttila attachment added Lsusb.txt https://bugs.launchpad.net/bugs/1349768/+attachment/4165029/+files/Lsusb.txt
2014-07-29 11:05:07 Tero Marttila attachment added ProcCpuinfo.txt https://bugs.launchpad.net/bugs/1349768/+attachment/4165030/+files/ProcCpuinfo.txt
2014-07-29 11:05:10 Tero Marttila attachment added ProcInterrupts.txt https://bugs.launchpad.net/bugs/1349768/+attachment/4165031/+files/ProcInterrupts.txt
2014-07-29 11:05:11 Tero Marttila attachment added ProcModules.txt https://bugs.launchpad.net/bugs/1349768/+attachment/4165032/+files/ProcModules.txt
2014-07-29 11:05:13 Tero Marttila attachment added UdevDb.txt https://bugs.launchpad.net/bugs/1349768/+attachment/4165033/+files/UdevDb.txt
2014-07-29 11:05:16 Tero Marttila attachment added UdevLog.txt https://bugs.launchpad.net/bugs/1349768/+attachment/4165034/+files/UdevLog.txt
2014-07-29 11:05:18 Tero Marttila attachment added WifiSyslog.txt https://bugs.launchpad.net/bugs/1349768/+attachment/4165035/+files/WifiSyslog.txt
2014-07-29 11:05:43 Tero Marttila linux (Ubuntu): status Incomplete Confirmed
2014-07-29 14:32:48 Tim Gardner nominated for series Ubuntu Trusty
2014-07-29 14:32:48 Tim Gardner bug task added linux (Ubuntu Trusty)
2014-07-29 14:32:48 Tim Gardner nominated for series Ubuntu Utopic
2014-07-29 14:32:48 Tim Gardner bug task added linux (Ubuntu Utopic)
2014-07-29 14:33:16 Tim Gardner linux (Ubuntu Trusty): status New In Progress
2014-07-29 14:33:16 Tim Gardner linux (Ubuntu Trusty): assignee Chris J Arges (arges)
2014-07-29 14:33:25 Tim Gardner linux (Ubuntu Utopic): status Confirmed Fix Released
2014-07-29 16:49:30 Joseph Salisbury linux (Ubuntu Trusty): importance Undecided Medium
2014-07-29 16:49:32 Joseph Salisbury linux (Ubuntu Utopic): importance Undecided Medium
2014-07-29 17:21:32 Tim Gardner bug added subscriber Tim Gardner
2014-07-30 14:35:26 Chris J Arges linux (Ubuntu Utopic): status Fix Released New
2014-07-30 15:00:18 Brad Figg linux (Ubuntu): status New Confirmed
2014-08-14 01:15:14 Jay Vosburgh bug added subscriber Jay Vosburgh
2014-08-20 11:15:04 Tero Marttila attachment added minimal testcase for LP#1349768 on Ubuntu 14.04 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1349768/+attachment/4182427/+files/lp1349768.sh
2014-08-20 14:47:29 Chris J Arges tags apport-collected trusty apport-collected bug-exists-upstream trusty
2014-08-20 14:47:39 Chris J Arges linux (Ubuntu Utopic): assignee Chris J Arges (arges)
2014-08-20 14:47:42 Chris J Arges linux (Ubuntu Utopic): status Confirmed In Progress
2014-09-24 14:58:21 Chris J Arges description I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs loadbalancer and dnsmasq server for pxebooting servers. After updating linux-image 3.13.0-29.53 -> 3.13.0-32.57 I noticed that dnsmasq-tftp stopped working. pxeboot clients would hang on the "Loading ..../linux" TFTP transfer, with the transfer stalling roughly ~1000 blocks into the transfer: 10:30:51.011728 IP 10.1.1.2.43540 > 10.1.12.1.49165: UDP, length 1412 10:30:51.011924 IP 10.1.12.1.49165 > 10.1.1.2.43540: UDP, length 4 10:30:51.012012 IP 10.1.1.2.43540 > 10.1.12.1.49165: UDP, length 1412 10:30:51.012183 IP 10.1.12.1.49165 > 10.1.1.2.43540: UDP, length 4 stracing dnsmasq I noticed something very odd: sendto() on the socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start persistently returning EPERM in mid-transfer, even when dnsmasq continued to periodically retry: select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 1 (in [17], left {0, 249834}) recvfrom(17, "\0\4\3\352", 4096, 0, NULL, NULL) = 4 lseek(16, 1410816, SEEK_SET) = 1410816 read(16, "\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v"..., 1408) = 1408 sendto(17, "\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = 1412 select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 1 (in [17], left {0, 249839}) recvfrom(17, "\0\4\3\353", 4096, 0, NULL, NULL) = 4 lseek(16, 1412224, SEEK_SET) = 1412224 read(16, "*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277\221\\\307\372"..., 1408) = 1408 sendto(17, "\0\3\3\354*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = -1 EPERM (Operation not permitted) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) lseek(16, 1412224, SEEK_SET) = 1412224 read(16, "*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277\221\\\307\372"..., 1408) = 1408 sendto(17, "\0\3\3\354*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = -1 EPERM (Operation not permitted) This was with all iptables rules unloaded (so no OUTPUT -j DENY) and apparmor profiles torn down. I also noticed the following dmesgs appearing at roughly similar times to the tftp transfers getting stuck (although not coinciding exactly with the stall): [70325.516724] IPv6 header not found The error pointed to ipvs (which I am using on the same host as an IPv4 NAT loadbalancer): http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614 I then tore down the ipvs rules (service keepalived stop) and unloaded the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself - the stalled dnsmasq-tftp transfer resumed! This seems to be reproducible, i.e. modprobing ip_vs and starting keepalived will cause dnsmasq-tftp to stall again, and stopping/unloading will resume. This seems to happen reproducibly on boot with -32 and -30. This does NOT seem to happen with 3.13.0-29 which I was using up until now. --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Jul 29 13:43 seq crw-rw---- 1 root audio 116, 33 Jul 29 13:43 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.2 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/catcp2-swap InstallationDate: Installed on 2014-06-03 (56 days ago) InstallationMedia: Ubuntu-Server 14.04 LTS "Trusty Tahr" - Release amd64 (20140416.2) MachineType: Dell Inc. PowerEdge R410 Package: linux-image-3.13.0-32-generic 3.13.0-32.57 PackageArchitecture: amd64 PciMultimedia: ProcEnviron: TERM=xterm PATH=(custom, no user) LANG=C.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-32-generic root=/dev/mapper/hostname-root ro console=ttyS1,115200n8 console=tty0 nomdmonddf nomdmonisw ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4 RelatedPackageVersions: linux-restricted-modules-3.13.0-32-generic N/A linux-backports-modules-3.13.0-32-generic N/A linux-firmware 1.127.5 RfKill: Error: [Errno 2] No such file or directory Tags: trusty Uname: Linux 3.13.0-32-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 07/30/2013 dmi.bios.vendor: Dell Inc. dmi.bios.version: 1.12.0 dmi.board.name: 01V648 dmi.board.vendor: Dell Inc. dmi.board.version: A03 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr1.12.0:bd07/30/2013:svnDellInc.:pnPowerEdgeR410:pvr:rvnDellInc.:rn01V648:rvrA03:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R410 dmi.sys.vendor: Dell Inc. SRU Justificaton: [Impact] Users of ipvs may encounter dropped packets and message such as: [70325.516724] IPv6 header not found [Fix] Commit eb90b0c734ad793d5f5bf230a9e9a4dcc48df8aa. [Regression Potential] This changes NFPROTO_IPV4 to NFPROTO_IPV6 as we'd expect this nf_hook_ops struct to be in the code. -- I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs loadbalancer and dnsmasq server for pxebooting servers. After updating linux-image 3.13.0-29.53 -> 3.13.0-32.57 I noticed that dnsmasq-tftp stopped working. pxeboot clients would hang on the "Loading ..../linux" TFTP transfer, with the transfer stalling roughly ~1000 blocks into the transfer: 10:30:51.011728 IP 10.1.1.2.43540 > 10.1.12.1.49165: UDP, length 1412 10:30:51.011924 IP 10.1.12.1.49165 > 10.1.1.2.43540: UDP, length 4 10:30:51.012012 IP 10.1.1.2.43540 > 10.1.12.1.49165: UDP, length 1412 10:30:51.012183 IP 10.1.12.1.49165 > 10.1.1.2.43540: UDP, length 4 stracing dnsmasq I noticed something very odd: sendto() on the socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start persistently returning EPERM in mid-transfer, even when dnsmasq continued to periodically retry: select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 1 (in [17], left {0, 249834}) recvfrom(17, "\0\4\3\352", 4096, 0, NULL, NULL) = 4 lseek(16, 1410816, SEEK_SET) = 1410816 read(16, "\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v"..., 1408) = 1408 sendto(17, "\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = 1412 select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 1 (in [17], left {0, 249839}) recvfrom(17, "\0\4\3\353", 4096, 0, NULL, NULL) = 4 lseek(16, 1412224, SEEK_SET) = 1412224 read(16, "*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277\221\\\307\372"..., 1408) = 1408 sendto(17, "\0\3\3\354*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = -1 EPERM (Operation not permitted) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout) lseek(16, 1412224, SEEK_SET) = 1412224 read(16, "*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277\221\\\307\372"..., 1408) = 1408 sendto(17, "\0\3\3\354*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = -1 EPERM (Operation not permitted) This was with all iptables rules unloaded (so no OUTPUT -j DENY) and apparmor profiles torn down. I also noticed the following dmesgs appearing at roughly similar times to the tftp transfers getting stuck (although not coinciding exactly with the stall): [70325.516724] IPv6 header not found The error pointed to ipvs (which I am using on the same host as an IPv4 NAT loadbalancer): http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614 I then tore down the ipvs rules (service keepalived stop) and unloaded the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself - the stalled dnsmasq-tftp transfer resumed! This seems to be reproducible, i.e. modprobing ip_vs and starting keepalived will cause dnsmasq-tftp to stall again, and stopping/unloading will resume. This seems to happen reproducibly on boot with -32 and -30. This does NOT seem to happen with 3.13.0-29 which I was using up until now. --- AlsaDevices:  total 0  crw-rw---- 1 root audio 116, 1 Jul 29 13:43 seq  crw-rw---- 1 root audio 116, 33 Jul 29 13:43 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.2 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/catcp2-swap InstallationDate: Installed on 2014-06-03 (56 days ago) InstallationMedia: Ubuntu-Server 14.04 LTS "Trusty Tahr" - Release amd64 (20140416.2) MachineType: Dell Inc. PowerEdge R410 Package: linux-image-3.13.0-32-generic 3.13.0-32.57 PackageArchitecture: amd64 PciMultimedia: ProcEnviron:  TERM=xterm  PATH=(custom, no user)  LANG=C.UTF-8  SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-32-generic root=/dev/mapper/hostname-root ro console=ttyS1,115200n8 console=tty0 nomdmonddf nomdmonisw ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4 RelatedPackageVersions:  linux-restricted-modules-3.13.0-32-generic N/A  linux-backports-modules-3.13.0-32-generic N/A  linux-firmware 1.127.5 RfKill: Error: [Errno 2] No such file or directory Tags: trusty Uname: Linux 3.13.0-32-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 07/30/2013 dmi.bios.vendor: Dell Inc. dmi.bios.version: 1.12.0 dmi.board.name: 01V648 dmi.board.vendor: Dell Inc. dmi.board.version: A03 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr1.12.0:bd07/30/2013:svnDellInc.:pnPowerEdgeR410:pvr:rvnDellInc.:rn01V648:rvrA03:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R410 dmi.sys.vendor: Dell Inc.
2014-09-24 17:11:50 Tim Gardner linux (Ubuntu Utopic): status In Progress Fix Committed
2014-09-24 17:11:55 Tim Gardner linux (Ubuntu Trusty): status In Progress Fix Committed
2014-09-26 20:40:28 Launchpad Janitor linux (Ubuntu Utopic): status Fix Committed Fix Released
2014-10-10 15:15:55 Launchpad Janitor branch linked lp:ubuntu/precise-proposed/linux-lts-trusty
2014-10-14 15:51:32 Launchpad Janitor branch linked lp:ubuntu/trusty-proposed/linux-keystone
2014-10-16 07:05:54 Brad Figg tags apport-collected bug-exists-upstream trusty apport-collected bug-exists-upstream trusty verification-needed-trusty
2014-10-16 12:52:22 Tero Marttila tags apport-collected bug-exists-upstream trusty verification-needed-trusty apport-collected bug-exists-upstream trusty verification-done-trusty
2014-10-29 02:42:50 Launchpad Janitor linux (Ubuntu Trusty): status Fix Committed Fix Released
2014-10-29 02:42:50 Launchpad Janitor cve linked 2014-3610
2014-10-29 02:42:50 Launchpad Janitor cve linked 2014-3611
2014-10-29 02:42:50 Launchpad Janitor cve linked 2014-3646
2014-10-29 02:42:50 Launchpad Janitor cve linked 2014-3647
2014-10-29 02:42:49 Launchpad Janitor linux (Ubuntu Trusty): status Fix Committed Fix Released