Kernel panic on 3.8.0-29 when using ipvs

Bug #1238494 reported by Luc van Donkersgoed
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
High
Luis Henriques
Raring
Fix Released
Undecided
Luis Henriques
Saucy
Fix Released
High
Luis Henriques

Bug Description

SRU Justification:

Impact:

A NULL pointer dereferrence will occur when a user adds an IPVS service. This occurs since kernel 3.8.0-28.41 (Raring), after commit:

  dc7b3eb ipvs: Fix reuse connection if real server is dead

The NULL pointer occurs when accessing the ipvs variable in line 1658:

   1658 if (unlikely(sysctl_expire_nodest_conn(ipvs)) && cp && cp->dest &&
   1659 unlikely(!atomic_read(&cp->dest->weight)) && !iph.fragoffs &&
   1660 is_new_conn(skb, &iph)) {
   1661 ip_vs_conn_expire_now(cp);
   1662 __ip_vs_conn_put(cp);
   1663 cp = NULL;
   1664 }

Mainline kernel has this variable initialised earlier, with commit:

 0c12582 ipvs: add backup_only flag to avoid loops

Fix:

Apply commit 0c12582 "ipvs: add backup_only flag to avoid loops" fix the problem. Bug reporter has claimed success with a test kernel that contains this commit.

Testcase:

Simply running the command:

 sudo ipvsadm -A -u 10.0.50.4:53

Will trigger the bug.

---

In kernel 3.8.0-29 and higher (I've tested 3.8.0-30 and 3.8.0-31), the kernel panics when adding IPVS service. Specifically, when I execute the following command:

 sudo ipvsadm -A -u 10.0.50.4:53

The kernel immediately panics. I've reverted the kernel to 3.8.0-27, and IPVS executes without a problem.

The panic is completely reproducable, using a clean install, no extra packages installed, all packages upgraded.

I've attached the apport report of the system running 3.8.0-29.

Best,
Luc van Donkersgoed
---
AlsaDevices:
 total 0
 crw-rw---T 1 root audio 116, 1 Oct 11 10:04 seq
 crw-rw---T 1 root audio 116, 33 Oct 11 10:04 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.9.2-0ubuntu8.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 13.04
HibernationDevice: RESUME=UUID=566497ef-0abf-42f0-85ee-988bf9ba2034
InstallationDate: Installed on 2012-12-03 (311 days ago)
InstallationMedia: Ubuntu-Server 12.04 LTS "Precise Pangolin" - Release amd64 (20120424.1)
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MachineType: VMware, Inc. VMware Virtual Platform
MarkForUpload: True
Package: linux (not installed)
PciMultimedia:

ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.8.0-31-generic root=/dev/mapper/dnslb01-root ro
ProcVersionSignature: Ubuntu 3.8.0-31.46-generic 3.8.13.8
RelatedPackageVersions:
 linux-restricted-modules-3.8.0-31-generic N/A
 linux-backports-modules-3.8.0-31-generic N/A
 linux-firmware 1.106
RfKill: Error: [Errno 2] No such file or directory
Tags: raring
Uname: Linux 3.8.0-31-generic x86_64
UpgradeStatus: Upgraded to raring on 2013-10-10 (0 days ago)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
dmi.bios.date: 06/22/2012
dmi.bios.vendor: Phoenix Technologies LTD
dmi.bios.version: 6.00
dmi.board.name: 440BX Desktop Reference Platform
dmi.board.vendor: Intel Corporation
dmi.board.version: None
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 1
dmi.chassis.vendor: No Enclosure
dmi.chassis.version: N/A
dmi.modalias: dmi:bvnPhoenixTechnologiesLTD:bvr6.00:bd06/22/2012:svnVMware,Inc.:pnVMwareVirtualPlatform:pvrNone:rvnIntelCorporation:rn440BXDesktopReferencePlatform:rvrNone:cvnNoEnclosure:ct1:cvrN/A:
dmi.product.name: VMware Virtual Platform
dmi.product.version: None
dmi.sys.vendor: VMware, Inc.

Revision history for this message
Luc van Donkersgoed (lucvandonkersgoed) wrote :
Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1238494

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Luc van Donkersgoed (lucvandonkersgoed) wrote : BootDmesg.txt

apport information

tags: added: apport-collected raring
description: updated
Revision history for this message
Luc van Donkersgoed (lucvandonkersgoed) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Luc van Donkersgoed (lucvandonkersgoed) wrote : HookError_cloud_archive.txt

apport information

Revision history for this message
Luc van Donkersgoed (lucvandonkersgoed) wrote : Lspci.txt

apport information

Revision history for this message
Luc van Donkersgoed (lucvandonkersgoed) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Luc van Donkersgoed (lucvandonkersgoed) wrote : ProcEnviron.txt

apport information

Revision history for this message
Luc van Donkersgoed (lucvandonkersgoed) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Luc van Donkersgoed (lucvandonkersgoed) wrote : ProcModules.txt

apport information

Revision history for this message
Luc van Donkersgoed (lucvandonkersgoed) wrote : UdevDb.txt

apport information

Revision history for this message
Luc van Donkersgoed (lucvandonkersgoed) wrote : UdevLog.txt

apport information

Revision history for this message
Luc van Donkersgoed (lucvandonkersgoed) wrote : WifiSyslog.txt

apport information

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Luis Henriques (henrix) wrote :

I confirm I'm able to reproduce this bug in Raring using the instructions in the bug description. The problem is upstream commit:

dc7b3eb ipvs: Fix reuse connection if real server is dead

It causes a NULL pointer here:
   1658 if (unlikely(sysctl_expire_nodest_conn(ipvs)) && cp && cp->dest &&
   1659 unlikely(!atomic_read(&cp->dest->weight)) && !iph.fragoffs &&
   1660 is_new_conn(skb, &iph)) {
   1661 ip_vs_conn_expire_now(cp);
   1662 __ip_vs_conn_put(cp);
   1663 cp = NULL;
   1664 }
when accessing the ipvs variable. In mainline kernel, this variable is initialised; in the 3.8 kernel it is not. I'll post a test kernel with a fix for testing. Another option is to revert the above commit.

Changed in linux (Ubuntu):
importance: Undecided → High
tags: added: regression-update
Revision history for this message
Luis Henriques (henrix) wrote :

I've uploaded a test kernel here:

http://people.canonical.com/~henrix/lp1238494/

Basically, this is a 3.8.0-32.47 (the Raring kernel currently in the -proposed pocket) plus an additional commit:

0c12582 ipvs: add backup_only flag to avoid loops

This commit adds the initialisation of the ipvs variable to function ip_vs_in(). I can't reproduce the issue anymore using this kernel. Could you please give it a try and report back? Thanks.

Luis Henriques (henrix)
Changed in linux (Ubuntu):
assignee: nobody → Luis Henriques (henrix)
Revision history for this message
Luc van Donkersgoed (lucvandonkersgoed) wrote :

Hi, I'd like to test the latest kernel with the additional commit, but I don't know how..

I've installed the 3.8.0-32.47 kernel from the raring-proposed source. This kernel still panics (but you already know that), so I need to install the addition commit. Can you tell me how to do this?

Revision history for this message
Luis Henriques (henrix) wrote :

Hi Luc, in my previous comment (comment #15) I posted a link where I uploaded the test kernel. Please download the 4 packages into your disk. After that, you can install these packages from the command line by running the command:

 sudo dpkg -i *.deb

Thanks!

Revision history for this message
Luc van Donkersgoed (lucvandonkersgoed) wrote :

Hi Luis,

I can confirm that your kernel has fixed my problem. Adding a new service to ipvs does no longer generate a kernel panic.

I had to install the following extra packages to compile your kernel:

 crda
 iw
 wireless-regdb

Thanks for your fast response and solution.
Best,
Luc van Donkersgoed

Revision history for this message
Evan Callicoat (diopter) wrote :

I can confirm that this kernel fixes the problem for me as well, including with the new keepalived build from precise-proposed which fixes this bug https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1211876

Luis Henriques (henrix)
description: updated
Tim Gardner (timg-tpi)
Changed in linux (Ubuntu Raring):
status: New → Fix Committed
Changed in linux (Ubuntu Saucy):
status: Confirmed → Fix Released
Changed in linux (Ubuntu Raring):
assignee: nobody → Luis Henriques (henrix)
Revision history for this message
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-raring' to 'verification-done-raring'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-raring
tags: added: verification-done-raring
removed: verification-needed-raring
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (8.8 KiB)

This bug was fixed in the package linux - 3.8.0-33.48

---------------
linux (3.8.0-33.48) raring; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1242849

  [ Maximiliano Curia ]

  * SAUCE: (no-up) Only let characters through when there are active
    readers.
    - LP: #1208740

  [ Upstream Kernel Changes ]

  * cciss: fix info leak in cciss_ioctl32_passthru()
    - LP: #1188355
    - CVE-2013-2147
  * cpqarray: fix info leak in ida_locked_ioctl()
    - LP: #1188355
    - CVE-2013-2147
  * mount: consolidate permission checks
    - LP: #1226726
  * get rid of full-hash scan on detaching vfsmounts
    - LP: #1226726
  * Smack: Fix the bug smackcipso can't set CIPSO correctly
    - LP: #1236743
  * ipvs: add backup_only flag to avoid loops
    - LP: #1238494
  * tuntap: correctly handle error in tun_set_iff()
    - LP: #1229975
    - CVE-2013-4343
  * htb: fix sign extension bug
    - LP: #1240580
  * net: avoid to hang up on sending due to sysctl configuration overflow.
    - LP: #1240580
  * net: check net.core.somaxconn sysctl values
    - LP: #1240580
  * macvlan: validate flags
    - LP: #1240580
  * neighbour: populate neigh_parms on alloc before calling ndo_neigh_setup
    - LP: #1240580
  * bonding: modify only neigh_parms owned by us
    - LP: #1240580
  * fib_trie: remove potential out of bound access
    - LP: #1240580
  * bridge: don't try to update timers in case of broken MLD queries
    - LP: #1240580
  * tcp: cubic: fix overflow error in bictcp_update()
    - LP: #1240580
  * tcp: cubic: fix bug in bictcp_acked()
    - LP: #1240580
  * ipv6: don't stop backtracking in fib6_lookup_1 if subtree does not
    match
    - LP: #1240580
  * 8139cp: Fix skb leak in rx_status_loop failure path.
    - LP: #1240580
  * tun: signedness bug in tun_get_user()
    - LP: #1240580
  * ipv6: remove max_addresses check from ipv6_create_tempaddr
    - LP: #1240580
  * ipv6: Store Router Alert option in IP6CB directly.
    - LP: #1240580
  * ipv6: drop packets with multiple fragmentation headers
    - LP: #1240580
  * tcp: set timestamps for restored skb-s
    - LP: #1240580
  * net: usb: Add HP hs2434 device to ZLP exception table
    - LP: #1240580
  * tcp: initialize rcv_tstamp for restored sockets
    - LP: #1240580
  * ipv4: sendto/hdrincl: don't use destination address found in header
    - LP: #1240580
  * tcp: tcp_make_synack() should use sock_wmalloc
    - LP: #1240580
  * tipc: set sk_err correctly when connection fails
    - LP: #1240580
  * net: bridge: convert MLDv2 Query MRC into msecs_to_jiffies for
    max_delay
    - LP: #1240580
  * ICMPv6: treat dest unreachable codes 5 and 6 as EACCES, not EPROTO
    - LP: #1240580
  * tg3: Don't turn off led on 5719 serdes port 0
    - LP: #1240580
  * vhost_net: poll vhost queue after marking DMA is done
    - LP: #1240580
  * net: ipv6: tcp: fix potential use after free in tcp_v6_do_rcv
    - LP: #1240580
  * drm/radeon/si: Add support for CP DMA to CS checker for compute v2
    - LP: #1240580
  * sfc: Fix efx_rx_buf_offset() for recycled pages
    - LP: #1240580
  * cfq: explicitly use 64bit divide operation for 64bit arguments
    - LP: #1240580
  * drm/radeon/atom: wor...

Read more...

Changed in linux (Ubuntu Raring):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.