odd behavior with /proc/net/route reading via sh 'read'

[Scott Moser]

I saw some strange behavior when one of my scripts started failing to get the data it was looking for out of /proc/net/route

$ uname -r
$ dpkg -S /boot/vmlinux-$(uname -r)
linux-image-4.1.0-3-generic: /boot/vmlinuz-4.1.0-3-generic

$ id -u
1000
$ cp /proc/net/route /tmp/route
$ diff /proc/net/route /tmp/route
$ md5sum /proc/net/route /tmp/route
4b3663ec7554cf91764106ea9dcc2ada /proc/net/route
4b3663ec7554cf91764106ea9dcc2ada /tmp/route

## just cat the file
$ cat /proc/net/route
Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT
eth0 00000000 01A8F50A 0003 0 0 0 00000000 00 0
eth0 00A8F50A 00000000 0001 0 0 0 00F8FFFF 00 0
virbr0 007AA8C0 00000000 0001 0 0 0 00FFFFFF 00 0

## try to read it with sh while loop
$ sh -c 'while read line; do echo line: $line; done' < /proc/net/route
line: Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT
line: eth0 00A8F50A 00000000 0001 0 0 0 00F8FFFF 0 0 0
line: virbr0 007AA8C0 00000000 0001 0 0 0 00FFFFFF 0 0 0

## odd, where did the first line go?
##
## maybe its still there if we cat
$ cat /proc/net/route | sh -c 'while read line; do echo line: $line; done'
line: Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT
line: eth0 00000000 01A8F50A 0003 0 0 0 00000000 0 0 0
line: eth0 00A8F50A 00000000 0001 0 0 0 00F8FFFF 0 0 0
line: virbr0 007AA8C0 00000000 0001 0 0 0 00FFFFFF 0 0 0

## Yep, still there.
## as expected, the copied file shows sane behavior
$ sh -c 'while read line; do echo line: $line; done' < /tmp/route
line: Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT
line: eth0 00000000 01A8F50A 0003 0 0 0 00000000 0 0 0
line: eth0 00A8F50A 00000000 0001 0 0 0 00F8FFFF 0 0 0
line: virbr0 007AA8C0 00000000 0001 0 0 0 00FFFFFF 0 0 0

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: linux-image-4.1.0-3-generic 4.1.0-3.3
ProcVersionSignature: User Name 4.1.0-3.3-generic 4.1.3
Uname: Linux 4.1.0-3-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Aug 10 23:52 seq
 crw-rw---- 1 root audio 116, 33 Aug 10 23:52 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.18-0ubuntu6
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Mon Aug 10 23:55:09 2015
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: HP ProLiant DL360 Gen9
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.1.0-3-generic root=UUID=ebea97f1-7014-4fac-bc87-c8ff465c566f ro --- console=ttyS1,115200
RelatedPackageVersions:
 linux-restricted-modules-4.1.0-3-generic N/A
 linux-backports-modules-4.1.0-3-generic N/A
 linux-firmware 1.146
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 03/05/2015
dmi.bios.vendor: HP
dmi.bios.version: P89
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrP89:bd03/05/2015:svnHP:pnProLiantDL360Gen9:pvr:cvnHP:ct23:cvr:
dmi.product.name: ProLiant DL360 Gen9
dmi.sys.vendor: HP

[Brad Figg (brad-figg)]

This change was made by a bot.

[Scott Moser (smoser)]

Heres a hint, something goes foul around block size == content size.
$ cp /proc/net/route /tmp/out
$ ls -l /tmp/out
-rw-rw-r-- 1 ubuntu ubuntu 512 Aug 11 01:06 /tmp/out

$ outf="/tmp/out"; for exp in 1 2 3 4 5 6 7 8 9 10; do
   printf "bs=%-8s " "$((2**$exp))";
   out=$(dd if=/proc/net/route of=$outf bs=$((2**${exp})) 2>&1) ||
     { echo $?; break; }
   md5sum $outf | awk '{print $1}';
done
bs=2 dd4fb0207e97b95c910d3116293a5009
bs=4 dd4fb0207e97b95c910d3116293a5009
bs=8 dd4fb0207e97b95c910d3116293a5009
bs=16 dd4fb0207e97b95c910d3116293a5009
bs=32 dd4fb0207e97b95c910d3116293a5009
bs=64 dd4fb0207e97b95c910d3116293a5009
bs=128 dd4fb0207e97b95c910d3116293a5009
bs=256 da8a6ac4fe0557bad7aa9dbc1e45dfd3
bs=512 4b3663ec7554cf91764106ea9dcc2ada
bs=1024 4b3663ec7554cf91764106ea9dcc2ada

[Andy Whitcroft (apw)]

This appears to be an off-by-one issue when handling reads which are smaller than the overall returned size for the table. This is easily spotted in the read line by line mode triggered in the following test script:

  #!/bin/bash
  while read l
  do
          echo "$l"
  done </proc/net/route

Though in my tests it also triggers with cat /proc/net/route when you have more than 8K worth of routes, occuring at the 8K boundary.

The underlying issue is that the code does not correctly take into account the header when starting from anywhere but the beginning. This leads us to drop the first set of entries on restart.

[Andy Whitcroft (apw)]

Patch submited upstream for this issue:

Subject: ipv4: off-by-one in continuation handling in /proc/net/route

[Andy Whitcroft (apw)]

Patch accepted upstream and queued for stable.

[Andy Whitcroft (apw)]

commit 25b97c016b26039982daaa2c11d83979f93b71ab
Author: Andy Whitcroft <email address hidden>
Date: Thu Aug 13 20:49:01 2015 +0100

    ipv4: off-by-one in continuation handling in /proc/net/route

    When generating /proc/net/route we emit a header followed by a line for
    each route. When a short read is performed we will restart this process
    based on the open file descriptor. When calculating the start point we
    fail to take into account that the 0th entry is the header. This leads
    us to skip the first entry when doing a continuation read.

    This can be easily seen with the comparison below:

      while read l; do echo "$l"; done </proc/net/route >A
      cat /proc/net/route >B
      diff -bu A B | grep '^[+-]'

    On my example machine I have approximatly 10KB of route output. There we
    see the very first non-title element is lost in the while read case,
    and an entry around the 8K mark in the cat case:

      +wlan0 00000000 02021EAC 0003 0 0 400 00000000 0 0 0
      -tun1 00C0AC0A 00000000 0001 0 0 950 00C0FFFF 0 0 0

    Fix up the off-by-one when reaquiring position on continuation.

    Fixes: 8be33e955cb9 ("fib_trie: Fib walk rcu should take a tnode and key instead of a trie and a leaf")
    BugLink: http://bugs.launchpad.net/bugs/1483440
    Acked-by: Alexander Duyck <email address hidden>
    Signed-off-by: Andy Whitcroft <email address hidden>
    Signed-off-by: David S. Miller <email address hidden>

[Andy Whitcroft (apw)]

Arrived in the v4.2-rc8 rebase, will be in the 4.2.0-6.6 upload.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.2.0-7.7

---------------
linux (4.2.0-7.7) wily; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1490564
  * rebase to v4.2

  [ Wen Xiong ]

  * SAUCE: ipr: Byte swapping for device_id attribute in sysfs
    - LP: #1453892

  [ Upstream Kernel Changes ]

  * rebase to v4.2
    - LP: #1487345

 -- Tim Gardner <email address hidden> Wed, 26 Aug 2015 07:06:10 -0600