iproute2 crashes being reported since kernel version 3.13-0-59-generic

Bug #1481038 reported by Brian Murray
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
iproute2 (Ubuntu)
Invalid
High
Unassigned
Trusty
Invalid
High
Unassigned
linux (Ubuntu)
Fix Released
High
Unassigned
Trusty
Fix Released
High
Unassigned

Bug Description

There has recently been a tremendous uptick in the number of crashes being reported about iproute2 since 2015-07-27. This was initially noticed as the phasing of the iproute2 package in -updates for 14.04 (SRU bug 1470091) was stopped. Its worth noting that the same increase in crashes was not detected with the Vivid SRU. Additionally, it appears that the increase in crash rate is unrelated to the iproute2 SRU as it occurs with the version of the package in the release pocket or the updates pocket for 14.04.

The following is a database query showing the quantity of crashes on a daily basis about the iproute2 package for the version of the package in the release pocket and then the updates pocket.

ipdb> old_version
'3.12.0-2'
ipdb> counters_cf.get(old_vers_column, column_start='20150803', column_reversed=True)
OrderedDict([(u'20150803', 15), (u'20150802', 4), (u'20150801', 2), (u'20150731', 13), (u'20150730', 39), (u'20150729', 50), (u'20150728', 51), (u'20150727', 1), (u'20150623', 1), (u'20150523', 1), (u'20150519', 1), (u'20150511', 1), (u'20150509', 1), (u'20150419', 1), (u'20150413', 1), (u'20150329', 1), (u'20150302', 1), (u'20150220', 1), (u'20150130', 1), (u'20150129', 1), (u'20150128', 1), (u'20150103', 1), (u'20141021', 1), (u'20140930', 1), (u'20140917', 1), (u'20140904', 1), (u'20140826', 1), (u'20140818', 1), (u'20140808', 1), (u'20140801', 1), (u'20140723', 1)])
ipdb> new_version
'3.12.0-2ubuntu1'
ipdb> counters_cf.get(new_vers_column, column_start='20150803', column_reversed=True)
OrderedDict([(u'20150803', 7), (u'20150802', 1), (u'20150731', 8), (u'20150730', 24), (u'20150729', 25), (u'20150728', 2), (u'20150727', 1)])

Notice the increase starting on 20150728 for the old version of the package. I've reviewed the individual crashes from 20150729 and all of them (regardless of the iproute2 package version) were using the following kernel version. I'd attribute the drop in crashes on 20150801 and 20150802 to it being the weekend.

3.13.0-59-generic

Unfortunately, all the crashes from the iproute2 package are incomplete / corrupt so there isn't much to go on. Here is one example crash:

https://errors.ubuntu.com/oops/331373ac-3596-11e5-9ed9-fa163e22e467

I'll add all the OOPSes from 20150729 as an attachment if somebody wants to have a look at them. I'm also happy to run more queries of the OOPSes to see if we can determine a pattern.

Brad Figg (brad-figg)
Changed in linux (Ubuntu):
status: New → Triaged
Revision history for this message
Brian Murray (brian-murray) wrote :

This is the information from ProcCmdline for the crashes on 20150729:

 $ grep "^ " 20150729-iproute2-crashes.txt| awk -F, {'print $2'} | sort | uniq -c
      1
      3 /bin/ip link
     59 /sbin/ip route
      1 /sbin/ip route show to 0/0

I've tried to manually create a crash using "ip monitor", since it doesn't exit quickly, and the -59- kernel and was able to create a good / complete apport crash.

Changed in linux (Ubuntu):
importance: Undecided → High
tags: added: kernel-da-key trusty
Changed in linux (Ubuntu Trusty):
status: New → Triaged
importance: Undecided → High
Revision history for this message
Brian Murray (brian-murray) wrote :

This is a list of crashes in the error tracker about iproute2 from 20150729.

tags: removed: kernel-da-key
Revision history for this message
Brian Murray (brian-murray) wrote :

Its also worth noting all the ones I have found are on amd64 systems.

Changed in iproute2 (Ubuntu):
status: New → Triaged
Changed in iproute2 (Ubuntu Trusty):
status: New → Triaged
Changed in iproute2 (Ubuntu):
importance: Undecided → High
Changed in iproute2 (Ubuntu Trusty):
importance: Undecided → High
Revision history for this message
Stefan Bader (smb) wrote :

Maybe we should make sure that this remains a problem with the latest kernel *-61.100. The -59 kernel caused a lot of other problems. Usually those were related to executing a 64bit binary from a 32bit binary. But I would not completely rule out other odd failures. So if all the reports are with -59 and the numbers again go down it is probably related to that.

Revision history for this message
Brian Murray (brian-murray) wrote :

Stefan - that sounds like a good plan to me. Looking at the crashes from 2015-08-03 we can see the following:

 $ grep "^ " 20150803-iproute2-crashes.txt | awk -F, {'print $3'} | sort | uniq -c
     22 Linux 3.13.0-59-generic x86_64
      1 Linux 3.13.0-61-generic x86_64
      1 Linux 3.19.0-25-generic x86_64
      1 Linux 3.19.0-26-generic x86_64

Looking at the -61- crash it is actually a package installation failure so not a concern. The two 3.19 crashes are a package installation failure and a good, complete, iproute2 crash. So I think things are looking better.

I've attached the crashes I reviewed from 2015-08-03.

Revision history for this message
Brian Murray (brian-murray) wrote :

Here is the same database query from the description with information including today.

ipdb> old_vers_column
'Ubuntu 14.04:iproute2:3.12.0-2'
ipdb> counters_cf.get(old_vers_column, column_start='20150804', column_reversed=True, column_count=15)
OrderedDict([(u'20150804', 4), (u'20150803', 16), (u'20150802', 4), (u'20150801', 2), (u'20150731', 13), (u'20150730', 39), (u'20150729', 50), (u'20150728', 51), (u'20150727', 1), (u'20150623', 1), (u'20150523', 1), (u'20150519', 1), (u'20150511', 1), (u'20150509', 1), (u'20150419', 1)])
ipdb> new_vers_column
'Ubuntu 14.04:iproute2:3.12.0-2ubuntu1'
ipdb> counters_cf.get(new_vers_column, column_start='20150804', column_reversed=True, column_count=15)
OrderedDict([(u'20150804', 5), (u'20150803', 7), (u'20150802', 1), (u'20150731', 8), (u'20150730', 24), (u'20150729', 25), (u'20150728', 2), (u'20150727', 1)])

I'll run this query again later in the week to make sure we are all good.

Revision history for this message
Brian Murray (brian-murray) wrote :

Things look good to me now.

ipdb> counters_cf.get(old_vers_column, column_start='20150806', column_reversed=True)
OrderedDict([(u'20150806', 2), (u'20150805', 2), (u'20150804', 4), (u'20150803', 16), (u'20150802', 4), (u'20150801', 2), (u'20150731', 13), (u'20150730', 39), (u'20150729', 50), (u'20150728', 51), (u'20150727', 1), (u'20150623', 1), (u'20150523', 1), (u'20150519', 1), (u'20150511', 1), (u'20150509', 1), (u'20150419', 1), (u'20150413', 1), (u'20150329', 1), (u'20150302', 1), (u'20150220', 1), (u'20150130', 1), (u'20150129', 1), (u'20150128', 1), (u'20150103', 1), (u'20141021', 1), (u'20140930', 1), (u'20140917', 1), (u'20140904', 1), (u'20140826', 1), (u'20140818', 1), (u'20140808', 1), (u'20140801', 1), (u'20140723', 1)])

ipdb> counters_cf.get(new_vers_column, column_start='20150806', column_reversed=True)
OrderedDict([(u'20150804', 6), (u'20150803', 7), (u'20150802', 1), (u'20150731', 8), (u'20150730', 24), (u'20150729', 25), (u'20150728', 2), (u'20150727', 1)])

The remaining crashes with the old version are probably slow updaters.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Per former comments and no new related issues closing the bug to clean up.

Changed in iproute2 (Ubuntu):
status: Triaged → Invalid
Changed in iproute2 (Ubuntu Trusty):
status: Triaged → Invalid
Changed in linux (Ubuntu):
status: Triaged → Invalid
Changed in linux (Ubuntu Trusty):
status: Triaged → Invalid
Changed in linux (Ubuntu):
status: Invalid → Fix Released
Changed in linux (Ubuntu Trusty):
status: Invalid → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.