usbnet changes in 3.10 stop use of sierra_net devices

[Kai Kasurinen]

(see upstream bug report)
---
ApportVersion: 2.12.1-0ubuntu3
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: jozo 2529 F.... pulseaudio
 /dev/snd/controlC0: jozo 2529 F.... pulseaudio
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
DistroRelease: Ubuntu 13.10
EcryptfsInUse: Yes
HibernationDevice: RESUME=UUID=33d26b13-5a8c-45e9-b95d-1df45fea62d5
InstallationDate: Installed on 2012-12-22 (252 days ago)
InstallationMedia: Ubuntu 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120823.1)
MachineType: Hewlett-Packard HP Elite 7300 Series MT
MarkForUpload: True
Package: linux (not installed)
ProcFB:
 0 nouveaufb
 1 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.9.0-7-generic root=UUID=8905185c-9d82-498c-970c-6fdb9ee07c45 ro quiet splash vt.handoff=7
ProcVersionSignature: Ubuntu 3.9.0-7.15-generic 3.9.7
RelatedPackageVersions:
 linux-restricted-modules-3.9.0-7-generic N/A
 linux-backports-modules-3.9.0-7-generic N/A
 linux-firmware 1.113
RfKill:

Tags: saucy
Uname: Linux 3.9.0-7-generic x86_64
UpgradeStatus: Upgraded to saucy on 2013-06-19 (73 days ago)
UserGroups: adm cdrom dip libvirtd lpadmin plugdev sambashare sudo
dmi.bios.date: 05/18/2011
dmi.bios.vendor: AMI
dmi.bios.version: 7.05
dmi.board.name: 2AB5
dmi.board.vendor: PEGATRON CORPORATION
dmi.board.version: 1.01
dmi.chassis.asset.tag: CZC126149V
dmi.chassis.type: 3
dmi.chassis.vendor: Hewlett-Packard
dmi.modalias: dmi:bvnAMI:bvr7.05:bd05/18/2011:svnHewlett-Packard:pnHPElite7300SeriesMT:pvr1.01:rvnPEGATRONCORPORATION:rn2AB5:rvr1.01:cvnHewlett-Packard:ct3:cvr:
dmi.product.name: HP Elite 7300 Series MT
dmi.product.version: 1.01
dmi.sys.vendor: Hewlett-Packard

[John (john-redhat-bugs)]

Description of problem:

My Sierra Wireless modems (312U & 320U) not working.

Version-Release number of selected component (if applicable):

v3.10 kernel versions

How reproducible:

Always

Steps to Reproduce:
1.
2.
3.

Actual results:

ifconfig shows wwan0 not running;

"wwan0: flags=4227<UP,BROADCAST,NOARP,MULTICAST> mtu 1500"

Expected results:

"wwan0: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1500"

Additional info:

I've found the fix is to "make" and "make install" the v3.9 kernel driver files "sierra.c" and "sierra_net.c" from http://lxr.free-electrons.com/source/

In fact, it's only "sierra_net.c" that seems to be implicated. For more detailed info, please see: http://forums.whirlpool.net.au/forum-replies.cfm?t=2135188&p=14#r275

[penalvch (penalvch)]

Kai Kasurinen, thank you for taking the time to report this bug and helping to make Ubuntu better. Please execute the following command, as it will automatically gather debugging information, in a terminal:
apport-collect 1219256
When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

[Rob (rob-redhat-bugs)]

Hi,

Just updated this to reflect that F19 is also affected.

Is this an upstream issue? I see there's also a user on Arch with similar difficulties. https://bbs.archlinux.org/viewtopic.php?id=168656

[Bjørn (bjrn-redhat-bugs)]

Yes, this is definitely an upstream issue. It is a regression related to the
commit 7b0c5f21f ("sierra_net: keep status interrupt URB active") from Dan, so I was hoping he would look into it...

John has tested a number of different suggestions in addition to reverting that commit (which will cause another regression for devices needing it), but the exact trigger is still not known. It seems like the firmware of these devices somehow either need the repeated SYNC messages, or need some delayed SYNC message. But I have not been able to find out how to satisfy the firmware without completely disabling the changes from commit 7b0c5f21f.

The only DirectIP device I've got does "unfortunately" work without problems both with and without this fix, so I am unable to test either of the firmware bugs.

The full results of all John's testing is posted in the whirlpool thread. I believe this post in particualar shows where the problem is:
http://forums.whirlpool.net.au/forum-replies.cfm?t=2135188&p=15#r300

But it would be good if Dan or anyone else with a device/firmware needing commit 7b0c5f21f could get involved in the testing. As it is now, it looks like it is best to revert this commit.

[Rob (rob-redhat-bugs)]

Confirming this is still the case in Fedora 20.

[John (john-redhat-bugs)]

I've just received kernel 3.11.4 (3.11.4-101.fc18.i686) as an upgrade, and the problem persists.

I find that the kernel 3.9 drivers I've been using as a work-around will no longer compile with kernel 3.11.

So the problem is now getting serious. I can no longer use my Sierra 312U or 320U modems.

[John (john-redhat-bugs)]

I've discovered the kernel 3.9 drivers will compile and work in kernel 3.11 if the following line in sierra.c

  ifnum = port->number - serial->minor;

is replaced with:

  ifnum = port->port_number;

[Ville Pilvio (vpilvio)]

I confirm this bug. In 13.04 i386 the DNA (ISP) 4g modem Sierra Air:
lsusb:
Bus 006 Device 003: ID 1199:68a3 Sierra Wireless, Inc. MC8700 Modem

did not work out of the box but following these instructions funcioned just fine:
http://forum.ubuntu-fi.org/index.php?topic=42920.0

The forum is in finnish but basically required to create the file /lib/udev/rules.d/81-mm-sierra.rules with the contents:
SUBSYSTEM=="tty", ATTRS{idVendor}=="1199", ATTRS{idProduct}=="68a3", ENV{ID_MM_CANDIDATE}="0"
SUBSYSTEM=="net", ATTRS{idVendor}=="1199", ATTRS{idProduct}=="68a3", ENV{DEVTYPE}="net"

After the std upgrade to 13.10... the Sierra is no longer recognized as wwan0 and I for one have not been able to figure out yet what to do to fix this.

I would rate this severe, a confirmed bug and it affects at least all DNS clients with this modem.

Following these instructions: http://mycusthelp.net/SIERRAWIRELESS/_cs/AnswerDetail.aspx?aid=44 to install sierra's own drivers did not help!

If needed, I am willing n able to supply further info for debugging and fixing this issue.

[Rob (rob-redhat-bugs)]

Thanks John.

That's bought us some time..

[Bjørn (bjrn-redhat-bugs)]

I thought it was concluded that the serial driver had nothing to do with this problem? Quoting from the initial comment above:

   `In fact, it's only "sierra_net.c" that seems to be implicated. For more detailed info, please see: http://forums.whirlpool.net.au/forum-replies.cfm?t=2135188&p=14#r275`

So you shouldn't have to do anything in particular to build on v3.11. Just leave the v3.11 version of sierra.c as it is. It's fine.

As I couldn't come up with any working solution despite all the tests and debugging John did, I am currently considering posting a proposal to revert the v3.9 -> v3.10 patch that broke this. I assume that will cause regressions for other devices, which probably crash less after v3.10, but it still seems like the best alternative out of the limited options we have.

If nothing else, I hope it will wake up Dan :-)

[John (john-redhat-bugs)]

That's correct Bjørn. The combination of v3.11 sierra.c and v3.9 sierra_net.c does compile (and work), of course.

[Dan (dan-redhat-bugs)]

To be clear, the problem currently is that with the 3.10+ driver, the device does not deliver the Link Status Indication messages (0x78) and thus the net interface never gets set LOWER_UP (eg, RUNNING)?

[Bjørn (bjrn-redhat-bugs)]

I believe so, based on Johns testing as reported here: http://forums.whirlpool.net.au/forum-replies.cfm?t=2135188&p=15#r300

Quoting from that post:
---
Just before the switch statement in v3.9 I added this:

netdev_err(dev->net, 'hh.msgid.byte=%x (v3.9)\n", hh.msgid.byte);

clearly stamped with v3.9 for positive ID.

With v3.10, I get a solitary line in /var/log/messages:

Aug 22 00:04:51 localhost kernel: [ 496.732128] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62

That's in stark contrast to the multiple occurrences in v3.9:

Aug 22 06:54:45 localhost kernel: [ 528.845460] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:45 localhost dhclient[2090]: DHCPDISCOVER on wwan0 to 255.255.255.255 port 67 interval 3 (xid=0x327a6bfb)
Aug 22 06:54:45 localhost kernel: [ 528.854737] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:45 localhost kernel: [ 528.861368] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:45 localhost kernel: [ 528.869339] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:45 localhost kernel: [ 528.877367] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.885384] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.893371] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.901382] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.909380] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.917506] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.925260] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.933519] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.941388] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.949386] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.957505] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.965254] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.973508] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.981388] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.989387] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 528.997516] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 529.005270] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 529.013390] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 529.021513] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 529.029387] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 529.037510] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
Aug 22 06:54:46 localhost kernel: [ 529.045391] sierra_net 1-3:1.7 wwan0: hh.msgid.byte=62 (v3.9)
A...

[Dan (dan-redhat-bugs)]

I've been able to reproduce this on a Sierra device, and it's mystifying. Adding some printks shows that reverting my patch, the restart indication *IS* sent, but nothing is listening for it since the interrupt URB isn't submitted. So the RESTART indications pile up, and they all get received after the interrupt URB is submitted at device open() time. At some point, the queued RESTART indications may overflow the device's buffer, which was the cause of the original crash.

In any case, how about the attached patch? It only sends the SYNC when opening the device, after the interrupt URB is submitted. This avoids the issue of sending multiple SYNCs but being deaf to the result, but still works for my devices (USB306, 313U, MC7700):

[ 2103.881954] sierra_net 2-1:1.7 wwan0: register 'sierra_net' at usb-0000:00:1d.7-1, Sierra Wireless USB-to-WWAN Modem, 8a:3a:c0:0b:01:07
[ 2103.950189] systemd-udevd[6414]: renamed network interface wwan0 to wwp0s29f7u1i7
[ 2104.883775] scsi 9:0:0:0: Direct-Access SWI SD Card 2.31 PQ: 0 ANSI: 2
[ 2104.884296] sd 9:0:0:0: Attached scsi generic sg2 type 0
[ 2104.887382] sd 9:0:0:0: [sdb] Attached SCSI removable disk
[ 2148.270909] sierra_net_dosync: sent two SYNC messages
[ 2148.270947] IPv6: ADDRCONF(NETDEV_UP): wwp0s29f7u1i7: link is not ready
[ 2148.275125] sierra_net_status: firmware indicates response available
[ 2148.275931] sierra_net_kevent: received msg 0x62 len 68
[ 2148.275934] sierra_net_kevent: RESTART received code 0x00
[ 2148.283130] sierra_net_status: firmware indicates response available
[ 2148.284508] sierra_net_kevent: received msg 0x78 len 222
[ 2148.284521] IPv6: ADDRCONF(NETDEV_CHANGE): wwp0s29f7u1i7: link becomes ready

[Dan (dan-redhat-bugs)]

Created attachment 819297
sierra-send-sync-on-open.patch

[John (john-redhat-bugs)]

The patch works. I've applied it to sierra_net.c in the kernel 3.11 drivers (my current installation).

It's working with both the 312U and the 320U.

Thank you very much Dan and Bjørn.

John

[John (john-redhat-bugs)]

For the record I should document a further development. The patch of sierra_net is not required after all, and sierra_net is not the underlying cause of the problem.

Dan discovered and alerted me to the fact that the problem is generated by the usbnet.c driver. I have patched the usbnet.c driver in two 3.11 kernel releases now and can confirm that that patch gives me working Sierra 312U and 320U modems without any patching to sierra_net.c.

See: http://permalink.gmane.org/gmane.linux.usb.general/97907

John

[Dan (dan-redhat-bugs)]

I also tested the patch when it was posted upstream, and it works for my devices. I'm moving this bug to the kernel now that we know what the issue is.

For kernel people: this is commit 52f48d0d9aaa621ffa5e08d79da99a3f8c93b848 in dmiller's net tree and it's queued up for stable too.

[Josh (josh-redhat-bugs)]

Great, thanks Dan. This is already in rawhide and should hit the releases soon.

[Josh (josh-redhat-bugs)]

I went ahead and added this patch. It didn't make 3.11.10, which is the final 3.11 release.

[Fedora (fedora-redhat-bugs)]

kernel-3.11.10-300.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/kernel-3.11.10-300.fc20

[Fedora (fedora-redhat-bugs)]

Package kernel-3.11.10-300.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.11.10-300.fc20'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-22531/kernel-3.11.10-300.fc20
then log in and leave karma (feedback).

[Fedora (fedora-redhat-bugs)]

kernel-3.11.10-200.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/kernel-3.11.10-200.fc19

[Fedora (fedora-redhat-bugs)]

kernel-3.11.10-100.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/kernel-3.11.10-100.fc18

[Fedora (fedora-redhat-bugs)]

kernel-3.11.10-300.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.

[Fedora (fedora-redhat-bugs)]

kernel-3.11.10-200.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.

[Kai Kasurinen (kai-kasurinen)]

commit db6b5bf758051d320b6b3363ddd515760010e527
Author: Felix Fietkau <email address hidden>
Date: Tue Nov 12 16:34:41 2013 +0100

    usbnet: fix status interrupt urb handling

    [ Upstream commit 52f48d0d9aaa621ffa5e08d79da99a3f8c93b848 ]

    Since commit 7b0c5f21f348a66de495868b8df0284e8dfd6bbf
    "sierra_net: keep status interrupt URB active", sierra_net triggers
    status interrupt polling before the net_device is opened (in order to
    properly receive the sync message response).

    To be able to receive further interrupts, the interrupt urb needs to be
    re-submitted, so this patch removes the bogus check for netif_running().

...

https://www.kernel.org/pub/linux/kernel/v3.x/ChangeLog-3.12.4

[Kai Kasurinen (kai-kasurinen)]

Fixed:

linux (3.12.0-7.15) trusty; urgency=low

  [ Tim Gardner ]

  * Rebase to v3.12.4
  * Release tracker
    - LP: #1258947

  [ Upstream Kernel Changes ]

  * rebase to v3.12.4

 -- Tim Gardner <email address hidden> Sun, 08 Dec 2013 10:22:23 -0700

[Fedora (fedora-redhat-bugs)]

kernel-3.11.10-100.fc18 has been pushed to the Fedora 18 stable repository. If problems still persist, please make note of it in this bug report.

[Dirk (rptq)]

Will the fix from kernel bug 60521 be applied to the Saucy kernel as well? Currently I'm running a custom kernel with that fix in place and since then my Sierra Wireless modem (330U) is working again.

[Dirk (rptq)]

Apparently kernel 3.11.0-17-generic has the fix.
LP: #1275899