Ethernet not working on rpi3 b+

[Paul Larson]

[Impact]

Booting this image:

http://cdimage.ubuntu.com/releases/18.04/beta/ubuntu-18.04-beta-preinstalled-server-arm64+raspi3.img.xz

on a RaspberryPi 3B+, the ethernet port is not working.

Moving the EEE enabling into the PHY initialisation function, immediately after the connection to the PHY is established, which is long before phy_start is called, prevents the renegotiation and avoids the slowdown and failure.

[Fix]

Apply the attached patch and recompile

[Test]

Install a patched kernel and reboot, if the image boots and eth0 is
working, the fix is working

[Regression]

The patch is already in Cosmic and it fixes an isolated bug, so i say it's low

--

I'm trying a new image with kernel 4.15.0-1029.31 and ethernet works fine on my older rpi3 (v1.2) but with the b+ it doesn't seem to be working. Lower right LED is blinking, lower left is off.

I'll try to get some output via serial, no other way to connect to it at the moment

[mahmoh (mahmoh)]

As a work-around, I've noticed on the B+ that if you keep the Ethernet disconnected until it booting and then connect it when you see it hit "... Networking" then it'll pick up DHCP and then run cloud-init next happily.

[Steve Langasek (vorlon)]

Which version of the linux-raspi2 kernel last had ethernet working reliably?

Paul, should this bug not have blocked release of 4.15.0-1029.31 to -updates?

[Paolo Pisati (p-pisati)]

ubuntu@raspi64:~$ uname -a
Linux raspi64 4.15.0-1029-raspi2 #31-Ubuntu SMP PREEMPT Sat Nov 17 00:01:10 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

ubuntu@raspi64:~$ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 192.168.2.185 netmask 255.255.255.0 broadcast 192.168.2.255
        inet6 fe80::ba27:ebff:fed1:3a87 prefixlen 64 scopeid 0x20<link>
        ether b8:27:eb:d1:3a:87 txqueuelen 1000 (Ethernet)
        RX packets 78 bytes 9999 (9.9 KB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 68 bytes 11051 (11.0 KB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

this on a manually built arm64 image on the rpi3bp - ethernet works fine.

[Paolo Pisati (p-pisati)]

Ok, i can reproduce the issue, and while debugging it i found something else:

1) flash-kernel is broken

...
ubuntu@ubuntu:~$ sudo dpkg -i linux-image-4.15.0-1029-raspi2_4.15.0-1029.31~1806108take2_arm64.deb
dpkg: warning: downgrading linux-image-4.15.0-1029-raspi2 from 4.15.0-1029.31 to 4.15.0-1029.31~1806108take2
(Reading database ... 61762 files and directories currently installed.)
Preparing to unpack linux-image-4.15.0-1029-raspi2_4.15.0-1029.31~1806108take2_arm64.deb ...
Unpacking linux-image-4.15.0-1029-raspi2 (4.15.0-1029.31~1806108take2) over (4.15.0-1029.31) ...
Setting up linux-image-4.15.0-1029-raspi2 (4.15.0-1029.31~1806108take2) ...
Processing triggers for linux-image-4.15.0-1029-raspi2 (4.15.0-1029.31~1806108take2) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.15.0-1029-raspi2
flash-kernel: deferring update (trigger activated)
raspi3-firmware: deferring update via initramfs-tools (hook will be called later by the kernel package)
/etc/kernel/postinst.d/x-grub-legacy-ec2:
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... Generating /boot/grub/default file and setting the default boot entry to 0
Searching for GRUB installation directory ... found: /boot/grub
Testing for an existing GRUB menu.lst file ...

Could not find /boot/grub/menu.lst file. Would you like /boot/grub/menu.lst generated for you? (y/N) /usr/sbin/update-grub-legacy-ec2: line 1101: read: read error: 0: Bad file descriptor
run-parts: /etc/kernel/postinst.d/x-grub-legacy-ec2 exited with return code 1
dpkg: error processing package linux-image-4.15.0-1029-raspi2 (--install):
 installed linux-image-4.15.0-1029-raspi2 package post-installation script subprocess returned error exit status 1
Processing triggers for flash-kernel (3.90ubuntu3) ...
Unsupported platform.
dpkg: error processing package flash-kernel (--install):
 installed flash-kernel package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 linux-image-4.15.0-1029-raspi2
 flash-kernel
...

2) i saw a couple of systemd services fail:

ubuntu@ubuntu:~$ systemctl status systemd-modules-load.service
● systemd-modules-load.service - Load Kernel Modules
   Loaded: loaded (/lib/systemd/system/systemd-modules-load.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2018-01-28 15:58:19 UTC; 10 months 5 days ago
     Docs: man:systemd-modules-load.service(8)
           man:modules-load.d(5)
  Process: 676 ExecStart=/lib/systemd/systemd-modules-load (code=exited, status=1/FAILURE)
 Main PID: 676 (code=exited, status=1/FAILURE)

Jan 28 15:58:19 ubuntu systemd-modules-load[676]: Inserted module 'iscsi_tcp'
Jan 28 15:58:19 ubuntu systemd-modules-load[676]: Failed to find module 'ib_iser'
Jan 28 15:58:19 ubuntu systemd[1]: systemd-modules-load.service: Main process exited, code=exited, status=1/FAILURE
Jan 28 15:58:19 ubuntu systemd[1]: systemd-modules-load.service: Failed with result 'exit-code'.
Jan 28 15:58:19 ubuntu systemd[1]: Failed to start Load Kernel Modules.

this one seems just noise, while i don't know about this one:

ubuntu@...

[Paul Larson (pwlars)]

This is the first time I think we've been asked to test something on rpi3 arm64. The core 16 images we normally test the kernel snaps on are 4.4 based. I did try this on a core18 image and was unable to reproduce the problem, but it's not arm64. So this could be arm64 related, or as Paolo seems to be finding, could be related to something else like flash-kernel?

[Chris Wayne (cwayne)]

@Steve

My understanding from Paul is that we're only seeing this issue currently with the arm64 image we tested last week. While we're still standing up the b+ automated testing infrastructure for core18 (we currently need to working around some strange snapd behaviors), we just tested manually and are unable to reproduce this issue on the armhf core18 image with the same kernel. @Paul please correct me if I'm wrong here

[Paolo Pisati (p-pisati)]

Another issue: uboot's 'saveenv' doesn't work - stop at the uboot prompt, type 'saveenv' and the bootloader will "panic" - so there's no way to change the bootloader environment.

And a question: why copying vmlinuz and initrd to mmcblk0p1 and then loading from there, when we can point uboot to load them from mmclk0p2 /boot/?

[Paolo Pisati (p-pisati)]

Please, test linux-image and linux-modules from http://people.canonical.com/~ppisati/lp1806108/, after 'dpkg -i' both, copy vmlinuz and initrd.img from /boot to /boot/firmware.

The image still complains on boot though:

'Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings' but ethernet is working for me now - hope someone can explain what that message means.

[Paolo Pisati (p-pisati)]

Ok, new and final kernel:

http://people.canonical.com/~ppisati/lp1806108/

This the kernel has the hotfix for lp806108, get this tested and if it's ok, i'll push it

[Paolo Pisati (p-pisati)]

And for the record, the source code for the above kernel is here:

https://git.launchpad.net/~p-pisati/ubuntu/+source/linux/log/?h=b-raspi2_lp1806108

and this is the actual fix:

https://git.launchpad.net/~p-pisati/ubuntu/+source/linux/commit/?h=b-raspi2_lp1806108&id=4a415eec6df96dedef98977d043895355de7aa3c

I cherry-picked it from cosmic, hence the double "cherry picked from commit ..." line.

[Terry Rudd (terrykrudd)]

Nice work on this effort Paolo!

I followed all the progress in IRC.. nice and handy that smb had a
devise to test too.

Thank you

Terry

On 12/05/2018 06:27 AM, Paolo Pisati wrote:
> And for the record, the source code for the above kernel is here:
>
> https://git.launchpad.net/~p-pisati/ubuntu/+source/linux/log/?h=b-raspi2_lp1806108
>
> and this is the actual fix:
>
> https://git.launchpad.net/~p-pisati/ubuntu/+source/linux/commit/?h=b-raspi2_lp1806108&id=4a415eec6df96dedef98977d043895355de7aa3c
>
> I cherry-picked it from cosmic, hence the double "cherry picked from
> commit ..." line.
>

[Paul Larson (pwlars)]

I did get some errors when trying to install this, but I copied the vmlinuz and initrd to /boot/firmware and I can get an ip now.

I retried the install and still get the errors, here's the log of that:
ubuntu@ubuntu:/mnt$ sudo dpkg -i linux-image-4.15.0-1029-raspi2_4.15.0-1029.31_final1_arm64.deb
(Reading database ... 61742 files and directories currently installed.)
Preparing to unpack linux-image-4.15.0-1029-raspi2_4.15.0-1029.31_final1_arm64.deb ...
Unpacking linux-image-4.15.0-1029-raspi2 (4.15.0-1029.31~final1) over (4.15.0-1029.31~final1) ...
/var/lib/dpkg/info/linux-image-4.15.0-1029-raspi2.postrm ... removing pending trigger
Setting up linux-image-4.15.0-1029-raspi2 (4.15.0-1029.31~final1) ...
Processing triggers for linux-image-4.15.0-1029-raspi2 (4.15.0-1029.31~final1) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.15.0-1029-raspi2
flash-kernel: deferring update (trigger activated)
raspi3-firmware: deferring update via initramfs-tools (hook will be called later by the kernel package)
/etc/kernel/postinst.d/x-grub-legacy-ec2:
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ...

Could not find /boot/grub/menu.lst file. Would you like /boot/grub/menu.lst generated for you? (y/N) /usr/sbin/update-grub-legacy-ec2: line 1101: read: read error: 0: Bad file descriptor
run-parts: /etc/kernel/postinst.d/x-grub-legacy-ec2 exited with return code 1
dpkg: error processing package linux-image-4.15.0-1029-raspi2 (--install):
 installed linux-image-4.15.0-1029-raspi2 package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 linux-image-4.15.0-1029-raspi2

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-raspi2 - 4.15.0-1030.32

---------------
linux-raspi2 (4.15.0-1030.32) bionic; urgency=medium

  * linux-raspi2: 4.15.0-1030.32 -proposed tracker (LP: #1806661)

  * Ethernet not working on rpi3 b+ (LP: #1806108)
    - lan78xx: Move enabling of EEE into PHY init code

  [ Ubuntu: 4.15.0-43.46 ]

  * linux: 4.15.0-43.46 -proposed tracker (LP: #1806659)
  * System randomly hangs during suspend when mei_wdt is loaded (LP: #1803942)
    - SAUCE: base/dd: limit release function changes to vfio driver only
  * Workaround CSS timeout on AMD SNPS 3.0 xHC (LP: #1806838)
    - xhci: Allow more than 32 quirks
    - xhci: workaround CSS timeout on AMD SNPS 3.0 xHC
  * linux-buildinfo: pull out ABI information into its own package
    (LP: #1806380)
    - [Packaging] limit preparation to linux-libc-dev in headers
    - [Packaging] commonise debhelper invocation
    - [Packaging] ABI -- accumulate abi information at the end of the build
    - [Packaging] buildinfo -- add basic build information
    - [Packaging] buildinfo -- add firmware information to the flavour ABI
    - [Packaging] buildinfo -- add compiler information to the flavour ABI
    - [Packaging] buildinfo -- add buildinfo support to getabis
    - [Config] buildinfo -- add retpoline version markers
  * linux packages should own /usr/lib/linux/triggers (LP: #1770256)
    - [Packaging] own /usr/lib/linux/triggers
  * CVE-2018-12896
    - posix-timers: Sanitize overrun handling
  * CVE-2018-16276
    - USB: yurex: fix out-of-bounds uaccess in read handler
  * CVE-2018-10902
    - ALSA: rawmidi: Change resized buffers atomically
  * CVE-2018-18710
    - cdrom: fix improper type cast, which can leat to information leak.
  * CVE-2018-18690
    - xfs: don't fail when converting shortform attr to long form during
      ATTR_REPLACE
  * CVE-2018-14734
    - infiniband: fix a possible use-after-free bug
  * CVE-2018-18445
    - bpf: 32-bit RSH verification must truncate input before the ALU op
  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts

 -- Khalid Elmously <email address hidden> Thu, 06 Dec 2018 17:38:20 +0000