Full RAM on Pi4 isn't accessible when using u-boot

[Dave Jones]

Booting the latest eoan daily on a Pi4 with >1Gb of RAM, only the first 1Gb of RAM appears accessible:

    ubuntu@ubuntu:~$ free
                  total used free shared buff/cache available
    Mem: 931920 185484 504748 3664 241688 723820
    Swap: 0 0 0

However, adjusting the config.txt so that the kernel is booted directly without u-boot in between results in the full 4Gb being accessible:

    ubuntu@ubuntu:~$ free
                  total used free shared buff/cache available
    Mem: 3884464 189852 3457504 3656 237108 3636004
    Swap: 0 0 0

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Hui Wang (hui.wang)]

This is the log of armhf kernel + armhf uboot:

MMC: emmc2@7e340000: 0, mmcnr@7e300000: 1
Loading Environment from FAT... *** Warning - bad CRC, using default environment

In: serial
Out: serial
Err: serial
Net: Net Initialization Skipped
No ethernet found.
Unknown command 'usb' - try 'help'
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
777 bytes read in 47 ms (15.6 KiB/s)
## Executing script at 02400000
7315968 bytes read in 1424 ms (4.9 MiB/s)
20730891 bytes read in 3932 ms (5 MiB/s)
Kernel image @ 0x1000000 [ 0x000000 - 0x6fa200 ]
## Flattened Device Tree blob at 03000000
   Booting using the fdt blob at 0x3000000
   Using Device Tree in place at 03000000, end 0300d350

Starting kernel ...

[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 5.3.0-1004-raspi2 (root@hwang4-Vostro-5390) (gcc version 7.4.0 (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04.1)) #5 SMP Wed Sep 18 23:44:29 CST 2019 (Ubuntu 5.3.0-1004.5-raspi2 5.3.0)
[ 0.000000] CPU: ARMv7 Processor [410fd083] revision 3 (ARMv7), cr=30c5383d
[ 0.000000] CPU: div instructions available: patching division code
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
[ 0.000000] OF: fdt: Machine model: Raspberry Pi 4 Model B Rev 1.1
[ 0.000000] Memory policy: Data cache writealloc

....

ubuntu@ubuntu:~$ uname -a
Linux ubuntu 5.3.0-1004-raspi2 #5 SMP Wed Sep 18 23:44:29 CST 2019 armv7l armv7l armv7l GNU/Linux
ubuntu@ubuntu:~$ free
              total used free shared buff/cache available
Mem: 3916244 145836 3600732 21912 169676 3704604
Swap: 0 0 0
ubuntu@ubuntu:~$ sudo ifconfig wlan0 up
ubuntu@ubuntu:~$ sudo iwlist wlan0 scanning
wlan0 Scan completed :
          Cell 01 - Address: 58:C8:76:6B:C3:11
                    Channel:36
                    Frequency:5.18 GHz (Channel 36)
                    Quality=34/70 Signal level=-76 dBm
                    Encryption key:on
                    ESSID:"CMCC-T69d-5G"
                    Bit Rates:6 Mb/s; 9 Mb/s; 12 Mb/s; 18 Mb/s; 24 Mb/s
                              36 Mb/s; 48 Mb/s; 54 Mb/s
                    Mode:Master
                    Extra:tsf=0000000000000000
                    Extra: Last beacon: 4ms ago
                    IE: Unknown: 000C434D43432D543639642D3547
                    IE: Unknown: 01088C129824B048606C
                    IE: Unknown: 030124
                    IE: Unknown: 05040001000C
                    IE: WPA Version 1
                        Group Cipher : TKIP
                        Pairwise Ciphers (2) : TKIP CCMP
                        Authentication Suites (1) : PSK
                    IE: IEEE 802.11i/WPA2 Version 1
                        Group Cipher : TKIP
                        Pairwise Ciphers (2) : TKIP CCMP
                        Authentication Suites (1) : PSK
                    IE: Unknown: DD270050F204104A000110104400010210470010BC329E001DD811B2860158C8766BC311103C000102
                    IE: Unknown: 2D1AEF0117FFFF0000010000000000000000000000000C00...

[Hui Wang (hui.wang)]

For WiFi, I remember I found the firmware from someplace, then the WiFi worked.

ubuntu@ubuntu:~$ ls /lib/firmware/brcm/brcmfmac43455-sdio.* -la
-rw-r--r-- 1 root root 600487 Sep 1 05:45 /lib/firmware/brcm/brcmfmac43455-sdio.bin
-rw-r--r-- 1 root root 14036 Sep 1 05:45 /lib/firmware/brcm/brcmfmac43455-sdio.clm_blob
-rw-r--r-- 1 root root 2054 Sep 1 05:45 /lib/firmware/brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.txt
-rw-r--r-- 1 root root 2054 Sep 1 05:45 /lib/firmware/brcm/brcmfmac43455-sdio.txt
ubuntu@ubuntu:~$ ls /lib/firmware/brcm/brcmfmac43455-sdio.* | xargs md5sum
963eb0d4903040974ee88b4f85cb1f4f /lib/firmware/brcm/brcmfmac43455-sdio.bin
c5aeca0e33de4ae870986c517963fef7 /lib/firmware/brcm/brcmfmac43455-sdio.clm_blob
7b983812a8aee7bf701460436c686226 /lib/firmware/brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.txt
7b983812a8aee7bf701460436c686226 /lib/firmware/brcm/brcmfmac43455-sdio.txt

[Hui Wang (hui.wang)]

The uboot of armhf and configs I used for testing are:

https://github.com/agherzan/u-boot.git commit a671dabc696fc9505641b418f473ce6c6792af93 (HEAD -> v2019.07-rpi4-wip, origin/ag/v2019.07-rpi4-wip)

hwang4@hwang4-Vostro-5390:~/work/uboot/u-boot$ cat boot.cmd
setenv fdt_addr_r 0x03000000
fdt addr ${fdt_addr_r}
fdt get value bootargs /chosen bootargs
setenv bootargs "coherent_pool=1M 8250.nr_uarts=1 cma=64M bcm2708_fb.fbwidth=1824 bcm2708_fb.fbheight=984 bcm2708_fb.fbswap=1 smsc95xx.macaddr=DC:A6:32:0E:9B:C4 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 dwc_otg.fiq_fix_enable=2 console=ttyS0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 fsck.repair=yes rootwait rootflags=noload net.ifnames=0"
setenv kernel_addr_r 0x01000000
setenv ramdisk_addr_r 0x03100000
fatload mmc 0:1 ${kernel_addr_r} Image
fatload mmc 0:1 ${ramdisk_addr_r} initrd.img
setenv initrdsize $filesize
bootz ${kernel_addr_r} ${ramdisk_addr_r}:${initrdsize} ${fdt_addr_r}

hwang4@hwang4-Vostro-5390:~/work/uboot/u-boot$ cat /media/hwang4/system-boot/config.txt
[pi4]
# Enable DRM VC4 V3D driver on top of the dispmanx display stack
dtoverlay=vc4-fkms-v3d
max_framebuffers=2

enable_uart=1
#kernel=Image
kernel=u-boot.bin
device_tree_address=0x03000000
dtparam=i2c_arm=on
dtparam=spi=on
ramfsfile=initrd.img
ramfsaddr=0x3100000
#total_mem=2048

# Enable audio (loads snd_bcm2835)
dtparam=audio=on
disable_audio_dither=on

hwang4@hwang4-Vostro-5390:~/work/uboot/u-boot$ cat /media/hwang4/system-boot/cmdline.txt
dwc_otg.fiq_fix_enable=2 console=ttyS0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait rootflags=noload net.ifnames=0 snd_bcm2835.enable_headphones=1 snd_bcm2835.enable_hdmi=1 snd_bcm2835.enable_compat_alsa=1 initrd=0x3100000,0x13c540b

[Hui Wang (hui.wang)]

So let us do a comparison according to #4, maybe we could find sth different between yours and mine.

thx.

[Dave Jones (waveform)]

> For WiFi, I remember I found the firmware from someplace, then the WiFi worked.

> ubuntu@ubuntu:~$ ls /lib/firmware/brcm/brcmfmac43455-sdio.* -la
> -rw-r--r-- 1 root root 600487 Sep 1 05:45 /lib/firmware/brcm/brcmfmac43455-sdio.bin
> -rw-r--r-- 1 root root 14036 Sep 1 05:45 /lib/firmware/brcm/brcmfmac43455-sdio.clm_blob
> -rw-r--r-- 1 root root 2054 Sep 1 05:45 /lib/firmware/brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.txt
> -rw-r--r-- 1 root root 2054 Sep 1 05:45 /lib/firmware/brcm/brcmfmac43455-sdio.txt
> ubuntu@ubuntu:~$ ls /lib/firmware/brcm/brcmfmac43455-sdio.* | xargs md5sum
> 963eb0d4903040974ee88b4f85cb1f4f /lib/firmware/brcm/brcmfmac43455-sdio.bin
> c5aeca0e33de4ae870986c517963fef7 /lib/firmware/brcm/brcmfmac43455-sdio.clm_blob
> 7b983812a8aee7bf701460436c686226 /lib/firmware/brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.txt
> 7b983812a8aee7bf701460436c686226 /lib/firmware/brcm/brcmfmac43455-sdio.txt

Most of those hashes match the latest-but-one variant of the wifi firmware from Raspbian. Unfortunately, updating to the very latest is tricky for us as the firmware blobs appear split between linux-firmware (which holds the *.bin files) and linux-firmware-raspi (which holds the *.txt and *.clm_blob) files. After a bit of experimentation I've found that updating just the 43455 clm_blob and txt files fixes wifi for the Pi4 without breaking wifi on any of the earlier versions.

I'll open a separate ticket for this with a link to a PPA package and see if I can get it sponsored.

[Dave Jones (waveform)]

Figured it out; you're using the v2019.07-rpi4-wip branch while I based the latest u-boot on the rpi4 branch (which is slightly later and appears a bit more polished). After a bit of experimentation it turns out the crucial difference is in the config, specifically in a comment (which is why it took me a while to notice!):

    # CONFIG_ARCH_FIXUP_FDT_MEMORY is not set

This line is missing from the rpi4 branch. Because ARCH_FIXUP_FDT_MEMORY defaults to "y" this is enough to cause a call to arch_fixup_memory_banks() in the packaged version of u-boot, which is what causes the RAM to be limited to 1Gb (I haven't dug into the mechanism why; for now I'm content to flip the config switch and have a version that allows full access to the RAM, but I might have a look at this if I get a bit of time next week).

Anyway, I'll get a new version of u-boot built and see if I can get someone to sponsor it for release (might be too late, but it's not the end of the world if it doesn't make it into the release; it'll be a trivial update for users).

[Hui Wang (hui.wang)]

On 2019/10/11 下午10:18, Dave Jones wrote:
>> For WiFi, I remember I found the firmware from someplace, then the
> WiFi worked.
>
>> ubuntu@ubuntu:~$ ls /lib/firmware/brcm/brcmfmac43455-sdio.* -la
>> -rw-r--r-- 1 root root 600487 Sep 1 05:45 /lib/firmware/brcm/brcmfmac43455-sdio.bin
>> -rw-r--r-- 1 root root 14036 Sep 1 05:45 /lib/firmware/brcm/brcmfmac43455-sdio.clm_blob
>> -rw-r--r-- 1 root root 2054 Sep 1 05:45 /lib/firmware/brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.txt
>> -rw-r--r-- 1 root root 2054 Sep 1 05:45 /lib/firmware/brcm/brcmfmac43455-sdio.txt
>> ubuntu@ubuntu:~$ ls /lib/firmware/brcm/brcmfmac43455-sdio.* | xargs md5sum
>> 963eb0d4903040974ee88b4f85cb1f4f /lib/firmware/brcm/brcmfmac43455-sdio.bin
>> c5aeca0e33de4ae870986c517963fef7 /lib/firmware/brcm/brcmfmac43455-sdio.clm_blob
>> 7b983812a8aee7bf701460436c686226 /lib/firmware/brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.txt
>> 7b983812a8aee7bf701460436c686226 /lib/firmware/brcm/brcmfmac43455-sdio.txt
> Most of those hashes match the latest-but-one variant of the wifi
> firmware from Raspbian. Unfortunately, updating to the very latest is
> tricky for us as the firmware blobs appear split between linux-firmware
> (which holds the *.bin files) and linux-firmware-raspi (which holds the
> *.txt and *.clm_blob) files. After a bit of experimentation I've found
> that updating just the 43455 clm_blob and txt files fixes wifi for the
> Pi4 without breaking wifi on any of the earlier versions.
>
> I'll open a separate ticket for this with a link to a PPA package and
> see if I can get it sponsored.
OK,  thanks for sharing this information.
>

[Launchpad Janitor (janitor)]

This bug was fixed in the package u-boot - 2019.07+dfsg-1ubuntu3

---------------
u-boot (2019.07+dfsg-1ubuntu3) eoan; urgency=medium

  * Avoid device-tree memory fixup on Raspberry Pi 4; this allows access to
    the all the RAM on models with more than 1Gb (LP: #1847500)

 -- Dave Jones <email address hidden> Sat, 12 Oct 2019 01:02:29 +0100

[satmandu (satadru-umich)]

FYI ram was limited to 1Gb to avoid issues with DMA since the DMA needs to be restricted to the first 1Gb of RAM.

FYI mainline u-boot 2010/v2019.10 boots just fine with 4GB made available when using rpi_4_defconfig . No patching is needed.

[satmandu (satadru-umich)]

Whoops, I meant mainline u-boot using tag v2019.10.

(The ram restriction was more complicated on arm64 since there was some bounce buffer voodoo that needed to be done, IIRC.)

git clone --depth 1 --branch v2019.10 https://github.com/u-boot/u-boot.git
cd u-boot
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make rpi_4_defconfig
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make -j $(($(nproc) + 1))
sudo cp u-boot.bin /boot/firmware/uboot_rpi_4.bin

arm64 boot dmesg on arm64: https://paste.ubuntu.com/p/Z8Rn59Dsw3/

@rpi4:~$ free -m
              total used free shared buff/cache available
Mem: 3806 689 2852 10 263 3052
Swap: 4095 0 4095

[Dave Jones (waveform)]

@satmandu thanks for the information! I'd got as far as figuring out the dram_init call was getting the available RAM info from the mailbox, which only permitted a single bank to be reported, which was then getting used in the fdt fixup called by the bootm. As it happens the fdt /memory node was already correct so there was no need for that particular fixup. Hence the hack applied for now is just to disable the fdt fixup.

Bumping to 2019.10 would be ideal, but unfortunately it's now too late in the cycle before release to re-test all the other platforms using u-boot. Still, it's useful to know that I can scrub that hack when we bump to 2019.10 (which we can probably do next cycle) - saves me digging further into it!