[SRU] Bluetooth won't activate on the pi 400

Bug #1903048 reported by Dave Jones on 2020-11-05
44
This bug affects 5 people
Affects Status Importance Assigned to Milestone
bluez (Ubuntu)
Status tracked in Hirsute
Groovy
Undecided
Unassigned
Hirsute
High
Dave Jones

Bug Description

[Impact]

Without these patches, Bluetooth is inoperable on the recently released Raspberry Pi 400.

[Test Case]

* Boot the Ubuntu Desktop for Pi image on a Pi 400.
* Start the Settings application and switch to the Bluetooth tab
* Verify that Bluetooth is not enabled and attempting to activate it fails
* Enable the -proposed repository for the release (groovy)
* sudo apt update
* sudo apt install bluez
* sudo reboot
* Start the Settings application and switch to the Bluetooth tab
* Verify that Bluetooth is active and that Bluetooth devices (e.g. mice, mobile phones, headphones, etc.) can connect and operate correctly

[Regression Potential]

Extremely low (on groovy in particular, this has the same version of bluez as hirsute). The only significant risk is to non-Pi platforms or dongles which also use the Broadcom 43xx (or Cypress 305) chips for Bluetooth which might be inadvertently affected by these patches.

[Original Description]

The new Pi 400 has a slightly different Wifi/BT chip to the Pi4. Whilst wifi works happily, Bluetooth fails to operate. This doesn't appear to be an issue with either the firmware (the latest versions from upstream Raspbian have been tried), or the kernel (a known-good raspi kernel has been tested under Ubuntu), but with Bluez itself. Specifically, tracing the initialization with btmon on Raspbian and Ubuntu, the stack consistently fails when attempting to "Set Default PHY" on the latter. Curiously, under Ubuntu the adapter also appears to lack a MAC address.

Dave Jones (waveform) on 2020-11-05
Changed in bluez (Ubuntu):
assignee: nobody → Dave Jones (waveform)
tags: added: raspi
Henry Sprog (henry-sprog) wrote :

Is this the same as: #1903286?

Dave Jones (waveform) wrote :

@henry-sprog yup, that looks like the same

Attached a patch to fix this in hirsute; once landed will SRU this to groovy and earlier.

The attachment "lp1903048.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
tags: added: fr-901
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in bluez (Ubuntu):
status: New → Confirmed
tags: added: groovy
tags: added: hirsute
Łukasz Zemczak (sil2100) wrote :

Ok, I was asked to sponsor it, so I'm pushing the changes to hirsute and groovy.

I have filled MPs for both for the bluez repo (no write access). Also, a hirsute branch needs to be opened, so the first MP is for now only targeting groovy (invalidly):
https://code.launchpad.net/~sil2100/bluez/+git/bluez/+merge/393548
https://code.launchpad.net/~sil2100/bluez/+git/bluez/+merge/393549

Cheers.

Dave Jones (waveform) on 2020-11-10
summary: - Bluetooth won't activate on the pi 400
+ [SRU] Bluetooth won't activate on the pi 400
Dave Jones (waveform) on 2020-11-10
description: updated
Changed in bluez (Ubuntu):
importance: Undecided → High
status: Confirmed → Fix Committed
Sebastien Bacher (seb128) wrote :

Thanks for the work but it there any work to upstream those changes? I'm not happy to carry a sleep(1) hack in our package unless there is a strong reason and we are working on a way to replace it by a better solution

Henry Sprog (henry-sprog) wrote :

Test case works perfectly on the Pi400.

Henry Sprog (henry-sprog) wrote :

Thanks guys for all the work.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package bluez - 5.55-0ubuntu2

---------------
bluez (5.55-0ubuntu2) hirsute; urgency=medium

  * Added patches from the Raspberry Pi Foundation
    - d/p/raspi-bcm43xx-load-firmware.patch
    - d/p/raspi-bcm43xx-3wire.patch
    - d/p/raspi-cypress-305-bdaddr.patch
  * These patches fix Bluetooth operation on the Pi 400 (LP: #1903048)

 -- Dave Jones <email address hidden> Thu, 05 Nov 2020 13:39:07 +0000

Changed in bluez (Ubuntu Hirsute):
status: Fix Committed → Fix Released
Daniel van Vugt (vanvugt) wrote :

Łukasz and Dave, thanks for the fix.

I have renamed 'ubuntu' to 'hirsute' to keep with the existing naming scheme: https://git.launchpad.net/~bluetooth/bluez

Hi,

I implemented the fix detailed in the test case yesterday and the bug
still existed. I've double checked again this morning and done a
full-upgrade just to check nothing new was waiting and the bug still
exists on this Mate 20.10 install. Yesterday after applying the fix I
started getting random "System problem detected, report this?" popups
but none so far today.

Blueman Applet is set as a startup item and both blueman-applet and
blueman-tray appear as running processes but running blueman-manager or
blueman-adapters from the command line both fail with the message
'Blueman applet needs to be running' The Bluetooth panel item isnt shown.

OS: Ubuntu MATE 20.10 aarch64
Host: Raspberry Pi 400 Rev 1.0
Kernel: 5.8.0-1007-raspi

Tony

On 11/11/2020 04:32, Launchpad Bug Tracker wrote:
> This bug was fixed in the package bluez - 5.55-0ubuntu2
>
> ---------------
> bluez (5.55-0ubuntu2) hirsute; urgency=medium
>
> * Added patches from the Raspberry Pi Foundation
> - d/p/raspi-bcm43xx-load-firmware.patch
> - d/p/raspi-bcm43xx-3wire.patch
> - d/p/raspi-cypress-305-bdaddr.patch
> * These patches fix Bluetooth operation on the Pi 400 (LP: #1903048)
>
> -- Dave Jones <email address hidden> Thu, 05 Nov 2020 13:39:07
> +0000
>
> ** Changed in: bluez (Ubuntu Hirsute)
> Status: Fix Committed => Fix Released
>

Dave Jones (waveform) wrote :

On Tue, Nov 10, 2020 at 03:12:30PM -0000, Sebastien Bacher wrote:
>Thanks for the work but it there any work to upstream those changes? I'm
>not happy to carry a sleep(1) hack in our package unless there is a
>strong reason and we are working on a way to replace it by a better
>solution

The changes come from the Pi Foundation originally (though I've tidied
them up a tiny bit). They're not interested in upstreaming them
themselves, though they expressed no objections to my attempting to do
so when I raised the question.

I realize I'll probably have a bit of a battle on my hands with
something like the patch involving sleep(1), but I would note that the
sleep(1) hack is in the hciattach_bcm43xx module so it is not something
that would be applied to every single Bluetooth module, only to those
compatible with the BCM43xx line.

While 1 second is obviously a suspiciously round number for the delay, I
would further note that it's immediately after a firmware load over the
controlling UART and that that UART can (in certain configurations) be
the Pi's mini-UART which has some (minimal) buffering, but lacks any
hardware flow control. Finally, these patches were also intended to work
on the relatively slow, single core Pi Zero.

Obviously we don't support the Zero, and thus it's possible *we* might
be able to either do away with that sleep, or at least reduce the delay
required, but given this is one-off initialization, and the delay is all
of 1 second, I'm not convinced it's worth investing the time required to
figure out the minimum. Also, I would assume in the interests of
upstreaming the patch, the delay should be as widely applicable as
possible.

Anyway, I'll upstream what I can and we'll have to carry whatever's left
over.

Sebastien Bacher (seb128) wrote :

Sorry but I'm reverting that upload for now until the patches are properly upstreamed. We have been bitten too often by unforwarded changes that create issues or create maintainance burden over the years and we currently don't have the team capacity to deal with extra cost. If foundations would like to step up and take over bluez though that's a discussion we could have...

Changed in bluez (Ubuntu Hirsute):
status: Fix Released → Triaged
Sebastien Bacher (seb128) wrote :

> I'm not convinced it's worth investing the time required to figure out the minimum.

The value of the sleep is not really the issue but it feels like there should be a better way to tell when the system is ready, what are we waiting for? Is there any api call we could do to check if the thing we need is ready?

Henry Sprog (henry-sprog) wrote :

I understand the logic of the of doing this and don't disagree.
However, as this is an entry level device and I believe the only true
64bit O/S advertised for this device, the fact that it does not not
work out of the box may impact on the perception of potential new
Ubuntu users?

Reading Tony's post the Mate website does not say it will work on the
PI400 where as the Ubuntu site does. I think the main issue is that
people think it is the same board as the Pi4 and it is not. Just my
opinion.

On Thu, 2020-11-12 at 11:11 +0000, Sebastien Bacher wrote:
> Sorry but I'm reverting that upload for now until the patches are
> properly upstreamed. We have been bitten too often by unforwarded
> changes that create issues or create maintainance burden over the
> years
> and we currently don't have the team capacity to deal with extra
> cost.
> If foundations would like to step up and take over bluez though
> that's a
> discussion we could have...
>
> ** Changed in: bluez (Ubuntu Hirsute)
> Status: Fix Released => Triaged
>

Łukasz Zemczak (sil2100) wrote :

I personally think the revert is a bit extreme. Even though we are not maintainers of bluez, this being a critical feature for the Pi400 we would of course make sure that the changes get upstreamed as much as possible and in the meantime maintain the patches ourselves as many others related to the Pi (as in various other projects). Without this fix the Pi400 is basically broken, so we wanted to unblock users as soon as possible (for groovy) - the certification team is pushing hard on this since a while. Was the revert discussed somewhere before being performed?

Sebastien Bacher (seb128) wrote :

> Was the revert discussed somewhere before being performed?

I pinged you guys on IRC some days ago but didn't get any comment back

Sebastien Bacher (seb128) wrote :

> in the meantime maintain the patches ourselves as many others related to the Pi (as in various other projects)

I'm not asking for the patches to be merged upstream but at least to be sent there for review and have the appropriate tagging, it's easy to unblock from your side. Also I would like to see if we can get for a better way to probe for the system to be ready rather than relying on a random sleep

David Krauser (davidkrauser) wrote :

On Ubuntu 20.10 MATE on a pi400, I ran through the following steps:

* Boot the Ubuntu 20.10 MATE for Pi image on a Pi 400 (Was not a fresh installation)
* Start the Settings application and switch to the Bluetooth tab
* Verify that Bluetooth is not enabled and attempting to activate it fails
* sudo add-apt-repository ppa:waveform/pi-bluetooth
* sudo apt update
* sudo apt install bluez # I noticed there were other packages in the PPA that I did not install
* sudo nvi /etc/apt/sources.list.d/waveform-ubuntu-pi-bluetooth-groovy.list # Comment out PPA
* sudo apt update && sudo apt upgrade -y
* sudo reboot
* Start the Settings application and switch to the Bluetooth tab
* Pair a set of AirPods
* Ensure I can hear audio through the AirPods

Everything seemed to work as expected. Thank you for the patches :-)

Chris Wayne (cwayne18) wrote :

What's the status of this? We desperately need this fix in Groovy, as we're making lots of noise about the pi400 and this is a pretty glaring hole in functionality.

@seb128: Do you think you can re-revert this change? we are going
to work with upstream and give them access to the patch. I don't think
we should should prevent the device to work until this is done.
As Chris said in #20, there's a lot of hype on the pi400 and
bluetooth not working is definitely affecting user experience.
Thx in Advance

Dave Jones (waveform) wrote :
Download full text (6.3 KiB)

On Thu, Nov 12, 2020 at 11:11:23AM -0000, Sebastien Bacher wrote:
>Sorry but I'm reverting that upload for now until the patches are
>properly upstreamed. We have been bitten too often by unforwarded
>changes that create issues or create maintainance burden over the years
>and we currently don't have the team capacity to deal with extra cost.
>If foundations would like to step up and take over bluez though that's a
>discussion we could have...

My apologies; I omitted to note in the patch that the Pi 400 is a
certified platform, i.e. we're effectively committed to making it work
as best we can, which makes the status of whether upstream accept the
patches or not rather moot:

If upstream say "yes" to the patches, without further discussion: that's
great, but there'd presumably still be some delay before another version
makes its way down to us, so we'd be applying *some* patch to make it
work in the interim.

If upstream say "with these changes" to the patches: again great, but in
the interim, we'd again be applying *some* patch to make things work on
a certified platform while working through changes upstream.

If upstream say "not in a month of Sundays" to these patches: we'd be
left carrying *some* patch, and we'd do it because it's a certified
platform and we're committed to making it work.

 From our end user's perspective therefore, the application of this
patch-set, whether via upstream or not, and whether in this form or not,
is not a matter of "if", but "when". Given we have in our possession a
patch-set that fixes the problem, I at least don't have a good answer to
the hypothetical user's obvious follow-up question: "why not now?"

Anyway, onto technical matters:

>I'm not asking for the patches to be merged upstream but at least to be
>sent there for review and have the appropriate tagging, it's easy to
>unblock from your side. Also I would like to see if we can get for a
>better way to probe for the system to be ready rather than relying on a
>random sleep

The following involves a certain amount of educated guessing. I haven't
dug into this extensively, but I can offer some reasons for why sleep(1)
is being used (and how this could be refined a bit, but not much):

Consider the bcm43xx_load_firmware routine in hciattach_bcm43xx.c [1]
which is being called prior to the apparently arbitrarily inserted
sleep(1) line in the patch. It's a fairly typical firmware load routine
by all accounts:

1. Calls write() with a command (0x2e 0xfc) to place the device into a
    state where it's read to receive new firmware

2. Calls read_hci_event() to check the device responded "okay!"

3. Calls nanosleep() to wait a short while (50ms) for the device to be
    ready

4. Calls read() and write() repeatedly to write the firmware blob from a
    file to the UART

5. Calls nanosleep() again to wait another short while (200ms) after the
    firmware load

So the existing (pre-patch) firmware load routine already has a
seemingly arbitrary post-firmware-load delay of 200ms. If we have a look
at the BCM4334 datasheet [2], p.159 we can see "wait at least 150ms
after [power on] before initiating SDIO accesses" (SDIO being one of ...

Read more...

Sebastien Bacher (seb128) wrote :

@Matthieu, @Dave, thanks for the comments and details.

I'm not blocking work to land, the 20.10 SRU could be accepted now and I didn't revert in that serie.
The SRU team tries to ensure the fix is in the new serie so it doesn't get forgotten/regress when next version is out but I think it's pretty clear we are going to sort out the review details, that's not a blocker and SRU team has been happy in the past to let things in with the understand than $newserie is being sorted out

> If upstream say "yes" to the patches, without further discussion: that's
> great, but there'd presumably still be some delay before another version
> makes its way down to us, so we'd be applying *some* patch to make it work in the interim.

Again I'm not asking for the patches to be reviewed or accepted upstream before they are uploaded to Ubuntu, I'm just asking for them to be sent upstream and the reference to the email/report/merge request to be added to the patches. It would probably have been less work to just do it rather than type a long explanation here on why we need to distro patch in any case (which I never disagreed with)

Sorry again but we have been bitten too often in similar situation where people never upstreamed the work when they said they would because they got busy with other things, and we are the ones paying the maintainance cost later on.

Just to close the loop, I've just made fresh SD and USB boot disks from
the latest current Mate 20.10 image and both don't have working
Bluetooth, Ive updated both with the ppa and now both do work and I can
use mouse and headphones with no problems. I have no idea why my
original USB boot of Mate 20.10 didnt work after patching, there must be
another problem with it as well, my apologies for any confusion caused
by that.

Martin and Alan have been nagging us 'ordinary' people to get involved
and contribute where we can to help you out if possible, that's all I
was trying to do. Thanks for the patch, hope you get it upstreamed, etc OK.

Best wishes

Tony

Sebastien Bacher (seb128) wrote :

There is no intend here to block any fix for stable serie but in case that was not clear enough or if the SRU team didn't feel comfortable with hirsute lacking behind and I uploaded back the patches but I'm tagging the bug as 'block-proposed' instead which should be enough incensitive to sort out

tags: added: block-proposed
David Krauser (davidkrauser) wrote :

@tmolloy, thank you for circling back around. I'm glad the patches worked for you :-)
Note that there are other (unrelated) packages in @waveform's PPA that may get installed with an apt upgrade. I don't know what the state of those packages are. I disabled the PPA after installing bluez, just to be sure I didn't unintentionally install the other packages.

Iain Lane (laney) wrote :

The point for me is that we should be trying to avoid taking on indefinite technical debt (patches that have to be rebased forever). It's "fine" to include them ahead of any upstream schedule if we need to for hardware enablement, but we do need to have plans to make this be a non permanent situation. That's what Seb is asking for, and I'm totally on board with that.

btw, I don't think it's fair to characterise code review as pedantry

On Fri, Nov 13, 2020 at 09:37:34AM -0000, Sebastien Bacher wrote:
>@Matthieu, @Dave, thanks for the comments and details.
>
>I'm not blocking work to land, the 20.10 SRU could be accepted now and I didn't revert in that serie.
>The SRU team tries to ensure the fix is in the new serie so it doesn't get forgotten/regress when next version is out but I think it's pretty clear we are going to sort out the review details, that's not a blocker and SRU team has been happy in the past to let things in with the understand than $newserie is being sorted out

Ah, my apologies - I had mis-interpreted the reversion as applying to
the SRU. I've no issue with a reversion in hirsute, provided it doesn't
delay existing users from using their Bluetooth hardware.

>> If upstream say "yes" to the patches, without further discussion: that's
>> great, but there'd presumably still be some delay before another version
>> makes its way down to us, so we'd be applying *some* patch to make it work in the interim.
>
>Again I'm not asking for the patches to be reviewed or accepted upstream
>before they are uploaded to Ubuntu, I'm just asking for them to be sent
>upstream and the reference to the email/report/merge request to be added
>to the patches. It would probably have been less work to just do it
>rather than type a long explanation here on why we need to distro patch
>in any case (which I never disagreed with)

I estimate, as I think several others do judging by various comments,
that the patches in their current state don't stand much chance of
passing muster upstream, and I have no desire to waste their time with a
premature submission. At the very least, some justification needs adding
to the 1 second sleep: either a datasheet reference, which I've so far
failed to find, or some empirical measurements that it's actually
required (which I'm in the process of gathering; I've verified *a* delay
is required but not the minimum, or whether it can be combined with the
post-load nanosleep), along with evidence it doesn't break any existing
platforms (I've found a datasheet reference to back that bit up, but
tests are good too).

Anyway, sorry I mis-interpreted the release your action applied to; I
look forward to seeing the SRU land.

In the meantime, rest assured a submission will be made upstream as soon
as I think I've got a faint hope of it passing!

Hello Dave, or anyone else affected,

Accepted bluez into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/bluez/5.55-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in bluez (Ubuntu Groovy):
status: New → Fix Committed
tags: added: verification-needed verification-needed-groovy
description: updated
Tony Molloy (tmolloy) wrote :

I've just used a fresh image of Mate 20.10 and used -proposed and done a full update and reboot. My bluetooth mouse, headphones and external keyboard are working fine now with this build so this bug has been fixed for me.

Thank you all for your efforts in getting it sorted

Kind regards
Tony

Paul Larson (pwlars) wrote :

I also tried this on my pi400 and confirmed I can see and connect to bluetooth devices after updating to bluez 5.55-0ubuntu1.1 from groovy-proposed

tags: added: verification-done verification-done-groovy
removed: verification-needed verification-needed-groovy
Łukasz Zemczak (sil2100) wrote :

Releasing this a day earlier to get the update shipped to users ASAP.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package bluez - 5.55-0ubuntu1.1

---------------
bluez (5.55-0ubuntu1.1) groovy; urgency=medium

  * Added patches from the Raspberry Pi Foundation
    - d/p/raspi-bcm43xx-load-firmware.patch
    - d/p/raspi-bcm43xx-3wire.patch
    - d/p/raspi-cypress-305-bdaddr.patch
  * These patches fix Bluetooth operation on the Pi 400 (LP: #1903048)

 -- Dave Jones <email address hidden> Thu, 05 Nov 2020 13:39:07 +0000

Changed in bluez (Ubuntu Groovy):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for bluez has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

hujq (hujq) wrote :

Hi,
This update seems to break Bluetooth on the Pi 4 8GB. /usr/bin/btuart returns "bcm43xx_init Initialization timed out". It's working again after reverting to version 5.55-0ubuntu1.

Sebastien Bacher (seb128) wrote :

Could you open a new report about the regression using 'ubuntu-bug bluez'?

Dave Jones (waveform) wrote :

@hujq is there anything unusual about your bluetooth setup? miniuart-bt overlay or anything like that? I'm unable to replicate the failure here on a Pi 4 8GB

hujq (hujq) wrote :

I opened a new bug #1905627. And I didn't change anything regarding Bluetooth from Ubuntu 20.10 default configuration. There is no miniuart-bt in /boot/firmware/config.txt.

michel jacobs (0d0a) wrote :

I installed version 5.55-0ubuntu2
did not make a difference.
in journalctl is this :
bluetoothd[40120]: src/main.c:parse_controller_config() Key file does not have group “Controller”

yonailo (juan-fco-rodriguez) wrote :

Bluetooth does not work any more on my Raspberry Pi 400 :

system : Ubuntu 20.10 (groovy)
kernel : 5.8.0-1011-raspi
bluez : 5.55-0ubuntu1.1

It used to work when I installed the Ubuntu 20.10 image, the groovy-updates has done something wrong :(

Daniel van Vugt (vanvugt) wrote :

yonailo, that must be a different issue. :(

Please open a new bug by running:

  ubuntu-bug bluez

Daniel van Vugt (vanvugt) wrote :

Or maybe use bug 1905627 from now on :)

Dave Jones (waveform) wrote :

There's a version of bluez with the patches that I've submitted upstream built in the following PPA against the hirsute branch (the patch changes are pretty minimal; basically amounts to removal of the arbitrary sleep(1) as during December's tests I couldn't find a single platform that actually required that, even amongst the Pi's we don't support like the zero):

https://launchpad.net/~waveform/+archive/ubuntu/bluez

I'm happy to convert that to a git-ubuntu branch if that's preferable. Once sponsored, we should consider SRU'ing to focal given that Pi400 support is now being SRU'd there generally.

Sebastien Bacher (seb128) wrote :

Thanks Dave, the updated patches have been uploaded now, removing the block-proposed

tags: removed: block-proposed
Changed in bluez (Ubuntu Hirsute):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers