fwupd fails on s390x

[Frank Heimes]

On a 22.04.1 s390y system I've noted (again, since I saw that already before) that fwupd fails on s390x:

Aug 17 21:12:03 s1lp10 fwupd[9629]: 21:12:03:0555 FuContext Failed to load SMBIOS: neither SMBIOS or DT found
Aug 17 21:12:03 s1lp10 fwupd[9629]: 21:12:03:0591 FuPluginTpm failed to load eventlog: Failed to open file "/sys/kernel/security/tpm0/binary_bios_measurements": No such file or directory
Aug 17 21:12:03 s1lp10 dbus-daemon[671]: [system] Successfully activated service 'org.freedesktop.fwupd'
Aug 17 21:12:03 s1lp10 systemd[1]: Started Firmware update daemon.
Aug 17 21:12:03 s1lp10 fwupd[9629]: 21:12:03:0609 FuPluginAcpiFacp failed to load /sys/firmware/acpi/tables/FACP: Failed to open file "/sys/firmware/acpi/tables/FACP": No such file or directory
Aug 17 21:12:03 s1lp10 fwupd[9629]: 21:12:03:0609 FuPluginLinuxSleep could not open /sys/power/mem_sleep: Error opening file /sys/power/mem_sleep: No such file or directory

This is not surprising, since the firmware update for s390x components (system firmware as well as adapter/card firmware) is done with the help of the hardware management console (HMC).

And an s390x system does not offer:
/sys/firmware/acpi
nor
/sys/kernel/security/tpm0
/sys/power

So I'm wondering if fwupd can be removed or disabled on s390x,
or at least if fwupd runs, that it checks for the underlying arch, and in case of s390x just stops.

[Mario Limonciello (superm1)]

Some of these plugins definitely shouldn't be enabled on s390x. I think that's an upstream bug in the detection on what plugins to enable at build time.

The crash of the client though; that's interesting. Do you have any additional logging? Could you get a stack trade from the crash file?

[Frank Heimes (fheimes)]

I just saved the /var/log content ...

[Mario Limonciello (superm1)]

> And an s390x system does not offer:

This should hopefully fix all those messages in the syslog: https://github.com/fwupd/fwupd/pull/4928. If you can please try it on your hardware and confirm, I would appreciate it.

> So I'm wondering if fwupd can be removed or disabled on s390x,
> or at least if fwupd runs, that it checks for the underlying arch, and in case of s390x just stops.

If it's not used for 7200s the daemon will stop on all architectures. It will start back up from D-Bus activation if/when a client tries to communicate with it.

[Frank Heimes (fheimes)]

Thx Mario!
A patched package that includes:
https://github.com/fwupd/fwupd/pull/4928/commits/27880c0d256617c4db01ab75267b7f79dc846d2f
is currently building here:
https://launchpad.net/~fheimes/+archive/ubuntu/lp1987067

(Might take a bit before I can test it, until the above system is available again ...)

[Frank Heimes (fheimes)]

I now installed the modified package to a systems, rotated the logs and leave it now running for at least a day.
Will check the logs again tomorrow ...

[Mario Limonciello (superm1)]

Can you please also try to manually run fwupdmgr refresh? If it crashes like you had above I'd like to see the stack trace.

[Frank Heimes (fheimes)]

So that's what I get out of gdb (which is not a lot):

Reading symbols from /usr/bin/fwupdmgr...
Reading symbols from /usr/lib/debug/.build-id/d6/46e7a195bae49c7dcf1cd43104b1b9390b8b0d.debug...

warning: Can't open file /usr/lib/s390x-linux-gnu/libgmodule-2.0.so.0.7200.1 during file-backed mapping note processing

warning: Can't open file /usr/lib/s390x-linux-gnu/libglib-2.0.so.0.7200.1 during file-backed mapping note processing

warning: Can't open file /usr/lib/s390x-linux-gnu/libgobject-2.0.so.0.7200.1 during file-backed mapping note processing

warning: Can't open file /usr/lib/s390x-linux-gnu/libgio-2.0.so.0.7200.1 during file-backed mapping note processing
[New LWP 7393]
[New LWP 7394]

warning: .dynamic section for "/lib/s390x-linux-gnu/libicuuc.so.70" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/lib/s390x-linux-gnu/libicudata.so.70" is not at the expected address (wrong library or version mismatch?)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/s390x-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/fwupdmgr refresh'.
Program terminated with signal SIGTRAP, Trace/breakpoint trap.
--Type <RET> for more, q to quit, c to continue without paging--c
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=<optimized out>, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x3ff932d9a80 (LWP 7393))]
(gdb) bt
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=<optimized out>, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x000003ff92798146 in __pthread_kill_internal (signo=<optimized out>, threadid=<optimized out>) at pthread_kill.c:78
#2 0x000003ff92748aa0 in __GI_raise (sig=<optimized out>) at ../sysdeps/posix/raise.c:26
#3 0x000003ff92ce12de in g_log_default_handler () from /lib/s390x-linux-gnu/libglib-2.0.so.0
#4 0x000003ff92ce2864 in g_logv () from /lib/s390x-linux-gnu/libglib-2.0.so.0
#5 0x000003ff92ce2b0a in g_log () from /lib/s390x-linux-gnu/libglib-2.0.so.0
#6 0x000003ff92f84d5e in ?? () from /lib/s390x-linux-gnu/libgio-2.0.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[Just notice that this happened under extreme high load conditions.]

And btw, what I see now at the test system, running the patched package is:

Aug 22 19:39:00 zlin systemd[1]: Starting Refresh fwupd metadata and update motd...
Aug 22 19:39:00 zlin dbus-daemon[706]: [system] Activating via systemd: service name='org.freedesktop.fwupd' unit='fwupd.service' requested by ':1.125' (uid=112 pid=40813 comm="/usr/bin/fwupdmgr refresh " label="unconfined")
Aug 22 19:39:00 zlin fwupd[40819]: 19:39:00:0868 FuContext Failed to load SMBIOS: neither SMBIOS or DT found
Aug 22 19:39:00 zlin dbus-daemon[706]: [system] Successfully activated service 'org.freedesktop.fwupd'
Aug 22 19:39:00 zlin fwupdmgr[40813]: Updating lvfs
Aug 22 19:39:00 zlin fwupdmgr[40813]: Downloading…: 0%
Aug 22 19:40:00 zlin systemd[1]: fwupd-refresh.service: Main process exited, code=exited, status=1/FAILURE
Aug 22 19:40:00 zlin systemd[1]: fwupd-refresh.s...

[Mario Limonciello (superm1)]

Ah I think that's actually https://bugs.launchpad.net/ubuntu/+source/fwupd/+bug/1969976 now. Basically systemd caused breakage and isn't a priority for them to fix it.

A W/A is landed in fwupd that is still pending an archive admin to release the SRU for both focal and jammy for it.

[Frank Heimes (fheimes)]

I see - yeah, seems to be it.
Guess I should reach out to YC and ask him if he could incl. PR 4928 for kinetic as well with the W/A for LP#1969976 ...

[Frank Heimes (fheimes)]

Well, the changelog says that:
  * Backport configuration option for systemd refresh unit user
  * Run fwupd-refresh under a dedicated fwupd-refresh user (LP: #1969976)
got already fixed in 1.7.7-1.
And since I've added PR 4928 on top of fwupd-1.8.3 it should be already in.
(since I don't see a dedicated quilt patch for this, I assume it's incl. in the upstream code with 1.8.3)
In other words I 'think' that should be already fixed in my PPA version (1.8.3-1ubuntu1), no?

[Mario Limonciello (superm1)]

Ah I missed that your PPA was 1.8.3 based. It should have picked that change up then and you should now have a dedicated user that runs fwupdmgr refresh. So that can't be the problem.

It's unfortunate the stack trace is so useless. Can you readily reproduce or it's just under load?

[Frank Heimes (fheimes)]

Sorry, I've been distracted with other things for a while.

But since your last comment the system with the patched fwupd package actually idled only.

So I still see mesgs like this in the logs even today:

Sep 13 04:35:09 hwe0004 systemd[1]: Starting Firmware update daemon...
Sep 13 04:35:09 hwe0004 fwupd[74725]: 04:35:09:0585 FuContext Failed to load SMBIOS: neither SMBIOS or DT found
Sep 13 04:35:09 hwe0004 dbus-daemon[706]: [system] Successfully activated service 'org.freedesktop.fwupd'
Sep 13 04:35:09 hwe0004 systemd[1]: Started Firmware update daemon.
Sep 13 04:35:09 hwe0004 fwupdmgr[74719]: Updating lvfs
Sep 13 04:35:09 hwe0004 fwupdmgr[74719]: Downloading…: 0%
Sep 13 04:36:09 hwe0004 systemd[1]: fwupd-refresh.service: Main process exited, code=exited, status=1/FAILURE
Sep 13 04:36:09 hwe0004 systemd[1]: fwupd-refresh.service: Failed with result 'exit-code'.
Sep 13 04:36:09 hwe0004 systemd[1]: Failed to start Refresh fwupd metadata and update motd.

The failed download is probably okay, since the test system has limited connectivity.

I see in /etc/passwd:
fwupd-refresh:x:112:120:fwupd-refresh user,,,:/run/systemd:/usr/sbin/nologin
so it seems to use the special 'fwupd-refresh' user.

And running the refresh by hand does not succeed (due to the limited connectivity):
$ sudo fwupdmgr refresh
Updating lvfs
Downloading… [ | ]failed to download file: Connection timeout after 60001 ms
but it also does not crash.
Folder "/var/crash/" is actually empty - no new crash happened in the last 2+ weeks.

[Mario Limonciello (superm1)]

OK thanks. It sounds like this is all expected behavior now then.

[Frank Heimes (fheimes)]

Yes, I agree - and thx for your help on this!

[Frank Heimes (fheimes)]

I'm closing this bug as invalid,
since the code in question and desired "Convert HSI into a meson tristate-feature"
meanwhile landed in the latest fwupd package 1.8.4-2 in kinetic.
(I did a 'pull-lp-source fwupd kinetic' and double-checked that the patch is in that I used above).