fwupd consuming 100% CPU

[Dustin Kirkland ]

[Impact]

 * Corrupt yaml data is causing consumers of appstream data (such as gnome-software or fwupd) to get stuck in a spin and consume CPU and battery.

[Test Case]

 * Introduce corrupt data into /var/lib/app-info/yaml/
 * Execute sudo /usr/lib/x86_64-linux-gnu/fwupd/fwupd -v
 * Check CPU usage calms down after initialization

[Regression Potential]

 * This patch has been backported from a newer stable release upstream.
 * Due to the nature of backporting, it doesn't directly apply to Ubuntu and will need some adjustments made.
 * Regressions would occur in gnome-software not displaying software or fwupd not running properly.

[Other Info]

Original bug report:

fwupd process has been consuming 100% of my CPU for several days now (IBM Thinkpad x250 running fully updated 16.04).

gdb backtrace below.

Thread 2 (Thread 0x7f3272cf5700 (LWP 4496)):
#0 0x00007f3279b79e8d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1 0x00007f327ae9331c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#2 0x00007f327ae9342c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#3 0x00007f327ae93469 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#4 0x00007f327aeb9b45 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#5 0x00007f3279e4f6fa in start_thread (arg=0x7f3272cf5700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3272cf5700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139854651283200, 418179942412000561, 0, 140731891236207,
                139854651283904, 0, -312637405101744847, -312613451844084431}, mask_was_saved = 0}}, priv = {pad = {
              0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#6 0x00007f3279b85b5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.

Thread 1 (Thread 0x7f327c926900 (LWP 4486)):
#0 0x00007f3279e542b7 in __GI___pthread_rwlock_rdlock (rwlock=0x7f3279e43920 <__libc_setlocale_lock>)
    at pthread_rwlock_rdlock.c:135
        result = 0
        wake = false
#1 0x00007f3279aaf0e9 in __dcigettext (domainname=0x7f327aedee63 "glib20",
    msgid1=0x7f327b4ca11a "Invalid compressed data", msgid2=0x0, plural=0, n=0, category=5) at dcigettext.c:527
        __p = <optimized out>
        domain = <optimized out>
        binding = <optimized out>
        categoryname = <optimized out>
        categoryvalue = <optimized out>
        dirname = <optimized out>
        xdomainname = <optimized out>
        single_locale = <optimized out>
        retval = <optimized out>
        retlen = 93892446493824
        saved_errno = 0
        search = {domainname = 0x7ffeb262bb00 "`\234\353\teU", category = 166588464,
          localename = 0x556509ededa8 "p\362\355\teU", counter = 166589040, domain = 0x23,
          translation = 0x5565fffffffb <error: Cannot access memory at address 0x5565fffffffb>,
          translation_length = 93892446493792, msgid = {appended = 0x7ffeb262bb08 "",
            ptr = 0x52fb0a5631f15a00 <error: Cannot access memory at address 0x52fb0a5631f15a00>}}
        foundp = 0x0
        localename = <optimized out>
        domainname_len = <optimized out>
#2 0x00007f327b44338c in ?? () from /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
No symbol table info available.
#3 0x00007f327b3e95f9 in ?? () from /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
No symbol table info available.
#4 0x00007f327b40d315 in g_input_stream_read () from /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
No symbol table info available.
#5 0x00007f327c1ae569 in ?? () from /usr/lib/x86_64-linux-gnu/libappstream-glib.so.8
No symbol table info available.
#6 0x00007f3278c51a15 in ?? () from /usr/lib/x86_64-linux-gnu/libyaml-0.so.2
No symbol table info available.
#7 0x00007f3278c51cde in yaml_parser_update_buffer () from /usr/lib/x86_64-linux-gnu/libyaml-0.so.2
No symbol table info available.
#8 0x00007f3278c56498 in yaml_parser_fetch_more_tokens () from /usr/lib/x86_64-linux-gnu/libyaml-0.so.2
No symbol table info available.
#9 0x00007f3278c5b32f in yaml_parser_parse () from /usr/lib/x86_64-linux-gnu/libyaml-0.so.2
No symbol table info available.
#10 0x00007f327c1ae39b in ?? () from /usr/lib/x86_64-linux-gnu/libappstream-glib.so.8
No symbol table info available.
#11 0x00007f327c1ae890 in as_yaml_from_file () from /usr/lib/x86_64-linux-gnu/libappstream-glib.so.8
No symbol table info available.
#12 0x00007f327c1a9231 in ?? () from /usr/lib/x86_64-linux-gnu/libappstream-glib.so.8
No symbol table info available.
#13 0x00007f327c1a9867 in ?? () from /usr/lib/x86_64-linux-gnu/libappstream-glib.so.8
No symbol table info available.
#14 0x00007f327c1a9ee1 in ?? () from /usr/lib/x86_64-linux-gnu/libappstream-glib.so.8
No symbol table info available.
#15 0x00007f327c1aa994 in as_store_load () from /usr/lib/x86_64-linux-gnu/libappstream-glib.so.8
No symbol table info available.
#16 0x00005565094a59c7 in main ()
No symbol table info available.

[Richard Hughes (richard-hughes)]

Interesting bug. This looks to be spinning in libyaml, but without debug symbols it's a little hard to piece together. Could you please install the relevant debug symbols (at least libappstream-glib, fwupd and libyaml) and get a new backtrace please. If you can reproduce using "fwupd --verbose" that might throw some light on what's going on. Thanks!

[Mario Limonciello (superm1)]

For running fwupd verbosely the exact set of steps should be:

#sudo killall fwupd
#sudo /usr/lib/x86_64-linux-gnu/fwupd/fwupd -v

[humberto (humberto.rodrigues)]

This bug is draining my notebook battery

[Mario Limonciello (superm1)]

There's a few duplicates of this so far, but not a clear set of steps to reproduce this yet.

Can someone affected please do the following:
1) First confirm that you are on the latest version of fwupd in Ubuntu. This is version 0.7.0-0ubuntu4.2 for xenial.
2) Install the relevant debug symbols as mentioned by @hughsie.
For 64 bit:
http://ddebs.ubuntu.com/ubuntu/pool/main/f/fwupd/fwupd-dbgsym_0.7.0-0ubuntu4.2_amd64.ddeb
http://ddebs.ubuntu.com/ubuntu/pool/main/f/fwupd/libdfu1-dbgsym_0.7.0-0ubuntu4.2_amd64.ddeb
http://ddebs.ubuntu.com/ubuntu/pool/main/f/fwupd/libfwupd1-dbgsym_0.7.0-0ubuntu4.2_amd64.ddeb
http://ddebs.ubuntu.com/ubuntu/pool/main/a/appstream-glib/libappstream-glib8-dbgsym_0.5.13-1_amd64.ddeb
# sudo apt install libyaml-0-2-dbg

3) # sudo killall fwupd
4) # sudo /usr/lib/x86_64-linux-gnu/fwupd/fwupd -v
5) In a separate terminal: # fwupdmgr get-devices
6) In a separate terminal: # fwupdmgr monitor
5) Reproduce the issue

Hopefully the terminal output from (4, 5, 6) will be useful. If it's not, then when this is happening attaching to the process using gdb (details available https://wiki.ubuntu.com/Backtrace#Already_running_programs) and provide the backtrace.

The backtrace in this bug didn't have debug symbols installed, so it wasn't useful to figure out what was actually happening.

[zipokan (qwuenaz)]

Also have this bug for several days. (Ubuntu 16.04 LTS) Everytime I use my laptop I have to kill all the fwupd processes.

I will try to provide some backtraces following Mario's indication asap.

[zipokan (qwuenaz)]

erkang@erkang-XPS-13-9343:~$ sudo /usr/lib/x86_64-linux-gnu/fwupd/fwupd -v
21:07:14 Verbose debugging enabled (on console 1)
21:07:14 adding plugin /usr/lib/x86_64-linux-gnu/fwupd-plugins-1/libfu_plugin_steelseries.so
21:07:14 adding plugin /usr/lib/x86_64-linux-gnu/fwupd-plugins-1/libfu_plugin_test.so
21:07:14 performing init() on test

erkang@erkang-XPS-13-9343:~$ fwupdmgr get-devices
Erreur lors de l'appel de StartServiceByName pour org.freedesktop.fwupd : GDBus.Error:org.freedesktop.DBus.Error.TimedOut: Failed to activate service 'org.freedesktop.fwupd': timed out

I can't find the command fwupdmgr monitor

and once again when I tap top:

 PID UTIL. PR NI VIRT RES SHR S %CPU %MEM TEMPS+ COM.
 3240 root 20 0 238288 11080 9804 R 99,7 0,1 4:20.47 fwupd
 3274 root 20 0 236256 10796 9520 R 99,3 0,1 3:50.68 fwupd

[Mario Limonciello (superm1)]

Thanks for sharing. You seem to have two fwupd processes running in your
test. Do you get the same output if you kill -9 them first where the last
thing output was "performing init() on test"?

If so, my suspicion would be some corrupt appstream data. The only thing
that happens between loading the plugins and loading the configuration file
is that the appstream store is initialized.

Can you check if both of these report successfully?

appstreamcli validate /var/cache/app-info/xmls/fwupd.xml

appstreamcli validate /usr/share/app-info/xmls/org.freedesktop.fwupd.xml

@hughsie, any other thoughts?

On Mon, Jul 18, 2016 at 2:20 PM ZHANG Erkang <email address hidden> wrote:

> erkang@erkang-XPS-13-9343:~$ sudo /usr/lib/x86_64-linux-gnu/fwupd/fwupd -v
> 21:07:14 Verbose debugging enabled (on console 1)
> 21:07:14 adding plugin
> /usr/lib/x86_64-linux-gnu/fwupd-plugins-1/libfu_plugin_steelseries.so
> 21:07:14 adding plugin
> /usr/lib/x86_64-linux-gnu/fwupd-plugins-1/libfu_plugin_test.so
> 21:07:14 performing init() on test
>
> erkang@erkang-XPS-13-9343:~$ fwupdmgr get-devices
> Erreur lors de l'appel de StartServiceByName pour org.freedesktop.fwupd :
> GDBus.Error:org.freedesktop.DBus.Error.TimedOut: Failed to activate service
> 'org.freedesktop.fwupd': timed out
>
>
> I can't find the command fwupdmgr monitor
>
> and once again when I tap top:
>
>
> PID UTIL. PR NI VIRT RES SHR S %CPU %MEM TEMPS+ COM.
> 3240 root 20 0 238288 11080 9804 R 99,7 0,1 4:20.47 fwupd
> 3274 root 20 0 236256 10796 9520 R 99,3 0,1 3:50.68 fwupd
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1591868
>
> Title:
> fwupd consuming 100% CPU
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/fwupd/+bug/1591868/+subscriptions
>

[zipokan (qwuenaz)]

I rerun the command

sudo /usr/lib/x86_64-linux-gnu/fwupd/fwupd -v

and I check these two report, their validations are successful.

[Mario Limonciello (superm1)]

Thanks. Considering that can you attach GDB to the process while it's
using 100% CPU and get a BT?
Information how to do this is here:
https://wiki.ubuntu.com/Backtrace#Already_running_programs

On Tue, Jul 19, 2016 at 12:35 PM ZHANG Erkang <email address hidden> wrote:

> I rerun the command
>
> sudo /usr/lib/x86_64-linux-gnu/fwupd/fwupd -v
>
> and I check these two report, their validations are successful.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1591868
>
> Title:
> fwupd consuming 100% CPU
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/fwupd/+bug/1591868/+subscriptions
>

[Richard Hughes (richard-hughes)]

Also, could you try getting the output of "appstream-util search notgoingtoexist --verbose" please. Thanks!

[zipokan (qwuenaz)]

Following Mario's steps, I was blocked in the fourth step. The program hanged out and it doesn't work to press ctrl + c:

  PID UTIL. PR NI VIRT RES SHR S %CPU %MEM TEMPS+ COM.
17208 root 20 0 236256 10632 9352 R 99,3 0,1 2:51.77 fwupd
17756 root 20 0 236256 10732 9456 R 99,0 0,1 1:35.75 fwupd
17740 root 20 0 236256 10816 9536 R 92,0 0,1 2:01.69 fwupd

root@erkang-XPS-13-9343:~# gdb 2>&1 | tee gdb-fwupd.txt
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) handle SIG33 pass nostop noprint
Signal Stop Print Pass to program Description
SIG33 No No Yes Real-time event 33
(gdb) set pagination 0
(gdb) attach 17208
Attaching to process 17208
[New LWP 17219]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__stpcpy_sse2 () at ../sysdeps/x86_64/multiarch/../strcpy.S:29
29 ../sysdeps/x86_64/multiarch/../strcpy.S: Aucun fichier ou dossier de ce type.
(gdb) continue
Continuing.
^C

^C

^C

^C

Following Richard's indication, the command simply blocked.. I didn't get any output:

root@erkang-XPS-13-9343:~# appstream-util search notgoingtoexist --verbose

[Richard Hughes (richard-hughes)]

Can you upload a tarball of whatever is in /var/cache/app-info/yamls pls.

[zipokan (qwuenaz)]

I didn't find yamls, this is the xml file of fwupd

<?xml version="1.0" encoding="UTF-8"?>
<components version="0.9" origin="ubuntu-xenial-updates-multiverse">
  <component type="firmware">
    <id>com.dell.uefi124c207d.firmware</id>
    <name>XPS 15 9550/Precision 5510 System Update</name>
    <summary>Firmware for the Dell XPS 15 9550/Precision 5510</summary>
    <developer_name>Dell Inc.</developer_name>
    <description><p>Updating the system firmware improves performance.</p></description>
    <project_license>proprietary</project_license>
    <url type="homepage">http://support.dell.com/</url>
    <releases>
      <release version="66048" timestamp="1460050441">
        <location>https://secure-lvfs.rhcloud.com/downloads/ec95a09f68afa49e02420e59bb8889af80a709a8-firmware_1.2.0.wu.cab</location>
        <checksum filename="ec95a09f68afa49e02420e59bb8889af80a709a8-firmware_1.2.0.wu.cab" target="container" type="sha1">4902dcbeed2b1d3cf1e9fd14a4b1aedd5ce00a4f</checksum>
        <checksum filename="firmware.bin" target="content" type="sha1">319779f0d574595fb464a4651418c456c8787708</checksum>
        <size type="installed">9692626</size>
        <size type="download">9417733</size>
      </release>
      <release version="65811" timestamp="1453782467">
        <location>https://secure-lvfs.rhcloud.com/downloads/90bb8877b5e8a4e4a5a0ce56af37dc4be7cf0ae8-firmware_9550_5510.cab</location>
        <checksum filename="90bb8877b5e8a4e4a5a0ce56af37dc4be7cf0ae8-firmware_9550_5510.cab" target="container" type="sha1">4e474c1040888188169b0bab4443e99b8c5c9bc3</checksum>
        <checksum filename="firmware.bin" target="content" type="sha1">d3c18d62256b909d25e3d9ccd96d78b0b59a745e</checksum>
        <size type="installed">9660656</size>
        <size type="download">9380627</size>
      </release>
      <release version="65807" timestamp="1450425495">
        <location>https://secure-lvfs.rhcloud.com/downloads/93d92eb7b94d02bbb58ae50653c2329d2f11aae2-firmware (1).cab</location>
        <checksum filename="93d92eb7b94d02bbb58ae50653c2329d2f11aae2-firmware (1).cab" target="container" type="sha1">12bc5fdea950a5cebbf2c2ab0e807c098622d710</checksum>
        <checksum filename="firmware.bin" target="content" type="sha1">36d4a02a7f7b6e6a055656f06023f3a70e1898dd</checksum>
        <size type="installed">9657051</size>
        <size type="download">9376057</size>
      </release>
    </releases>
    <provides>
      <firmware type="flashed">124c207d-5db8-4d95-bd31-34fd971b34f9</firmware>
    </provides>
  </component>
  <component type="firmware">
    <id>com.dell.uefi33773727.firmware</id>
    <name>XPS 13 9350 System Update</name>
    <summary>Firmware for the Dell XPS 13 9350</summary>
    <developer_name>Dell Inc.</developer_name>
    <description><p>Updating the system firmware improves performance.</p></description>
    <project_license>proprietary</project_license>
    <url type="homepage">http://support.dell.com/</url>
    <releases>
      <release version="66307" urgency="critical" timestamp="1456822533">
        <location>https://secure-lvfs.rhcloud.com/downloads/d3afa577b500700365dcdaaef8a573338f4f044b-firmware.cab</location>
        <checksum filename=...

[Mario Limonciello (superm1)]

That might not really be the useful bit - but rather the xapian data in
/var/cache/app-info. Can you just tar up the whole directory and attach it?

On Mon, Jul 25, 2016, 14:05 ZHANG Erkang <email address hidden> wrote:

> I didn't find yamls, this is the xml file of fwupd
>
>
> <?xml version="1.0" encoding="UTF-8"?>
> <components version="0.9" origin="ubuntu-xenial-updates-multiverse">
> <component type="firmware">
> <id>com.dell.uefi124c207d.firmware</id>
> <name>XPS 15 9550/Precision 5510 System Update</name>
> <summary>Firmware for the Dell XPS 15 9550/Precision 5510</summary>
> <developer_name>Dell Inc.</developer_name>
> <description><p>Updating the system firmware improves
> performance.</p></description>
> <project_license>proprietary</project_license>
> <url type="homepage">http://support.dell.com/</url>
> <releases>
> <release version="66048" timestamp="1460050441">
> <location>
> https://secure-lvfs.rhcloud.com/downloads/ec95a09f68afa49e02420e59bb8889af80a709a8-firmware_1.2.0.wu.cab
> </location>
> <checksum
> filename="ec95a09f68afa49e02420e59bb8889af80a709a8-firmware_1.2.0.wu.cab"
> target="container"
> type="sha1">4902dcbeed2b1d3cf1e9fd14a4b1aedd5ce00a4f</checksum>
> <checksum filename="firmware.bin" target="content"
> type="sha1">319779f0d574595fb464a4651418c456c8787708</checksum>
> <size type="installed">9692626</size>
> <size type="download">9417733</size>
> </release>
> <release version="65811" timestamp="1453782467">
> <location>
> https://secure-lvfs.rhcloud.com/downloads/90bb8877b5e8a4e4a5a0ce56af37dc4be7cf0ae8-firmware_9550_5510.cab
> </location>
> <checksum
> filename="90bb8877b5e8a4e4a5a0ce56af37dc4be7cf0ae8-firmware_9550_5510.cab"
> target="container"
> type="sha1">4e474c1040888188169b0bab4443e99b8c5c9bc3</checksum>
> <checksum filename="firmware.bin" target="content"
> type="sha1">d3c18d62256b909d25e3d9ccd96d78b0b59a745e</checksum>
> <size type="installed">9660656</size>
> <size type="download">9380627</size>
> </release>
> <release version="65807" timestamp="1450425495">
> <location>
> https://secure-lvfs.rhcloud.com/downloads/93d92eb7b94d02bbb58ae50653c2329d2f11aae2-firmware
> (1).cab</location>
> <checksum
> filename="93d92eb7b94d02bbb58ae50653c2329d2f11aae2-firmware (1).cab"
> target="container"
> type="sha1">12bc5fdea950a5cebbf2c2ab0e807c098622d710</checksum>
> <checksum filename="firmware.bin" target="content"
> type="sha1">36d4a02a7f7b6e6a055656f06023f3a70e1898dd</checksum>
> <size type="installed">9657051</size>
> <size type="download">9376057</size>
> </release>
> </releases>
> <provides>
> <firmware
> type="flashed">124c207d-5db8-4d95-bd31-34fd971b34f9</firmware>
> </provides>
> </component>
> <component type="firmware">
> <id>com.dell.uefi33773727.firmware</id>
> <name>XPS 13 9350 System Update</name>
> <summary>Firmware for the Dell XPS 13 9350</summary>
> <developer_name>Dell Inc.</developer_name>
> <description><p>Updating the system firmware improves
> performance.</p></desc...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mario Limonciello (superm1)]

So I think I've come up with a reliable way to reproduce this by placing a corrupted yaml file in /var/lib/app-info.

Here is the file I placed (notice the extra s at the start):

s---
File: DEP-11
MediaBaseUrl: http://appstream.ubuntu.com/media/main
Origin: ubuntu-xenial-backports-main
Priority: 40
Time: '2016-07-28T18:38:36.877230+00:00'
Version: '0.8'

By running fwupd with that in place, it hangs and pegs at 100% CPU.

The BT of course isn't too surprising where it's hung:

#0 0x00007ffff4249b6f in yaml_parser_parse () from /usr/lib/x86_64-linux-gnu/libyaml-0.so.2
#1 0x00007ffff779e5eb in as_node_yaml_process_layer (parser=parser@entry=0x7fffffffe030, parent=parent@entry=0x555555832c40) at as-yaml.c:225
#2 0x00007ffff779eae0 in as_yaml_from_file (file=file@entry=0x555555830560, cancellable=cancellable@entry=0x0, error=error@entry=0x7fffffffe340) at as-yaml.c:371
#3 0x00007ffff7799481 in as_store_load_yaml_file (error=0x7fffffffe340, cancellable=0x0, file=0x555555830560, store=0x555555795880) at as-store.c:1129
#4 as_store_from_file_internal (store=store@entry=0x555555795880, file=file@entry=0x555555830560, id_prefix=id_prefix@entry=0x7ffff77a29ae "system", cancellable=cancellable@entry=0x0,
    error=error@entry=0x7fffffffe340) at as-store.c:1371
#5 0x00007ffff7799ab7 in as_store_load_app_info_file (error=0x7fffffffe340, cancellable=0x0, path_xml=0x555555798460 "/var/lib/app-info/yaml/yaml.yml", id_prefix=0x7ffff77a29ae "system",
    store=0x555555795880) at as-store.c:2068
#6 as_store_load_app_info (store=store@entry=0x555555795880, id_prefix=id_prefix@entry=0x7ffff77a29ae "system", path=path@entry=0x555555798490 "/var/lib/app-info/yaml",
    flags=flags@entry=AS_STORE_LOAD_FLAG_APP_INFO_SYSTEM, cancellable=cancellable@entry=0x0, error=error@entry=0x7fffffffe4d8) at as-store.c:2118
#7 0x00007ffff779a131 in as_store_search_app_info (store=store@entry=0x555555795880, flags=flags@entry=AS_STORE_LOAD_FLAG_APP_INFO_SYSTEM, id_prefix=id_prefix@entry=0x7ffff77a29ae "system",
    path=path@entry=0x555555798430 "/var/lib/app-info", cancellable=cancellable@entry=0x0, error=error@entry=0x7fffffffe4d8) at as-store.c:2333
#8 0x00007ffff779abe4 in as_store_search_per_system (error=<optimized out>, cancellable=0x0, flags=<optimized out>, store=<optimized out>) at as-store.c:2535
#9 as_store_load (store=<optimized out>, flags=<optimized out>, cancellable=0x0, error=<optimized out>) at as-store.c:2683
#10 0x000055555555d9c7 in main ()

[Richard Hughes (richard-hughes)]

There's a workaround here: https://github.com/hughsie/appstream-glib/commit/2ec31bac7bd77e7b39c502276f4acec21ed9d576

Although, whatever wrote that invalid YAML file needs to be fixed, or gnome-software is going to show very few applications...

[Mario Limonciello (superm1)]

Just so it's clear - I haven't randomly reproduced this like all the people in this thread have, I manually inserted an error into a file and ran into identical problems as reported in this thread.

I don't know tons about how the YAML that ends up in /var/lib/app-info/yaml is generated, but I thought it was all based upon packages in the Ubuntu archives.

Since it's not a 100% reproduction rate, my conjecture would be a mirroring problem or download problem that introduced some corruption along the way. It would be great for someone who is inherently affected without doing anything wrong to share their contents of /var/lib/app-info/yaml while affected.

[Anders Kaseorg (andersk)]

The infinite loop is not in libyaml (yaml_parser_parse), but rather in its caller in appstream-glib (as_node_yaml_process_layer), which is ignoring the error code returned by yaml_parser_parse and blindly going around the loop again. This patch fixes it.

[Anders Kaseorg (andersk)]

Also submitted upstream: https://github.com/hughsie/appstream-glib/pull/129

[Richard Hughes (richard-hughes)]

I've further extended Anders patch to return a helpful GError: https://github.com/hughsie/appstream-glib/commit/64029dfe02b85afe611a279380a84fca51208d4c

I've rolled up these three patches into one and backported to the stable branch in: https://github.com/hughsie/appstream-glib/commit/60033e61394562d706e703ad33e0d7e6583d83ce

I guess pulling that last commit into 16.04 makes sense, if nothing else so it pinpoints where exactly and why the file is corrupt.

[Mario Limonciello (superm1)]

I was running some tests with that backport applied to 0.5.13 on Ubuntu as well as running the master branch and was getting segfaults now with that same corrupt file I was working with.

Starting program: /usr/bin/appstream-util validate /home/test/yaml.yml
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
/home/test/yaml.yml: [New Thread 0x7ffff10cc700 (LWP 29156)]
[New Thread 0x7ffff08cb700 (LWP 29157)]

Thread 1 "appstream-util" received signal SIGSEGV, Segmentation fault.
__GI___libc_free (mem=0x3) at malloc.c:2952
2952 malloc.c: No such file or directory.
(gdb) bt full
#0 __GI___libc_free (mem=0x3) at malloc.c:2952
        ar_ptr = <optimized out>
        p = <optimized out>
        hook = 0x0
#1 0x00007ffff7bb501b in as_node_destroy_node_cb (node=<optimized out>, user_data=<optimized out>) at as-node.c:140
        data = 0x555555796b40
#2 0x00007ffff6dff5f5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#3 0x00007ffff6dff5a9 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#4 0x00007ffff6e00281 in g_node_traverse () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#5 0x00007ffff7bb55d2 in as_node_unref (node=0x7fffe4009ad0) at as-node.c:157
No locals.
#6 0x00007ffff7bc2f91 in glib_autoptr_cleanup_AsNode (_ptr=<synthetic pointer>) at as-node.h:197
No locals.
#7 as_yaml_from_file (file=file@entry=0x555555798500, cancellable=cancellable@entry=0x0, error=error@entry=0x7fffffffdc70) at as-yaml.c:474
        node = <optimized out>
        content_type = <optimized out>
        parser = {error = YAML_NO_ERROR, problem = 0x0, problem_offset = 0, problem_value = 0, problem_mark = {index = 0, line = 0, column = 0}, context = 0x0, context_mark = {index = 0, line = 0,
            column = 0}, read_handler = 0x0, read_handler_data = 0x0, input = {string = {start = 0x0, end = 0x0, current = 0x0}, file = 0x0}, eof = 0, buffer = {start = 0x0, end = 0x0, pointer = 0x0,
            last = 0x0}, unread = 0, raw_buffer = {start = 0x0, end = 0x0, pointer = 0x0, last = 0x0}, encoding = YAML_ANY_ENCODING, offset = 0, mark = {index = 0, line = 0, column = 0},
          stream_start_produced = 0, stream_end_produced = 0, flow_level = 0, tokens = {start = 0x0, end = 0x0, head = 0x0, tail = 0x0}, tokens_parsed = 0, token_available = 0, indents = {start = 0x0,
            end = 0x0, top = 0x0}, indent = 0, simple_key_allowed = 0, simple_keys = {start = 0x0, end = 0x0, top = 0x0}, states = {start = 0x0, end = 0x0, top = 0x0},
          state = YAML_PARSE_STREAM_START_STATE, marks = {start = 0x0, end = 0x0, top = 0x0}, tag_directives = {start = 0x0, end = 0x0, top = 0x0}, aliases = {start = 0x0, end = 0x0, top = 0x0},
          document = 0x0}
        parser_cleanup = <optimized out>
        data = 0x0
        conv = <optimized out>
        info = 0x7fffe4009b90
        file_stream = <optimized out>
        stream_data = <optimized out>
#8 0x00007ffff7bbd481 in as_store_load_yaml_file (error=0x7fffffffdc70, cancellable=0x0, file=0x555555798500, store=0x55555579a1f0) at as-store.c:1...

[Richard Hughes (richard-hughes)]

Mario, can you upload /home/test/yaml.yml somewhere please.

[Mario Limonciello (superm1)]

I don't have it handy over the weekend but it was just the same test file I
was using above put into /var with the extra s.

On Sat, Aug 20, 2016, 02:20 Richard Hughes <email address hidden> wrote:

> Mario, can you upload /home/test/yaml.yml somewhere please.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1591868
>
> Title:
> fwupd consuming 100% CPU
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/appstream-glib/+bug/1591868/+subscriptions
>

[Richard Hughes (richard-hughes)]

Agreed; reproduced -- grab this on top of the last patch:

commit 481055cd02c5244bede06ca4a06ab55602d01505
Author: Richard Hughes <email address hidden>
Date: Sun Aug 21 08:59:45 2016 +0100

    trivial: Use the correct autoptr cleanup on yaml error

:100644 100644 f65f75b... 48e51f0... M libappstream-glib/as-yaml.c

[Mario Limonciello (superm1)]

Seems still not enough, that one is loading from data, I was having the crash in loading from a file. I've added this which seems to cover it.
https://github.com/hughsie/appstream-glib/pull/133

[Martin Pitt (pitti)]

It seems the patch here is not sufficient, so unsubscribing sponsors for now. Please resubscribe once there is an uploadable patch.

[Mario Limonciello (superm1)]

For Yakkety: this is fixed in appstream-glib 0.6.2.
For xenial: these are the 3 commits needed to fix the problem:
https://github.com/hughsie/appstream-glib/commit/60033e61394562d706e703ad33e0d7e6583d83ce
https://github.com/hughsie/appstream-glib/commit/481055cd02c5244bede06ca4a06ab55602d01505
https://github.com/hughsie/appstream-glib/commit/57625db18b2cfe6e9d8a39a5d0cf1ed35af50e02

[Launchpad Janitor (janitor)]

This bug was fixed in the package appstream-glib - 0.6.2-0ubuntu1

---------------
appstream-glib (0.6.2-0ubuntu1) yakkety; urgency=medium

  * New upstream version.
  * Fixes daemon hangs with corrupt data (LP: #1591868)
  * Drop gcc shift warnings patch.

 -- Mario Limonciello <email address hidden> Tue, 30 Aug 2016 13:42:09 -0500

[Chris J Arges (arges)]

Hello Dustin, or anyone else affected,

Accepted appstream-glib into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/appstream-glib/0.5.13-1ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Mario Limonciello (superm1)]

I confirmed the fix is doing as intended on xenial. fwupd doesn't peg the CPU or crash from the bad data. It just exits.

$ sudo /usr/lib/x86_64-linux-gnu/fwupd/fwupd -v
16:23:19 Verbose debugging enabled (on console 1)
16:23:19 adding plugin /usr/lib/x86_64-linux-gnu/fwupd-plugins-1/libfu_plugin_test.so
16:23:19 performing init() on test
16:23:19 adding plugin /usr/lib/x86_64-linux-gnu/fwupd-plugins-1/libfu_plugin_steelseries.so
FuMain: failed to load AppStream data: scanner error: mapping values are not allowed in this context at ln:2 col:5
16:23:19 FuMain: failed to load AppStream data: scanner error: mapping values are not allowed in this context at ln:2 col:5

[Steve Lorimer (steve-lorimer)]

How can I get this fix in Xenial?

[Mario Limonciello (superm1)]

It is currently in the proposed repository. If no problems are reported as
regressions in the next few days it will transition into the updates
repository.

If you would like to enable proposed to test it yourself you can follow the
steps here:
https://wiki.ubuntu.com/Testing/EnableProposed

Please do report back if it resolves the problem for you. If the problem
persists after applying the update you may need to file an additional bug
to diagnose further.

On Mon, Sep 5, 2016, 04:01 Steve Lorimer <email address hidden> wrote:

> How can I get this fix in Xenial?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1591868
>
> Title:
> fwupd consuming 100% CPU
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/appstream-glib/+bug/1591868/+subscriptions
>

[Launchpad Janitor (janitor)]

This bug was fixed in the package appstream-glib - 0.5.13-1ubuntu3

---------------
appstream-glib (0.5.13-1ubuntu3) xenial; urgency=medium

  * Catch errors from libyaml and display proper warnings
    rather than spin on them. (LP: #1591868)

 -- Mario Limonciello <email address hidden> Tue, 30 Aug 2016 13:51:56 -0500

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for appstream-glib has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Marcelo Fernandez (fernandezm)]

Hi, I'm fully updated and still having this issue. Do you know what could be wrong?

Thankyou

marcelo@marcelo-notebook:~$ apt show libappstream-glib8
Package: libappstream-glib8
Version: 0.5.13-1ubuntu4
Priority: optional
Section: libs
Source: appstream-glib
Origin: Ubuntu
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Matthias Klumpp <email address hidden>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 312 kB
Depends: libarchive13, libc6 (>= 2.14), libgcab-1.0-0 (>= 0.4), libgdk-pixbuf2.0-0 (>= 2.25.2), libglib2.0-0 (>= 2.46), libsoup2.4-1 (>= 2.52), libuuid1 (>= 2.16), libyaml-0-2
Homepage: http://people.freedesktop.org/~hughsient/appstream-glib/
Task: ubuntu-desktop, ubuntu-usb, kubuntu-desktop, edubuntu-desktop, edubuntu-usb, xubuntu-core, xubuntu-desktop, mythbuntu-desktop, lubuntu-core, ubuntustudio-desktop-core, ubuntustudio-desktop, ubuntu-gnome-desktop, ubuntukylin-desktop, ubuntu-mate-core, ubuntu-mate-desktop, ubuntu-mate-cloudtop
Supported: 5y
Download-Size: 101 kB
APT-Manual-Installed: no
APT-Sources: http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
Description: GNOME library to access AppStream services
 AppStream is a cross-distribution effort for creating and sharing
 metadata about software components available in the package repositories
 of a distribution.
 .
 This package contains a GObject-based library to read and write AppStream
 XML metadata and retrieve additional data such as screenshots.
 .
 As opposed to libappstream, this library does not use an on-disk cache but
 reads data directly from the XML or DEP-11 YAML files.

[Jairo Andres Moreno (nebulatronica146)]

Hi all,
I updated too my files. But, I'm still experiencing high CPU consumption by fwupd and gnome-software.

Package: libappstream-glib8
Version: 0.5.13-1ubuntu4
Priority: optional
Section: libs
Source: appstream-glib
Origin: Ubuntu
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Matthias Klumpp <email address hidden>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 312 kB
Depends: libarchive13, libc6 (>= 2.14), libgcab-1.0-0 (>= 0.4), libgdk-pixbuf2.0-0 (>= 2.25.2), libglib2.0-0 (>= 2.46), libsoup2.4-1 (>= 2.52), libuuid1 (>= 2.16), libyaml-0-2
Homepage: http://people.freedesktop.org/~hughsient/appstream-glib/
Task: ubuntu-desktop, ubuntu-usb, kubuntu-desktop, edubuntu-desktop, edubuntu-usb, xubuntu-core, xubuntu-desktop, mythbuntu-desktop, lubuntu-core, ubuntustudio-desktop-core, ubuntustudio-desktop, ubuntu-gnome-desktop, ubuntukylin-desktop, ubuntu-mate-core, ubuntu-mate-desktop, ubuntu-mate-cloudtop
Supported: 5y
Download-Size: 101 kB
APT-Manual-Installed: no
APT-Sources: http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
Description: GNOME library to access AppStream services
 AppStream is a cross-distribution effort for creating and sharing
 metadata about software components available in the package repositories
 of a distribution.
 .
 This package contains a GObject-based library to read and write AppStream
 XML metadata and retrieve additional data such as screenshots.
 .
 As opposed to libappstream, this library does not use an on-disk cache but
 reads data directly from the XML or DEP-11 YAML files.

[Mario Limonciello (superm1)]

If you're experiencing this bug with a fully updated system, please report a new bug report. In the new bug report please include these details:

https://bugs.launchpad.net/ubuntu/+source/appstream-glib/+bug/1591868/comments/4

[Jørgen Dokken (dokken92)]

I had this similar problem until I uninstalled gnome-software, I have no idea why that did the trick.

[Rémy Montangé (w4u)]

To correct the issue :

killall gnome-software
sudo killall fwupd

sudo apt-get update
sudo apt-get dist-upgrade

sudo apt-get autoremove gnome-software
sudo apt-get install gnome-software

[Kasun Siyambalapitiya (kasun-siyambalapitiya)]

Even though the bug is marked as fixed, I am still having the issue though the os (Ubuntu 16.04)is up to date

[Mario Limonciello (superm1)]

The existing bug is fixed. Please file a new bug.

On Wed, Jul 26, 2017, 22:59 Kasun Siyambalapitiya <
<email address hidden>> wrote:

> Even though the bug is marked as fixed, I am still having the issue
> though the os (Ubuntu 16.04)is up to date
>
> ** Attachment added: "fwupd is consuming 100%"
>
> https://bugs.launchpad.net/ubuntu/+source/appstream-glib/+bug/1591868/+attachment/4922155/+files/fwupd.png
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1591868
>
> Title:
> fwupd consuming 100% CPU
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/appstream-glib/+bug/1591868/+subscriptions
>

[sul aga (sul-aga)]

can someone please explain what is the solution. I am having the same issue. I have followed these commands as suggested by "Rémy Montangé" then restarted my system but after restarting the same thing is happening again fwupd is using 100% of my cpu

killall gnome-software
sudo killall fwupd

sudo apt-get update
sudo apt-get dist-upgrade

sudo apt-get autoremove gnome-software
sudo apt-get install gnome-software

thank you

[Kasun Siyambalapitiya (kasun-siyambalapitiya)]

Hi Sul aga, as Mario suggested I have created a new bug [https://bugs.launchpad.net/ubuntu/+source/appstream-glib/+bug/1707110] please mark it affects to you too.

Thank you