apt-get update returns "AppStream cache update completed, but some metadata was ignored due to errors." periodically

Bug #1644498 reported by Thomas A. F. Thorne on 2016-11-24
334
This bug affects 69 people
Affects Status Importance Assigned to Milestone
appstream (Ubuntu)
Medium
Matthias Klumpp
Xenial
Medium
Matthias Klumpp

Bug Description

[Impact]

 * The recent update of the Ubuntu asgen instance yielded it to generate modern metadata. While we could tell asgen to generate old-style metadata, fixing appstream is the better idea, since we already applied similar patches to appstream-glib (and having this package hold back updates is a bad idea). Currently, the unknown metadata is considered to be broken by appstream and it displays a warning about that issue during an `apt update`

 * The attached patch includes a second change to make AppStream more robust against broken YAML data, ensuring LP #1579712 can not be triggered on purpose or due to broken disks creating damaged YAML structure.

[Test Case 1]

 * apt update
 * Check for errors about metadata being ignored

[Test Case 2]

 * appstreamcli refresh --force
 * Check for errors about metadata being ignored
 * Check if we ignored data that should be present
   (search for it via `appstreamcli search <term>`)
 * Run `appstreamcli status` and see if we still have information on the same amount of components as prior to the update.

[Regression Potential]

 * Medium; The works case would be KDE Discover and GNOME Software loosing information on available software, but this risk is very low and the patch to fix this bug is very simple and has almost no regression potential. The YAML parser fix patch is a bit more complex, but should have no sideeffects.

[Original report]

Periodically I get emails from a machine with the subject:

Cron <root@Jenkins> if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi

And a body of:

AppStream cache update completed, but some metadata was ignored due to errors.

I have a machine monitored by Munin. One of the plugins used on that machine is /etc/munin/plugins/apt which periodically counts the number of packets that can be upgraded. This is achieved by a crontab entry installed from the package which looks like this:

#
# cron-jobs for munin-node
#

MAILTO=root

# If the APT plugin is enabled, update packages databases approx. once
# an hour (12 invokations an hour, 1 in 12 chance that the update will
# happen), but ensure that there will never be more than two hour (7200
# seconds) interval between updates..
*/5 * * * * root if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi

For many months everything has been running smoothly. On the 23rd November 2016 I started getting the "AppStream cache update completed, but some metadata was ignored due to errors." messages at a rate of about ten a day.

I did a quick Google search which turned up a handful of bugs from earlier this year, such as
Bug1575248. That is marked as Fixed Released though, so I am raising this new bug.

If there are any further diagnostic details I can provide, please let me know.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: appstream 0.9.4-1ubuntu1
ProcVersionSignature: Ubuntu 4.4.0-47.68-generic 4.4.24
Uname: Linux 4.4.0-47-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
Date: Thu Nov 24 09:57:18 2016
InstallationDate: Installed on 2013-05-08 (1295 days ago)
InstallationMedia: Ubuntu 13.10 "Saucy Salamander" - Alpha amd64 (20130508)
JournalErrors:
 -- Logs begin at Wed 2016-11-23 19:25:01 GMT, end at Thu 2016-11-24 09:57:24 GMT. --
 Nov 24 02:22:17 hostname systemd-tmpfiles[27330]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
ProcEnviron:
 TERM=xterm-256color
 SHELL=/bin/bash
 PATH=(custom, no user)
 LANG=en_GB.UTF-8
 LANGUAGE=en_GB:en
SourcePackage: appstream
UpgradeStatus: Upgraded to xenial on 2016-08-24 (91 days ago)

Thomas A. F. Thorne (tafthorne) wrote :
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in appstream (Ubuntu):
status: New → Confirmed
Jay Anderson (tubadogs) wrote :

Originally posted to bug 1575248.

On 23 Nov 2016:
Noticed that I was having this issue ~20 Nov 2016. This is the most recent mention that Google found. I will provide details if there is interest.

And on 24 Nov 2016:
Yesterday was the second time that I had noticed this issue, so I started searching and ran across this. Unsure when it actually started and how many times it has occurred.

Matthias Klumpp (ximion) wrote :

Can you please run
`sudo appstreamcli refresh --verbose --force` and attach the output here?

Thomas A. F. Thorne (tafthorne) wrote :
Download full text (21.1 KiB)

Thank you. I could not figure out which combination of commands to send appsteamcli. I had tried status and validate without success. Here is the requested output:

manager@Jenkins:~$ sudo appstreamcli refresh --verbose --force
[sudo] password for manager:
** (appstreamcli:32754): DEBUG: Refreshing AppStream cache
** (appstreamcli:32754): DEBUG: Reading: /usr/share/app-info/xmls/org.freedesktop.fwupd.xml
** (appstreamcli:32754): DEBUG: Reading: /var/lib/app-info/yaml/gb.archive.ubuntu.com_ubuntu_dists_xenial_main_dep11_Components-amd64.yml.gz
** (appstreamcli:32754): DEBUG: Reading: /var/lib/app-info/yaml/gb.archive.ubuntu.com_ubuntu_dists_xenial_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:32754): DEBUG: Reading: /var/lib/app-info/yaml/gb.archive.ubuntu.com_ubuntu_dists_xenial_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:32754): DEBUG: Reading: /var/lib/app-info/yaml/security.ubuntu.com_ubuntu_dists_xenial-security_main_dep11_Components-amd64.yml.gz
** (appstreamcli:32754): DEBUG: Reading: /var/lib/app-info/yaml/security.ubuntu.com_ubuntu_dists_xenial-security_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:32754): DEBUG: Reading: /var/lib/app-info/yaml/security.ubuntu.com_ubuntu_dists_xenial-security_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:32754): DEBUG: Reading: /var/lib/app-info/yaml/security.ubuntu.com_ubuntu_dists_xenial-security_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:32754): DEBUG: Reading: /var/lib/app-info/yaml/gb.archive.ubuntu.com_ubuntu_dists_xenial_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:32754): DEBUG: Reading: /var/lib/app-info/yaml/gb.archive.ubuntu.com_ubuntu_dists_xenial-updates_main_dep11_Components-amd64.yml.gz
** (appstreamcli:32754): DEBUG: Reading: /var/lib/app-info/yaml/gb.archive.ubuntu.com_ubuntu_dists_xenial-updates_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:32754): DEBUG: Reading: /var/lib/app-info/yaml/gb.archive.ubuntu.com_ubuntu_dists_xenial-updates_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:32754): DEBUG: Reading: /var/lib/app-info/yaml/gb.archive.ubuntu.com_ubuntu_dists_xenial-updates_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:32754): DEBUG: Reading: /var/cache/app-info/xmls/fwupd.xml
** (appstreamcli:32754): DEBUG: Detected colliding ids: plan.desktop was already added with the same priority.
** (appstreamcli:32754): DEBUG: Detected colliding ids: flcheckers.desktop was already added with the same priority.
** (appstreamcli:32754): DEBUG: Detected colliding ids: flblocks.desktop was already added with the same priority.
** (appstreamcli:32754): DEBUG: Detected colliding ids: flsudoku.desktop was already added with the same priority.
** (appstreamcli:32754): DEBUG: Replaced 'libreoffice-startcenter.desktop' with data of higher priority.
** (appstreamcli:32754): DEBUG: Replaced 'libreoffice-math.desktop' with data of higher priority.
** (appstreamcli:32754): DEBUG: Replaced 'eog.desktop' with data of higher priority.
** (appstreamcli:32754): DEBUG: Replaced 'display-im6.desktop' with data of higher priority.
** (appstreamcli:32754): DEBUG: Replaced 'libreoffice-draw.desktop' with ...

Matthias Klumpp (ximion) wrote :

Okay, the complaint appears because of these warnings, and those warnings exist because AS detected a component without a name, summary or ID, which should never happen.
We'd need to figure out why those components exist.

Matthias Klumpp (ximion) wrote :

I think I found the issue, I'll prepare a patch.

Changed in appstream (Ubuntu):
assignee: nobody → Matthias Klumpp (ximion)
status: Confirmed → In Progress
Jay Anderson (tubadogs) wrote :
Download full text (22.0 KiB)

Here's my output if it will help.

root@HP-Pavilion:~# appstreamcli refresh --force --verbose
** (appstreamcli:18192): DEBUG: Refreshing AppStream cache
** (appstreamcli:18192): DEBUG: Reading: /usr/share/app-info/xmls/org.freedesktop.fwupd.xml
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/security.ubuntu.com_ubuntu_dists_xenial-security_main_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/security.ubuntu.com_ubuntu_dists_xenial-security_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/security.ubuntu.com_ubuntu_dists_xenial-security_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/security.ubuntu.com_ubuntu_dists_xenial-security_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/us.archive.ubuntu.com_ubuntu_dists_xenial_main_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/us.archive.ubuntu.com_ubuntu_dists_xenial_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/us.archive.ubuntu.com_ubuntu_dists_xenial_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/us.archive.ubuntu.com_ubuntu_dists_xenial_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/us.archive.ubuntu.com_ubuntu_dists_xenial-backports_main_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/us.archive.ubuntu.com_ubuntu_dists_xenial-backports_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/us.archive.ubuntu.com_ubuntu_dists_xenial-backports_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/us.archive.ubuntu.com_ubuntu_dists_xenial-backports_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/us.archive.ubuntu.com_ubuntu_dists_xenial-updates_main_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/us.archive.ubuntu.com_ubuntu_dists_xenial-updates_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/us.archive.ubuntu.com_ubuntu_dists_xenial-updates_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/lib/app-info/yaml/us.archive.ubuntu.com_ubuntu_dists_xenial-updates_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:18192): DEBUG: Reading: /var/cache/app-info/xmls/fwupd.xml
** (appstreamcli:18192): DEBUG: Detected colliding ids: eog.desktop was already added with a higher priority.
** (appstreamcli:18192): DEBUG: Detected colliding ids: libreoffice-base.desktop was already added with a higher priority.
** (appstreamcli:18192): DEBUG: Detected colliding ids: org.gnome.FileRoller.desktop was already added with a higher priority.
** (appstreamcli:18192): DEBUG: Detected colliding ids: libreoffice-impress.desktop was already ad...

Changed in appstream (Ubuntu):
status: In Progress → Fix Committed
status: Fix Committed → In Progress
Thomas A. F. Thorne (tafthorne) wrote :

> I think I found the issue, I'll prepare a patch.

That is great news. Is there a work around I could try that would silence the emails for a while? Let me know if you would like me to try out the updated binary.

Matthias Klumpp (ximion) wrote :

A workaround would be to update to the appstream/libappstream package in xenial-backports or to just sent the output of stderr to /dev/null to get rid of the message.

Installing the fixed file is not really a work around but the error piping would work. I'll live with the noise for now and try to get the updated package installed once it reaches xenial-backports.

vigilian (vigilian) wrote :

I don't have any mail monitoring
Atteint:1 http://ppa.launchpad.net/x2go/stable/ubuntu xenial InRelease
Atteint:2 http://be.archive.ubuntu.com/ubuntu xenial InRelease
Réception de:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Atteint:4 http://download.virtualbox.org/virtualbox/debian xenial InRelease
Réception de:5 http://be.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Réception de:6 http://be.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Atteint:7 https://apt.dockerproject.org/repo ubuntu-xenial InRelease
Réception de:8 http://security.ubuntu.com/ubuntu xenial-security/main amd64 DEP-11 Metadata [66,9 kB]
Réception de:9 http://security.ubuntu.com/ubuntu xenial-security/main DEP-11 64x64 Icons [43,3 kB]
Réception de:10 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 DEP-11 Metadata [19,4 kB]
Réception de:11 http://security.ubuntu.com/ubuntu xenial-security/universe DEP-11 64x64 Icons [25,6 kB]
Réception de:12 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 DEP-11 Metadata [212 B]
Réception de:13 http://be.archive.ubuntu.com/ubuntu xenial-updates/main amd64 DEP-11 Metadata [293 kB]
Réception de:14 http://be.archive.ubuntu.com/ubuntu xenial-updates/main DEP-11 64x64 Icons [182 kB]
Réception de:15 http://be.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 DEP-11 Metadata [119 kB]
Réception de:16 http://be.archive.ubuntu.com/ubuntu xenial-updates/universe DEP-11 64x64 Icons [137 kB]
Réception de:17 http://be.archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 DEP-11 Metadata [2.520 B]
Réception de:18 http://be.archive.ubuntu.com/ubuntu xenial-backports/main amd64 DEP-11 Metadata [208 B]
Réception de:19 http://be.archive.ubuntu.com/ubuntu xenial-backports/universe amd64 DEP-11 Metadata [212 B]
Réception de:20 http://be.archive.ubuntu.com/ubuntu xenial-backports/multiverse amd64 DEP-11 Metadata [212 B]
Réception de:21 https://download.01.org/gfx/ubuntu/16.04/main xenial InRelease [3.651 B]
1.200 ko réceptionnés en 0s (1.221 ko/s)
AppStream cache update completed, but some metadata was ignored due to errors.
Lecture des listes de paquets... Fait
W: https://download.01.org/gfx/ubuntu/16.04/main/dists/xenial/InRelease: Signature by key 09D6EF97BFB38E916EF060E756A3DEF863961D39 uses weak digest algorithm (SHA1)

So I don't know your patch is already deployed but seems not

Matthias Klumpp (ximion) wrote :

Patch for this bug is attached.
I included a change unrelated to this issue in the patch too, to make AppStream more robust against on-disk data corruption (or someone trying to DOS it by intentionally inserting broken data).

03_yaml-parser-errors.patch: Make the YAML parser deal with errors properly (reject the broken document) instead of running into an infinite loop if things go bad.

04_modern-metadata.patch: The fix for this bug. Unfortunately I couldn't just backport the upstream changes, since they contain bigger behavior and ABI changes, so this patch is only based on the upstream bits and needs proper testing.
It adds support for modern AppStream metadata component types to the parser, making this warning disappear and having AS process all data again.

Matthias Klumpp (ximion) on 2016-11-27
description: updated
Changed in appstream (Ubuntu):
importance: Undecided → Medium
Nikita Rotsky (rotzke) wrote :

Thanks, Matthias, the patch has fixed that bug. I was able to upgrade "appstream" package to 0.10.1 without any probs afterward. Cheers!

The attachment "appstream_0.9.4-1ubuntu2_LP1644498_v1.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Matthias Klumpp (ximion) on 2016-11-27
Changed in appstream (Ubuntu):
status: In Progress → Fix Committed
Daniel Holbach (dholbach) wrote :

Is this just an issue in Xenial?

Matthias Klumpp (ximion) wrote :

Yes, Yakkety and Zesty are not affected.

Matthias Klumpp (ximion) wrote :

(The attached patch solves an introspection ABI mismatch that I overlooked initially)

Iain Lane (laney) wrote :

Uploaded, thank you!

Changed in appstream (Ubuntu Xenial):
status: New → In Progress
Changed in appstream (Ubuntu):
status: Fix Committed → Fix Released
Changed in appstream (Ubuntu Xenial):
assignee: nobody → Matthias Klumpp (ximion)
Krino Hoogestraat (krino) wrote :

Bug still persists after update: 20161128 18:00 GMT.

Matthias Klumpp (ximion) wrote :

@krino: Are you sure that you updated the AppStream package?
Because it looks like Laney's upload didn't reach x-u-proposed yet (and therefore also isn't in x-updates yet).

Iain Lane (laney) wrote :

Right, wait for a call for testing on this bug - before then you won't get it unless you build the package yourself.

Changed in appstream (Ubuntu Xenial):
importance: Undecided → Medium
Matthias Klumpp (ximion) wrote :

The Debian bug is about the warning message not being actionable when it appears, it does *not* appear at Debian under normal circumstances, hence it is a completely different issue (and one that isn't easily solvable).

Changed in appstream (Debian):
importance: Unknown → Undecided
status: Unknown → New
affects: appstream (Debian) → debian
Changed in debian:
status: New → Invalid
3DRaven (3draven) wrote :

I confirm this bug. Ubuntu 16.04.

Hello Thomas, or anyone else affected,

Accepted appstream into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/appstream/0.9.4-1ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in appstream (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed
no longer affects: debian
paz (mozit) wrote :

No more errors on Xenial ...

 sudo apt update && sudo apt upgrade
[sudo] password for just:
Hit:1 http://ppa.launchpad.net/noobslab/themes/ubuntu xenial InRelease
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://archive.canonical.com/ubuntu xenial InRelease
Ign:4 http://download.opensuse.org/repositories/home:/Horst3180/xUbuntu_16.04 InRelease
Get:5 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Hit:6 http://download.opensuse.org/repositories/home:/Horst3180/xUbuntu_16.04 Release
Get:7 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial-proposed InRelease [253 kB]
Fetched 560 kB in 2s (225 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Thank you!

Matthias Klumpp (ximion) wrote :

If one of you uses KDE, it would be useful to check if Plasma Discover still works as expected.

Krino Hoogestraat (krino) wrote :

If I use
sudo apt update && sudo apt full-upgrade
the bug recurs:

krino@krino-desktop:~$ sudo apt update && sudo apt full-upgrade
[sudo] Passwort für krino:
OK:1 http://ppa.launchpad.net/bitcoin/bitcoin/ubuntu xenial InRelease
Holen:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
OK:3 http://archive.ubuntu.com/ubuntu xenial InRelease
Holen:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Holen:5 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Holen:6 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 DEP-11 Metadata [19,4 kB]
Holen:7 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [366 kB]
Holen:8 http://security.ubuntu.com/ubuntu xenial-security/universe DEP-11 64x64 Icons [25,6 kB]
Holen:9 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 DEP-11 Metadata [212 B]
Holen:10 http://security.ubuntu.com/ubuntu xenial-security/main amd64 DEP-11 Metadata [68,1 kB]
Holen:11 http://security.ubuntu.com/ubuntu xenial-security/main DEP-11 64x64 Icons [39,7 kB]
Holen:12 http://archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages [362 kB]
Holen:13 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 DEP-11 Metadata [121 kB]
Holen:14 http://archive.ubuntu.com/ubuntu xenial-updates/universe DEP-11 64x64 Icons [145 kB]
Holen:15 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 DEP-11 Metadata [2.516 B]
Holen:16 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [431 kB]
Holen:17 http://archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages [425 kB]
Holen:18 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 DEP-11 Metadata [293 kB]
Holen:19 http://archive.ubuntu.com/ubuntu xenial-updates/main DEP-11 64x64 Icons [184 kB]
Holen:20 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 DEP-11 Metadata [212 B]
Holen:21 http://archive.ubuntu.com/ubuntu xenial-backports/multiverse amd64 DEP-11 Metadata [212 B]
Holen:22 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 DEP-11 Metadata [208 B]
Es wurden 2.789 kB in 4 s geholt (603 kB/s).
AppStream cache update completed, but some metadata was ignored due to errors.
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
Alle Pakete sind aktuell.
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
Paketaktualisierung (Upgrade) wird berechnet... Fertig
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 0 nicht aktualisiert.
krino@krino-desktop:~$

Dan Mick (dmick-m) wrote :

Updated package removed error message for me:

# apt-cache policy appstream
appstream:
  Installed: 0.9.4-1ubuntu2
  Candidate: 0.9.4-1ubuntu2
  Version table:
     0.10.1-1~ubuntu16.04.1 100
        100 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
 *** 0.9.4-1ubuntu2 400
        400 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     0.9.4-1ubuntu1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
     0.9.4-1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

Krino Hoogestraat (krino) wrote :

Thanks for the fix!
Problem appears to have been solved (0.10.1).

Bogdan Yurov (nick4fake) wrote :

This is still an issue for me on Xenial:

** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'gvim.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'display-im6.q16.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'gimp.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'org.gnome.Contacts.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'pidgin.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'org.gnome.font-viewer.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'chromium-browser.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'owncloud.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'gnome-system-monitor-kde.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'eog.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'fcitx.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'virtualbox.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'unity-appearance-panel.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'org.gnome.Calendar.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'software-properties-drivers.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'org.gnome.Books.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'ubuntu-mate-software.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'vim.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'libreoffice-draw.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'abiword.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'upstart-monitor.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'boinc-manager.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'tracker-preferences.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'comix.desktop': The component is invalid.
** (appstreamcli:1017): DEBUG: WARNING: Skipped component 'PlayOnLinux.desktop': The component is invalid.

(and many more lines)

I am still getting this error after running

sudo apt update

I m running fully upgraded Ubuntu 16.04.1 - here's appstream info

apt-cache policy appstream
appstream:
  Installed: 0.9.4-1ubuntu1
  Candidate: 0.9.4-1ubuntu1
  Version table:
     0.10.1-1~ubuntu16.04.1 100
        100 http://gb.archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
 *** 0.9.4-1ubuntu1 500
        500 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     0.9.4-1 500
        500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

As you can see I have 0.9.4-1ubuntu1 instead of 0.9.4-1ubuntu2 [that Dan Mick (dmick-m) has].

Running:

sudo apt install appstream
Reading package lists... Done
Building dependency tree
Reading state information... Done
appstream is already the newest version (0.9.4-1ubuntu1).
appstream set to manually installed.

Should I just wait - or is there something else I need to do?

On a side note, because I usually run a custom command:

alias maintain='set -x; sudo apt update; sudo apt upgrade; sudo apt autoremove; sudo apt autoclean; set +x'

the error goes away on a subsequent sudo apt update (that I run to reproduce the error) until some time passes and then it reappears. I think I read somewhere this has to do with the sudo apt upgrade running but I 'm not sure.

Hope that info helps.

Matthias Klumpp (ximion) wrote :

@Nikolaos: The update isn't distributed for everyone yet, to actually get it you will have to enable the x-proposed-updates source and get the package from there.

Daemmon (daemmon) wrote :

Per post #25, the proposed fix worked for me.

tags: added: verification-done
removed: verification-needed
kalaora (kalaoke) wrote :

Hi,
i'm still getting the error after running apt-get update :

Réception de:100 http://fr.archive.ubuntu.com/ubuntu xenial-proposed/restricted amd64 DEP-11 Metadata [201 B]
55,4 Mo réceptionnés en 8s (6 210 ko/s)
AppStream cache update completed, but some metadata was ignored due to errors.
Lecture des listes de paquets... Fait
W: Erreur de GPG : http://fr.archive.ubuntu.com/ubuntu xenial InRelease : Au moins une signature non valable a été rencontrée.
W: The repository 'http://fr.archive.ubuntu.com/ubuntu xenial InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: Erreur de GPG : http://fr.archive.ubuntu.com/ubuntu xenial-updates InRelease : Au moins une signature non valable a été rencontrée.
W: The repository 'http://fr.archive.ubuntu.com/ubuntu xenial-updates InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: Erreur de GPG : http://security.ubuntu.com/ubuntu xenial-security InRelease : Au moins une signature non valable a été rencontrée.
W: The repository 'http://security.ubuntu.com/ubuntu xenial-security InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: Erreur de GPG : http://fr.archive.ubuntu.com/ubuntu xenial-proposed InRelease : Au moins une signature non valable a été rencontrée.
W: The repository 'http://fr.archive.ubuntu.com/ubuntu xenial-proposed InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.

i have install the fix version. The command apt-cache policy appstream give :

appstream:
  Installé : 0.9.4-1ubuntu2
  Candidat : 0.9.4-1ubuntu2
 Table de version :
 *** 0.9.4-1ubuntu2 400
        400 http://fr.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     0.9.4-1ubuntu1 500
        500 http://fr.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
     0.9.4-1 500
        500 http://fr.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

May i miss something?

luc

Matthias Klumpp (ximion) wrote :

@kalaora: Can you please run `sudo appstreamcli refresh --force --verbose` and paste the output somewhere?

Download full text (4.5 KiB)

I'm still getting the error after running apt-get update

appstream:
   Installerad: 0.9.4-1ubuntu2
   Kandidat: 0.9.4-1ubuntu2
   Versionstabell:
      0.10.1-1~ubuntu16.04.1 100
         100 http://se.archive.ubuntu.com/ubuntu xenial-backports/main
amd64 Packages
  *** 0.9.4-1ubuntu2 400
         400 http://se.archive.ubuntu.com/ubuntu xenial-proposed/main
amd64 Packages
         100 /var/lib/dpkg/status
      0.9.4-1ubuntu1 500
         500 http://se.archive.ubuntu.com/ubuntu xenial-updates/main
amd64 Packages
      0.9.4-1 500
         500 http://se.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

#
Läs:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Bra:2 http://se.archive.ubuntu.com/ubuntu xenial InRelease

Läs:3 http://se.archive.ubuntu.com/ubuntu xenial-updates InRelease [102
kB]

Bra:4 http://archive.canonical.com/ubuntu xenial InRelease

Läs:5 http://se.archive.ubuntu.com/ubuntu xenial-backports InRelease
[102 kB]
Läs:6 http://se.archive.ubuntu.com/ubuntu xenial-proposed InRelease [253
kB]
Bra:7 http://download.virtualbox.org/virtualbox/debian xenial InRelease

Läs:8 http://security.ubuntu.com/ubuntu xenial-security/main amd64
Packages [189 kB]
Läs:9 http://se.archive.ubuntu.com/ubuntu xenial-updates/main amd64
Packages [440 kB]
Läs:10 http://security.ubuntu.com/ubuntu xenial-security/main i386
Packages [184 kB]
Läs:11 http://se.archive.ubuntu.com/ubuntu xenial-updates/main i386
Packages [433 kB]
Läs:12 http://security.ubuntu.com/ubuntu xenial-security/main
Translation-en [77,7 kB]
Läs:13 http://security.ubuntu.com/ubuntu xenial-security/main amd64
DEP-11 Metadata [68,2 kB]
Läs:14 http://security.ubuntu.com/ubuntu xenial-security/main DEP-11
64x64 Icons [47,6 kB]
Läs:15 http://se.archive.ubuntu.com/ubuntu xenial-updates/main amd64
DEP-11 Metadata [292 kB]
Läs:16 http://security.ubuntu.com/ubuntu xenial-security/universe amd64
Packages [64,5 kB]
Läs:17 http://security.ubuntu.com/ubuntu xenial-security/universe i386
Packages [62,3 kB]
Läs:18 http://security.ubuntu.com/ubuntu xenial-security/universe amd64
DEP-11 Metadata [19,4 kB]
Läs:19 http://security.ubuntu.com/ubuntu xenial-security/universe DEP-11
64x64 Icons [25,6 kB]
Läs:20 http://se.archive.ubuntu.com/ubuntu xenial-updates/main DEP-11
64x64 Icons [180 kB]
Läs:21 http://security.ubuntu.com/ubuntu xenial-security/multiverse
amd64 DEP-11 Metadata [212 B]
Läs:22 http://se.archive.ubuntu.com/ubuntu xenial-updates/universe amd64
Packages [368 kB]
Läs:23 http://se.archive.ubuntu.com/ubuntu xenial-updates/universe i386
Packages [364 kB]
Läs:24 http://se.archive.ubuntu.com/ubuntu xenial-updates/universe amd64
DEP-11 Metadata [121 kB]
Läs:25 http://se.archive.ubuntu.com/ubuntu xenial-updates/universe
DEP-11 64x64 Icons [145 kB]
Läs:26 http://se.archive.ubuntu.com/ubuntu xenial-updates/multiverse
amd64 DEP-11 Metadata [2 520 B]
Läs:27 http://se.archive.ubuntu.com/ubuntu xenial-backports/main amd64
DEP-11 Metadata [208 B]
Läs:28 http://se.archive.ubuntu.com/ubuntu xenial-backports/universe
amd64 DEP-11 Metadata [212 B]
Läs:29 http://se.archive.ubuntu.com/ubuntu xenial-backports/multiverse
amd64 DEP-11 Metadata [212 B]
Läs:...

Read more...

Matthias Klumpp (ximion) wrote :

You skipped the important information... Please run `sudo appstreamcli refresh --force --verbose` again and attach the *full* log to this report (or paste it somewhere and link it here).

Den 2016-12-09 kl. 17:00, skrev Matthias Klumpp:
> You skipped the important information... Please run `sudo appstreamcli
> refresh --force --verbose` again and attach the *full* log to this
> report (or paste it somewhere and link it here).
>

Ok, here is the link to the report file

https://drive.google.com/file/d/0BxNbM4-pYIHpVENhLWdWaldWaEE/view?usp=sharing

--
//Tommy

Matthias Klumpp (ximion) wrote :

Okay, what is the version of your libappstream3 package? You must have updated both to the version in x-u-proposed, otherwise this patch doesn't work.

Den 2016-12-09 kl. 18:39, skrev Matthias Klumpp:
> Okay, what is the version of your libappstream3 package? You must have
> updated both to the version in x-u-proposed, otherwise this patch
> doesn't work.
>
Updated to libappstream3/xenial-proposed

apt-cache policy libappstream3
libappstream3:
  Installerad: 0.9.4-1ubuntu2
  Kandidat: 0.9.4-1ubuntu2
  Versionstabell:
 *** 0.9.4-1ubuntu2 400
        400 http://se.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     0.9.4-1ubuntu1 500
        500 http://se.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
     0.9.4-1 500
        500 http://se.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
#
**
** (appstreamcli:9765): DEBUG: Removing old rebuild-dir from previous database rebuild.
AppStream cache update completed successfully.

Now it seems to work

Thanks!

--
//Tommy

B. T. (bt425) wrote :
Download full text (20.9 KiB)

I'm getting the following error, yet I don't understand what I need to do about it. Help please. Thanks in advance.

** (appstreamcli:11741): DEBUG: Refreshing AppStream cache
** (appstreamcli:11741): DEBUG: Reading: /usr/share/app-info/xmls/org.freedesktop.fwupd.xml
** (appstreamcli:11741): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial_main_dep11_Components-amd64.yml.gz
** (appstreamcli:11741): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:11741): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:11741): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:11741): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_main_dep11_Components-amd64.yml.gz
** (appstreamcli:11741): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:11741): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:11741): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:11741): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_main_dep11_Components-amd64.yml.gz
** (appstreamcli:11741): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:11741): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:11741): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:11741): DEBUG: Reading: /var/cache/app-info/xmls/fwupd.xml
** (appstreamcli:11741): DEBUG: Detected colliding ids: plan.desktop was already added with the same priority.
** (appstreamcli:11741): DEBUG: Detected colliding ids: flcheckers.desktop was already added with the same priority.
** (appstreamcli:11741): DEBUG: Detected colliding ids: flblocks.desktop was already added with the same priority.
** (appstreamcli:11741): DEBUG: Detected colliding ids: flsudoku.desktop was already added with the same priority.
** (appstreamcli:11741): DEBUG: Replaced 'libreoffice-writer.desktop' with data of higher priority.
** (appstreamcli:11741): DEBUG: Replaced 'eog.desktop' with data of higher priority.
** (appstreamcli:11741): DEBUG: Replaced 'update-manager.desktop' with data of higher priority.
** (appstreamcli:11741): DEBUG: Replaced 'gnome-session-properties.desktop' with data of higher priority.
** (appstreamcli:11741): DEBUG: Replaced 'upstart-monitor.desktop' with data of higher priority.
** (appstreamcli:11741): DEBUG: Replaced 'fcitx-configtool.desktop' with data of higher priority.
** (appstreamcli:11741): DEBUG: Replaced '...

Robie Basak (racb) wrote :

We have both success and not-fixed reports from -proposed testing. However, amongst those who reported not-fixed, I can't find any confirmation that the reporters did actually have both necessary -proposed packages updated, except for Tommy who confirmed that updating both packages did work.

So it seems to me that those reporting not-fixed all didn't have the necessary packages from -proposed installed, so I'll treat this as a successful verification.

Robie Basak (racb) wrote :

We do however have a couple of autopkgtest regressions. Please see http://autopkgtest.ubuntu.com/packages/a/appstream/xenial/amd64 and http://autopkgtest.ubuntu.com/packages/i/isenkram/xenial/amd64. Please could the people driving this SRU check those and comment?

Robie Basak (racb) wrote :

(these also impact other architectures - see http://people.canonical.com/~ubuntu-archive/pending-sru.html)

Matthias Klumpp (ximion) wrote :

The AppStream CI failure is an error on the tests and can be ignored (I can also provide a patch to fix that).
The Isenkram issue looks very weird, since the GIR ABI hasn't been touched and it doesn't seem to operate on AppStream-specific stuff there - I will investigate that problem later.

Urijah (urijah2016) wrote :

Why hasn't the patch been pushed from Proposed yet?

Urijah (urijah2016) wrote :

^ Anyway, I used the fix from proposed and it worked for me.

I confirm that the bug is fixed.

LC_ALL=C apt-cache policy appstream libappstream3
appstream:
  Installed: 0.9.4-1ubuntu2
  Candidate: 0.9.4-1ubuntu2
  Version table:
     0.10.1-1~ubuntu16.04.1 100
        100 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
 *** 0.9.4-1ubuntu2 400
        400 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     0.9.4-1ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
     0.9.4-1 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
libappstream3:
  Installed: 0.9.4-1ubuntu2
  Candidate: 0.9.4-1ubuntu2
  Version table:
 *** 0.9.4-1ubuntu2 400
        400 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     0.9.4-1ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
     0.9.4-1 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial

Matthias Klumpp (ximion) wrote :

Eww, this dropped off my todo list accidentally between the years.
Sorry guys, I'll update the patch soon so it can be distributed via the regular updates.

Norman (normal-norman) wrote :

Any chance this will be fixed before the 2nd point release of Ubuntu Xenial Xerus?

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package appstream - 0.9.4-1ubuntu2

---------------
appstream (0.9.4-1ubuntu2) xenial; urgency=medium

  * yaml-parser-errors.patch: Deal with errors in the YAML structure
    in the best way possible.
  * modern-metadata.patch: Recognize and return modern AppStream
    component types. (LP: #1644498)

 -- Matthias Klumpp <email address hidden> Thu, 24 Nov 2016 22:42:06 +0100

Changed in appstream (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for appstream has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Robie Basak (racb) wrote :

Looks like this caused a regression in Xenial tracked in bug 1663695.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers