aptd crashes when PPAs have non-ascii descriptions

[mathieg2]

[Impact]
When someone adds a PPA with a description that contains non-ascii letters, she later on gets apport dialogs about aptd crashes:

Title: <type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't decode byte 0xc3 in position 24: ordinal not in range(128)

[Test Case]
$ sudo add-apt-repository ppa:ts.sch.gr
$ sudo apt-get update
# Just ensure that there's at least one update coming from that PPA.
# E.g. install the numlockx package from Precise, there's then an
# update in that PPA:
$ sudo apt-get install numlockx=1.2-2
$ sudo apt-get update
$ update-manager

No need to update anything, it's enough to select numlockx in the update manager so that the PPA description is pulled.
Then wait approximately 30 seconds for apport to display the crash dialog.
Or, ls /var/crash/*apt*

[Regression Potential]
There are lots of dbus.String() calls in aptdaemon that might trigger similar errors [which are likely a red herring --cjwatson], but this case should be OK now, both for python2 and python3.

[Original bug report]
No noticable problems - this message came up during boot

ProblemType: Crash
DistroRelease: Ubuntu 12.04
Package: aptdaemon 0.43+bzr805-0ubuntu1
ProcVersionSignature: Ubuntu 3.2.0-23.36-generic-pae 3.2.14
Uname: Linux 3.2.0-23-generic-pae i686
NonfreeKernelModules: nvidia
Annotation: Search failed
ApportVersion: 2.0.1-0ubuntu1
Architecture: i386
Date: Tue Apr 10 07:45:48 2012
ExecutablePath: /usr/sbin/aptd
PackageArchitecture: all
SourcePackage: aptdaemon
Title: <type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't decode byte 0xc3 in position 24: ordinal not in range(128)
TransactionDepends: [[], [], [], [], [], [], []]
TransactionErrorCode: error-unknown
TransactionKwargs: {'filters': [u'unknown']}
TransactionLocale: en_GB.UTF-8
TransactionOutput:

TransactionPackages: [[], [], [], [], [], []]
TransactionRole: role-pk-query
UpgradeStatus: Upgraded to precise on 2012-02-26 (44 days ago)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sinay (sinays)]

An other problem that occurs with Firefox 11 is the colour distortion when running Youtube. My Ubuntu is 12.04 A, my MB is a Gigabyte P55-UD6, and the video card is a NVIDIA GEForce 8400 GS. Tks.

[Carlos Carreras (carlosmaxo7)]

This bug affect me using skype call. acer d255 processor n450.

[Glauber Franco (gnfranco)]

My system not is a i386 machine, it is a AMD64 machine.

Your report is about a 32bits machine.
Will it have some problem with a 64bits machine?

[Glauber Franco (gnfranco)]

Annotation: Search failed
ApportVersion: 2.0.1-0ubuntu1
Architecture: i386
Date: Tue Apr 10 07:45:48 2012

[[e-mail] (leonidasemail)]

My system is a i386
ubuntu beta

[timothy gordon (lbsc-lead)]

Every time I install using apt I get the error message and send report to the devs. I have not experienced any ui bugs when the error comes up so its probably some back-end thing that doesnt effect the gui.

Timothy Gordon
Operator and Holder
slashRoot: coffee house and tech dojo
60 Main street New Paltz, New York
[phone]

[Christan Adéen (christian-adeen-i)]

The dialog "System has encountered an error" (Or something similar not 100% sure) is popping up always after boot or waking up from suspend The details of the dialog tells its a duplicate of this bug. . I'm not noticing anything wrong with the system other then the update manager not telling me that there are updates available. Running Update manager manually gives this error: (Might be another problem then this bug)

"Failed to download repository information

Check your Internet connection.

W:Failed to fetch http://security.ubuntu.com/ubuntu/dists/precise-security/Release Unable to find expected entry 'independent/source/Sources' in Release file (Wrong sources.list entry or malformed file)
, W:Failed to fetch http://se.archive.ubuntu.com/ubuntu/dists/precise/Release Unable to find expected entry 'independent/source/Sources' in Release file (Wrong sources.list entry or malformed file)
, W:Failed to fetch http://se.archive.ubuntu.com/ubuntu/dists/precise-updates/Release Unable to find expected entry 'independent/source/Sources' in Release file (Wrong sources.list entry or malformed file)
, W:Failed to fetch http://se.archive.ubuntu.com/ubuntu/dists/precise-backports/Release Unable to find expected entry 'independent/source/Sources' in Release file (Wrong sources.list entry or malformed file)
, E:Some index files failed to download. They have been ignored, or old ones used instead."

My internet connection is fine and I can update on laptop om same router and that system is not reporting this error and also not the above update-manager error.

Desktop System specs (The one with this problems) AMD x2 5600 + 4gb DDR2 ram ATI radeon 4870 grapics (running "ubuntu supported" ati drivers) regular mechanic HDD 200g + an 500g for storage. Motherboard no idea.

Laptop VAIO VGN FW21M Core 2 duo 4 gb DDR2 ram som crappy mobile ati graphics (Running same driver as above) 420gb mechanical HDD

Both system running Ubuntu 12.04. Fresh install on laptop. Updated from 11.10 fresh install on desktop.

[Mark Stosberg (markstos)]

I also get notified about this issue after time I login, since I upgrade to 12.04. I don't know what the trigger is.

[Vic (ubuntu-slagter)]

After upgrading to 12.04 from 11.10 I get this message.
When I login using XFCE window mananger I do not get this error, only with the Ubuntu and Ubuntu 2D window manager.

[Alkis Georgopoulos (alkisg)]

I'm wondering if the problem is non-ascii characters in PPA descriptions.

Guys any of you have non-ascii characters in the output of this command?
sudo apt-key list
sudo apt-key list | iconv -f ascii

E.g. I have this, which contains greek chars, so it might be what's crashing aptd:

pub 1024D/9849BE20 2009-05-27
uid Hellenic Schools Technical Support Team (Τεχνική Στήριξη ΣΕΠΕΗΥ) <email address hidden>
sub 2048g/15480249 2009-05-27

[Alkis Georgopoulos (alkisg)]

The problem is probably that dbus.String() expects a unicode string, and aptdaemon is sending a utf-8 string.
Maybe is should be using value.decode("utf-8") instead:
self._last_package = dbus.String(value.decode("utf-8"))

Python example:

$ python
>>> import dbus
>>> s="Hi in Greek is Γεια"
>>> dbus.String(s.decode("utf-8"))
dbus.String(u'Hi in Greek is \u0393\u03b5\u03b9\u03b1')
>>> dbus.String(s)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 15: ordinal not in range(128)

[rpr nospam (rpr-nospam)]

Alkis, the output of "sudo apt-key list" on my system contains a non-ascii character:

pub 2048R/DD7FB8CC 2010-06-28
uid Fabre François (multiboot gpg key) <email address hidden>
sub 2048R/6FECE640 2010-06-28

[François Rey (fmjrey)]

Hey I also have the same entry:
pub 2048R/DD7FB8CC 2010-06-28
uid Fabre François (multiboot gpg key) <email address hidden>
sub 2048R/6FECE640 2010-06-28

[Fotis Tsamis (ftsamis)]

It seems that Alkis is correct!

The variable 'value' inside the last_package method (aptdaemon/pkcompat.py, line 670), which is called a little after the 'Check' button is pressed, contains the package name, the available new version, the architecture AND the PPA's description string (separated with a semicolon).

E.g.: numlockx;1.2-4;i386;Αποθετήριο Τεχνικής Στήριξης ΣΕΠΕΗΥ

Now, if the PPA's description contains only ascii letters it's all good. If not (as in the example above) we have an exception.
In line 671, the 'value' variable (a str object) is passed as a parameter to the dbus.String() function which expects a unicode object. So dbus.String tries to convert it using a plain .decode() which uses ascii as the default codec and boom: 'ascii codec can't decode [...]' :)

As Alkis stated the solution is to convert the 'value' variable to a unicode object with value.decode('utf-8') before passing it to dbus.String() so it will try to convert the str to unicode using the utf-8 codec instead of ascii.

I have included a patch fixing the problem.
Fotis

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "patch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[Alkis Georgopoulos (alkisg)]

@toda: you changed the bug status to "fix committed". That means you're a developer that has solved the problem, and the Ubuntu developers don't need to work on it any more. Is that the case, or you changed it by accident?

I'm reverting the status, I'll make a quantal branch merge proposal in a while and ask for a Precise SRU if it's accepted.

[Alkis Georgopoulos (alkisg)]

A better fix is needed to properly solve the problem (python2 + 3 compatibility, 108 more dbus.String() calls to check...), so I'd better unassign it from me and leave it for someone who's willing to tackle it more thoroughly. Thanks! :)

[Colin Watson (cjwatson)]

To recapitulate my position on IRC: the other dbus.String calls aren't relevant. The problem here is that at the point when retrieving information from python-apt, aptdaemon doesn't convert it to the types it requires internally. Worrying about the other dbus.String calls would miss the point and would at best mask other bugs, possibly in ways that would cause data corruption rather than crashes.

[Launchpad Janitor (janitor)]

This bug was fixed in the package aptdaemon - 0.45+bzr861-0ubuntu4

---------------
aptdaemon (0.45+bzr861-0ubuntu4) quantal; urgency=low

  * debian/patches/apt-dbus-text-api.patch:
    - Convert Origin.label from python-apt API to python-dbus API, for
      Python 2 compatibility (LP: #978654).
 -- Colin Watson <email address hidden> Mon, 17 Sep 2012 15:12:38 +0100

[Dept.Técnico (Extreme Micro S.L.) (tecnicos-extreme-micro)]

We confirm that the patch suited by Fotis solved the crashes. Some one can update precise package too?

[Clint Byrum (clint-fewbar)]

Hello mathieg2, or anyone else affected,

Accepted aptdaemon into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/aptdaemon/0.43+bzr805-0ubuntu5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Adam Conrad (adconrad)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package aptdaemon - 0.43+bzr805-0ubuntu5

---------------
aptdaemon (0.43+bzr805-0ubuntu5) precise-proposed; urgency=low

  * debian/patches/apt-dbus-text-api.patch:
    - Convert Origin.label from python-apt API to python-dbus API, for
      Python 2 compatibility (LP: #978654).
 -- Colin Watson <email address hidden> Mon, 17 Sep 2012 15:17:56 +0100