"Import Key File" fails when the path of the file has special characters

[Bruno Nova]

[Impact]

Importing a key that is in a path that contains special characters (like "/home/$USER/Transferências") fails silently.

[Test Case]

1. Download a PGP key for a Debian repository (for example, the VideoLAN
   team's key: download.videolan.org/pub/debian/videolan-apt.asc)
2. Move the key file to a folder that contains special characters, or rename
   the file to contain them (example: VídeoLAN.asc).
3. Open "Software & Updates" from the Dash, then select the Authentication
   tab.
4. Press the "Import Key File..." button and then choose the key file.
   Admin authentication should then be requested, so authenticate yourself.

   If the bug is fixed: the key should now appear in the list of keys.
   If not: the key was not added.

[Regression Potential]

If the fix is not correct, I suppose it could potentially break the "import key" feature even more, so it wouldn't work even in "normal" paths.

[Original Report]

The "Import Key File..." button in the "Authentication" tab fails silently when the path of the selected key file includes special characters.

I'm using Ubuntu in Portuguese, so the "/home/$USER/Downloads" folder is named "/home/$USER/Transferências" here.
Adding a key from that folder fails, but adding a key from "/home/$USER" (no special characters) succeeds.

The problem is that, in softwareproperties/gtk/SoftwarePropertiesGtk.py in method SoftwarePropertiesGtk.add_key_clicked() at around line 952:

        if res == Gtk.ResponseType.ACCEPT:
            try:
                if not self.backend.AddKey(chooser.get_filename()):
                    error(self.window_main,
                          _("Error importing selected file"),
                          _("The selected file may not be a GPG key file "
                            "or it might be corrupt."))
            except dbus.DBusException as e:
                if e._dbus_error_name == 'com.ubuntu.SoftwareProperties.PermissionDeniedByPolicy':
                    logging.error("Authentication canceled, changes have not been saved")

"self.backend.AddKey(chooser.get_filename())" throws a "org.freedesktop.DBus.Python.UnicodeEncodeError" exception for paths with special characters.
This is probably a side effect of switching from Python 2 to 3 (and 'bytes' now being used in network communications where 'str' was previously used).

Another issue is that the error isn't reported to the user, and it should be.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: software-properties-gtk 0.92.37.1
ProcVersionSignature: Ubuntu 3.13.0-37.64-generic 3.13.11.7
Uname: Linux 3.13.0-37-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.5
Architecture: amd64
CurrentDesktop: Unity
Date: Tue Oct 14 13:22:21 2014
EcryptfsInUse: Yes
InstallationDate: Installed on 2014-10-13 (0 days ago)
InstallationMedia: Ubuntu 14.04.1 LTS "Trusty Tahr" - Release amd64 (20140722.2)
PackageArchitecture: all
SourcePackage: software-properties
UpgradeStatus: No upgrade log present (probably fresh install)

[Bruno Nova (brunonova)]

Here's a traceback of the exception:

org.freedesktop.DBus.Python.UnicodeEncodeError: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/dbus/service.py", line 707, in _message_cb
    retval = candidate_method(self, *args, **keywords)
  File "/usr/lib/python3/dist-packages/softwareproperties/dbus/SoftwarePropertiesDBus.py", line 288, in AddKey
    return self.add_key(path)
  File "/usr/lib/python3/dist-packages/softwareproperties/SoftwareProperties.py", line 800, in add_key
    if not os.path.exists(path):
  File "/usr/lib/python3.4/genericpath.py", line 19, in exists
    os.stat(path)
UnicodeEncodeError: 'ascii' codec can't encode character '\xea' in position 20: ordinal not in range(128)

[Bruno Nova (brunonova)]

I finally found the cause of the problem!
It seems that the DBus service (or the whole DBus) is not running in a UTF-8 locale, thus causing errors in UTF-8 strings.

The workaround is to edit softwareproperties/SoftwareProperties.py and add:
    path = path.encode('utf-8')
in method "def add_key(self, path)" around line 800 before the line:
    if not os.path.exists(path):

This encoding issue may be causing problems in other parts of the program, and other programs.

Meanwhile, I found another bug:
Dragging and dropping a key file to the list doesn't work, raising an exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/softwareproperties/gtk/SoftwarePropertiesGtk.py", line 759, in on_auth_drag_data_received
    keydata = selection.data.strip()
AttributeError: 'SelectionData' object has no attribute 'data'

[Bruno Nova (brunonova)]

Reported that other bug I mentioned in Bug #1383289.

[Brian Murray (brian-murray)]

Michael - Could you have a look at this?

[Launchpad Janitor (janitor)]

This bug was fixed in the package software-properties - 0.96.1

---------------
software-properties (0.96.1) vivid; urgency=low

  * fix autopkgtest failure
 -- Michael Vogt <email address hidden> Mon, 01 Dec 2014 08:04:47 +0100

[Brian Murray (brian-murray)]

Hello Bruno, or anyone else affected,

Accepted software-properties into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/software-properties/0.92.37.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Bruno Nova (brunonova)]

I updated to -proposed and tested but, unfortunately, the bug isn't fixed!
The other bug is fixed, though.

Adding a key in "/home/$USER/Transferências" didn't work. Nothing happened after authenticating as admin.
However, if I run software-properties-dbus manually (by executing "sudo python3 /usr/lib/software-properties/software-properties-dbus"), it works.

After further investigation, I would say that software-properties-dbus is started in the C locale (non UTF-8), making it fail in paths with special characters.
I checked the file "/proc/$software-properties-dbus-PID/environ", and it's almost empty. It only contains the variables DBUS_STARTER_ADDRESS and DBUS_STARTER_BUS_TYPE, but no locale related variables.

@Michael, that's probably why I said that the fix with your changes worked (https://code.launchpad.net/~brunonova/software-properties/lp1381050/+merge/238873/comments/613431).
It seems that your simplification doesn't work after all. :)

@Michael, check the diff here: https://code.launchpad.net/~brunonova/software-properties/lp1381050/+merge/238873
Adding that line fixes this bug, I've checked it now.
Do you want to make this change yourself, or shall I do it?

[Brian Murray (brian-murray)]

I've followed the test case provided for Wily and the bug does not seem fixed there either, so I am going to set the task back to In Progress and look at Bruno Nova's change that was missed.

[Brian Murray (brian-murray)]

Using the change provided (path = path.encode('utf-8') # ensure path is encoded in UTF-8) still doesn't seem to fix the issue for me on Wily.

[Brian Murray (brian-murray)]

Here is the Traceback I see

ipdb> n
dbus.exceptions.DBusException: org.freedesktop.DBus.Python.UnicodeEncodeError: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/dbus/service.py", line 707, in _message_cb
    retval = candidate_method(self, *args, **keywords)
  File "/usr/lib/python3/dist-packages/softwareproperties/dbus/SoftwarePropertiesDBus.py", line 288, in AddKey
    return self.add_key(path)
  File "/usr/lib/python3/dist-packages/softwareproperties/SoftwareProperties.py", line 804, in add_key
    path = path.encode("utf-8")
  File "/usr/lib/python3.4/genericpath.py", line 19, in exists
    os.stat(path)
UnicodeEncodeError: 'ascii' codec can't encode character '\xed' in position 58: ordinal not in range(128)

[Bruno Nova (brunonova)]

I added a comment to the merge request.

[Mathew Hodson (mhodson)]

https://launchpad.net/ubuntu/+source/software-properties/0.92.37.4 has been removed from trusty-proposed.

The new software-properties 0.92.37.5 package reverted the changes for "fix importing keys from path with special chars" so this bug shouldn't block the SRU of the new package.

[Launchpad Janitor (janitor)]

This bug was fixed in the package software-properties - 0.92.37.5

---------------
software-properties (0.92.37.5) trusty-proposed; urgency=medium

  * cloudarchive: Enable support for the Liberty Ubuntu Cloud Archive on
    14.04 (LP: #1472586).
  * Revert SRU changes for "fix importing keys from path with special chars"
    due to verification-failed in trusty-proposed.

 -- Corey Bryant <email address hidden> Wed, 02 Sep 2015 12:31:11 -0400