[software-properties-gtk] can not delete, enable or modify any software source with non-ASCII characters in the comment

[Олег АндреичЪ]

[Test Case]
1) add a unicode character to your /etc/apt/sources.list as a comment
2) launch software-properties-gtk
3) try enabling or disabling the line containing a unicode comment
4) authenticate
5) observe the line is unchanged

Install software-properties-gtk from trusty-propopsed and then reboot.

With the new version of software-properties-dbus running you should be able to enable and disable the entry in sources list.

I've tried to reenable PPA's after upgrading the system, so there is no reaction on delete button and no any changes after modifying data via modify dialog. I can't even enable a source.

ProblemType: BugDistroRelease: Ubuntu 12.10
Package: software-properties-gtk 0.92.9
ProcVersionSignature: Ubuntu 3.5.0-17.28-generic 3.5.5
Uname: Linux 3.5.0-17-generic i686
ApportVersion: 2.6.1-0ubuntu4
Architecture: i386
Date: Sat Oct 20 16:07:59 2012
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release i386 (20111012)
MarkForUpload: True
PackageArchitecture: all
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=ru_RU.UTF-8
 SHELL=/bin/bashSourcePackage: software-properties
UpgradeStatus: Upgraded to quantal on 2012-10-19 (1 days ago)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Olivier Febwin (febcrash)]

I've the same problem.
It seems that this is due to "é" character

[jonaternet (jonaternet)]

Same for me, I have french locale with the letter éin description. I can modify/remove/enable the repositories that don't contain this letter, not these with that contaion it. see attachments.

[Robert Roth (evfool)]

As this happens for many languages (as the disabled during upgrade comment contains non-ASCII characters), I am marking this as a Medium priority issue.

[PhoneixS (phoneixsegovia)]

Ok, I have tried it and as a workaround you can delete that conflictin characters from sources manually.
How to:
1º Open terminal.
2º cd /etc/apt/sources.list.d
3º Open each .list file that can't be changed as root (sudo nano irie-blender-precise.list, for example).
4º Remove (or change) any character that isn't ascii.
5º Save the file and repeat the process as needed.

[Matt Fischer (mfisch)]

It's not all non-English characters, for example, the name of one of the commenters works fine in a comment: Олег АндреичЪ

[Matt Fischer (mfisch)]

So I played with this some tonight and cannot make much sense of the results. After adding my debugging stuff, it seemed to work fine. I restarted software-properties-dbus with --debug and also added some code into software-properties-gtk to compare the source line. As I said, it worked fine, here are my results. I used the accented e that had previously caused issues.

In this example, I'm removing my stormcloud PPA's which have the accented e in the comment.

mfisch@caprica:~$ sudo /usr/lib/software-properties/software-properties-dbus --debug
DEBUG:root:waiting for connections
mfisch: deb-src http://ppa.launchpad.net/cooperjona/stormcloud/ubuntu quantal main #é qqqq
mfisch: deb-src http://ppa.launchpad.net/cooperjona/stormcloud/ubuntu quantal main #é qqqq
DEBUG:root:SourcesListModified signal
mfisch: deb http://ppa.launchpad.net/cooperjona/stormcloud/ubuntu quantal main #é f
mfisch: deb http://ppa.launchpad.net/cooperjona/stormcloud/ubuntu quantal main #é f
DEBUG:root:SourcesListModified signal

You can also see what software-properties-gtk reported, I was comparing the source_entry with the str(source_entry) and they're the same.

remove button clicked
source_entry: deb-src http://ppa.launchpad.net/cooperjona/stormcloud/ubuntu quantal main #é qqqq
str(source_entry): deb-src http://ppa.launchpad.net/cooperjona/stormcloud/ubuntu quantal main #é qqqq
remove button clicked
source_entry: deb http://ppa.launchpad.net/cooperjona/stormcloud/ubuntu quantal main #é f
str(source_entry): deb http://ppa.launchpad.net/cooperjona/stormcloud/ubuntu quantal main #é f

The next step will be to get in between the code I annotated and the dbus service. Perhaps someone else can continue this.

[Tommy_CZ (t-kijas)]

I have similar problem, I could not enable nor disable any of software sources, but after I tried to disable everything, that remained after upgrade from 12.04 to 12.10 (with statement: "Disabled during upgrade" or something like that, in Czech language: " zakázáno při povýšení na quantal") I am able to disable/enable some of them now (not still not those with statement).
Upgrade was partially successful, because it crashed and I had to resolve it manually.

[elhoir (jfarroyo82)]

hi there,
Yesterday i upgraded to Raring Ringtail, and i have the same problem in 13.04 (still alpha)

[Matej Kovacic (matej-kovacic)]

I have Quantal (upgraded recently), and can confirm the problem.

In fact the problem are non-english characters. When you upgrade, upgrade tool would disable old sources and add commnet they are disabled. In slovenian language comment is like: "onemogočeno ob nadgradnji na quantal". It contains slovenian character "č".

When I changed it to "onemogoceno ob nadgradnji na quantal" ("c" instead of "č"), everything works fine.

It is a nasty bug for all who don't know how to edit sources.list from commandline, because you can not enable, remove or edit disabled sources from GUI tool.

I think solution should be quite trivial, so it would be great if developers could solve that with higher priority.

[Paulo J. S. Silva (pjssilva)]

It is time for a new update and the bug is still here. This means that everyone that is using Ubuntu in Portuguese or any other language that has accents in the default comment will be in trouble as pointed out by the other comments above.

Is this really only of medium importance? It affects all the users of many different languages!

[Sami Jaktholm (sjakthol)]

This bug is actually caused by python-apt which tries to open the sourceslist files with the default encoding.

As the dbus daemon backend is executed in a clean environment, the default encoding is not the same the system uses - it's the one hardcoded to python if every detection method fails (and that's not UTF-8). The backend fails to open files with unicode chars and ignores the sources inside them.

However, the GUI runs in an user session that has UTF-8 as default encoding. Thus the frontend is able to read the files and show the sources in them. When user tries to toggle or edit the sources, the backend fails to find the entry to toggle (as it was ignored) and nothing happens (well, backend throws an exception which the frontend quietly ignores).

So yes, the solution is very trivial: just make python-apt open all sourceslist files with UTF-8 encoding. However, the tricky part is to get it to work in both Python 2 and 3...

[Jason (jason-willson)]

Confirmed problem after upgrade to Raring. This happened during the last upgrade (precise to quantal), and I don't remember exactly what I did, but I think it involved something with the .save files. If I find out what works, I'll post.

[Jorge Suárez de Lis (ys)]

As a workaround, I did this to replace the problematic comments with ASCII-friendly ones. Notice that this only works with the Spanish locale. You must replace the message according with what message is shown to you on the comment section.

sudo sed -ri 's/(.*)\(desactivado en la actualización a raring\)(.*)/\1disabled on update\2/' /etc/apt/sources.list.d/*.list

[Fernando Conesa (brobollon)]

The solution of Jorge worked for me without problem.
The change in the message won't affect forthcoming updates right?
I mean, for the 13.10, the message will be changed again?

[Daniel Hartwig (wigs)]

On 7 May 2013 06:11, Fernando Conesa <email address hidden> wrote:
> The solution of Jorge worked for me without problem.
> The change in the message won't affect forthcoming updates right?
> I mean, for the 13.10, the message will be changed again?
>

You can always use a text editor to remove any offending characters.

[Paulo J. S. Silva (pjssilva)]

I attach a script that should be able to fix the files and still keep reasonable approximations of the original comments with non-ASCII characters. It assumes that the bad files are using UTF-8 encoding. To use it you should (as root, that is using sudo):

0) Copy recusively /etc/apt/sources.list.d to a safe location
1) Copy the fix-source-list.sh (attached file) to /etc/apt/sources.list.d and chdir there
2) Make it executable
3) run it typing "sudo ./fix-source-list.sh"
4) Delete fix-source-list.sh from /etc/apt/sources.list.d

It will create new version of the *.list files with the non-ASCII characters converted to an ASCII approximation.

Now try "apt-get update", if it works OK, try to edit your sources using software-properties.

If something goes wrong, delete the bad /etc/apt/sources.list.d and copy back the backup you made in step 0 (you did it, right?).

[Matej Kovacic (matej-kovacic)]

I just upgraded to 13.04 and the problem is still here. This is just sad. I think the only realiable and fast solution would be to pass that information to our transaltion teams and ask them to remove special characters and instead just use basic ASCII alphabet.

P. S. Did anyone passed the infromation about bug python-apt developers?

[Tommy_CZ (t-kijas)]

The bug still exists, I cannot believe noone fixed it yet. Matej, I don't think so this is the permanent solution, because software-properties "should" be able to handle it properly.

[Tommy_CZ (t-kijas)]

The funny thing is that when you add a repository, you cannot disable it, nor delete, nor change. And I mean repository without commentary with different characters. The whole application is unusable then.

[Matej Kovacic (matej-kovacic)]

Yes, I agree. Anyway, I passed my "solution" to slovenian translators and they will change the translation. Unfortunately this is a bad solution (and working only for slovenian users), but it seems is the only one, which is available at the moment.

[Tommy_CZ (t-kijas)]

I have one question, but I don't need the answer, it is a question to all of us.... don't you think, that this situation, that disables the possibility to enable/disable software sources for many people in the world, will help Ubuntu?

I do not use my computer for work (so much), but others do and they can become pretty angry that canonical released Ubuntu 13.04 with such a bug and until now it is not resolved.

This is not professional, the possibility to add software sources is one of the most important ways to use OS!

It is just a warning. Something like that will not be tolerated by massess. Let's think about that.

[Steve Langasek (vorlon)]

Interestingly, I can reproduce this; but when I tried to debug it (by running python3 /usr/lib/software-properties/software-properties-dbus manually from a root shell and watching its output), the bug was no longer reproducible. This pointed to a problem specific to the process environment - and indeed, if I run software-properties-dbus with LANG=C, I get the following error (repeatedly):

WARNING:root:could not open file '/etc/apt/sources.list'

This points to a bug in aptsources.sourceslist.SourcesList.load(), which is using the default encoding (ascii) when opening the file for reading.

[Maarten Jacobs (maarten256)]

I just came across the same issue after I upgraded to 13.10 Saucy from 13.04 Raring and I could no longer change my Software Sources.

Turned out I had a ô hiding in one of the comments in my sources.list. Things worked fine after I changed that to an o without ^.

[moray33 (moray33)]

After one year, this bug still ocurrs. This should be high priority, to many users this must be a pain in the ass, to put it blunty.

[Paulo J. S. Silva (pjssilva)]

Yeap, bug still there. At least the script I did to workaround it, avaiable in comment #20, still works.

[moray33 (moray33)]

Thanks Paulo, at least it's a help for some users, but probably not for the regular user. I hope this bug will be solved before the next version update because it's important and annoying.

[Tuomo Sipola (tuomosipola)]

Confirmed. Happened when upgrading from 12.10 to 13.04 and now persists after upgrade from 13.04 to 13.10. I am using Finnish translation, so there are special characters when the repositories are commented out by the upgrader.

[moray33 (moray33)]

I told the Spanish translators about this issue and they changed the translation to this bug doesn't happen, in the meantime it's fixed.

[Fernando Conesa (brobollon)]

The 3 last versions of Ubuntu with the same problem. This is quite annoying. I really don't understand why it isn't fixed.

[Julian Andres Klode (juliank)]

I will take care of this Friday or Saturday

[Angel Guzman Maeso (shakaran)]

@juliank some progress with this bug?

[Fernando Conesa (brobollon)]

no solution so far?

[Sebastien Bacher (seb128)]

@Julian: are you still working on that? Did you see the merge request from Steve linked to this bug?

[Brian Murray (brian-murray)]

I did some testing of Steve's changes and found them to be insufficient.

[Brian Murray (brian-murray)]

I've tested Steve's changes again on Trusty and Saucy and they do work, I believe last time I had just not kill and restarted software-properties-dbus.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 0.9.3.2ubuntu2

---------------
python-apt (0.9.3.2ubuntu2) trusty; urgency=medium

  * Fix pep8 error.
 -- Brian Murray <email address hidden> Fri, 14 Mar 2014 14:20:43 -0700

[Julian Andres Klode (juliank)]

Attaching a better approach then the currently broken one.

Instead of using unicode in Python 2, keep bytes there. This will work all the time, because those objects do not care about encoding at all. An additional unicode() is provided to decode an entry, considering it as UTF-8.

In Python 3, open the files in UTF-8, and provide a bytes() method to encode them as UTF-8 again.

Please note that this bug does not happen if you do not run programs in the non-unicode C locale. If you want to interface with utf-8 files in a language-agnostic environment, use the C.UTF-8 locale. Other apps will break just as well.

[Adam Collard (adam-collard)]

The fix for this bug introduced tracebacks when trying to save empty sources lists see LP: #1294531

[Julian Andres Klode (juliank)]

This change has been reverted in upstream release 0.9.3.4. I am working on an alternative way of doing this.

[Julian Andres Klode (juliank)]

I believe we should fix this in software-properties. The most flexible way to do this is to use something like

if sys.getfilesystemencoding() == "ascii" and not "LANG" in os.environ:
    os.putenv("LANG", "C.UTF-8")
    with open("/etc/default/locale") as fobj:
        for line in fobj:
            line = line.split("#")[0].strip()
            if line:
                key, value = line.split("=", 1)
                os.putenv(key, value)

        os.execv(sys.argv[0], sys.argv)

before running other code in the dbus service. This will take care to setup the system's default locale settings and then re-exec() itself so that Python picks this up (it will not change filesystem encoding otherwise). In case no locale is set in /etc/default/locale, it falls back to LANG=C.UTF-8.

This can be changed to use systemd's localed later on (I do not know if it is currently installed), using:

 bus = dbus.SystemBus()
 localed = bus.get_object("org.freedesktop.locale1", "/org/freedesktop/locale1")
 loc = dbus.Interface(localed, 'org.freedesktop.locale1')
 props = dbus.Interface(localed, 'org.freedesktop.DBus.Properties')

 for locale in props.Get('org.freedesktop.locale1', 'Locale'):
  name, val = locale.split("=", 1)
  os.putenv(name, val)

to get the values instead of reading /etc/default/locale.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "unicode.diff" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Michael Vogt (mvo)]

Thanks Julian and Brian for your diagnosis here. I agree that software-properties-dbus should setup a C.UTF-8 environment. The lp:~mvo/software-properties/lp1069019 branch should address this and at least ensure that the sources.list editing works.

[Brian Murray (brian-murray)]

Hello Олег, or anyone else affected,

Accepted software-properties into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/software-properties/0.92.37 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

With the software-properties packages version 0.92.37 from trusty-proposed I am to enable and disable a souces.list entry with unicode in it.

[b_b (brunobergot)]

I confirm that software-properties 0.92.37 (available in proposed repo after trusty upgrade) fix the bug. Thanks a lot \o/

[Launchpad Janitor (janitor)]

This bug was fixed in the package software-properties - 0.92.37

---------------
software-properties (0.92.37) trusty-proposed; urgency=low

  * software-properties-dbus: force C.utf-8 locale if C is used
    to ensure utf-8 support when reading sources.list LP: #1069019
 -- Michael Vogt <email address hidden> Thu, 17 Apr 2014 20:44:08 +0200

[Scott Kitterman (kitterman)]

The verification of the Stable Release Update for software-properties has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package software-properties - 0.92.37

---------------
software-properties (0.92.37) trusty-proposed; urgency=low

  * software-properties-dbus: force C.utf-8 locale if C is used
    to ensure utf-8 support when reading sources.list LP: #1069019
 -- Michael Vogt <email address hidden> Thu, 17 Apr 2014 20:44:08 +0200

[Brian Murray (brian-murray)]

Hello Олег, or anyone else affected,

Accepted software-properties into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/software-properties/0.92.28.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Mathew Hodson (mhodson)]

Changing back to Fix Committed for saucy. The package software-properties 0.92.28.1 is still in -proposed, and there was no reason given for the change.