Ubiquity crashes with UnicodeDecodeError in apt_pkg.size_to_str

[lisiano256]

After Ubiquity finishes copying files, it creates the users on the system (Or a bit later), then crashes.
I used Grub2 on my USB drive to boot the raring ISO via loopback. Installed with a custom partition layout (Reused old Home and formated Root).

ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: ubiquity 2.13.12
ProcVersionSignature: Ubuntu 3.8.0-7.15-generic 3.8.0
Uname: Linux 3.8.0-7-generic x86_64
ApportVersion: 2.8-0ubuntu4
Architecture: amd64
CasperVersion: 1.330
Date: Mon Feb 25 20:08:39 2013
LiveMediaBuild: Ubuntu 13.04 "Raring Ringtail" - Alpha amd64 (20130225)
MarkForUpload: True
ProcEnviron:
 TERM=unknown
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: ubiquity
UpgradeStatus: No upgrade log present (probably fresh install)

[Ubuntu Foundations Team Bug Bot (crichton)]

Exception during installation:
Feb 25 18:08:19 ubuntu /plugininstall.py: Traceback (most recent call last):
Feb 25 18:08:19 ubuntu /plugininstall.py: File "/usr/lib/ubiquity/ubiquity/install_misc.py", line 277, in pulse
Feb 25 18:08:19 ubuntu /plugininstall.py: AcquireProgress.pulse(self, owner)
Feb 25 18:08:19 ubuntu /plugininstall.py: File "/usr/lib/python3/dist-packages/apt/progress/text.py", line 187, in pulse
Feb 25 18:08:19 ubuntu /plugininstall.py: val += ' %sB' % apt_pkg.size_to_str(worker.current_size)
Feb 25 18:08:19 ubuntu /plugininstall.py: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 1: invalid start byte
Feb 25 18:08:19 ubuntu /plugininstall.py:

[Colin Watson (cjwatson)]

This is mainly a python-apt bug (compare bug 1031882; python-apt's own code will need the same fix), but there's another manifestation of the same problem in ubiquity proper.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubiquity - 2.13.13

---------------
ubiquity (2.13.13) raring; urgency=low

  [ Dmitrijs Ledkovs ]
  * Bump Standards version and override some lintian tags (ubiquity is
    weird).
  * Override expander style, which is hardcoded in light-themes to a
    light color. (LP: #1100922)
  * Consolidate CssProvider and add make minimal progress bar width 10px,
    in preparation for dot-progress of all steps.
  * Fix colour unit test, the green is no longer what it used to be...
  * Switch to "NotShowIn=KDE" for ubiquity-gtk desktop files, as most !kde
    desktop environments use gtk frontend.
  * Show progress dots for plugin pages and show progress bar only during
    slideshow.

  [ Colin Watson ]
  * Decode the output of apt_pkg.size_to_str() if necessary when
    substituting it into Unicode strings (LP: #877756, #1132918).
  * Update translations from Launchpad.

  [ Aurélien Gâteau ]
  * Turns the Ubiquity KDE window into a regular window and rework
    the layout of widgets to provide more space to the main content.
    The labels on the left now wrap to the next line if they are too long,
    ensuring they are never truncated.
  * KDE: Improve the layout of Prepare step
  * KDE: fix integration of the webview responsible for showing the slideshow
  * KDE: fix various issues with right-to-left languages

  [ Julien Lavergne ]
  * Use feh to draw the wallpaper on Lubuntu when using ubiquity-dm.

  [ Ma Jun (Kylin Team) ]
  * Fix referencing variable before assignment (LP: #945027)

  [ Kobe Lee ]
  * Show OS column in manual partitioning page. (LP: #1100694)

  [ Stéphane Graber ]
  * Fix crash in ubi-console-setup when quickly moving back and worth between
    pages. (LP: #944614)

  [ Jonathan Riddell ]
  * Add back compositing to kwin on live CD (in ubiquity-dm), whatever was wrong
    in 2008 should be fixed now and its making us look ugly
 -- Dmitrijs Ledkovs <email address hidden> Thu, 07 Mar 2013 19:53:52 +0000

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Screatch (screatch)]

I can confirm that installer crashes with latest Ubuntu 13.04 Beta 2 on Installing Languages. Workaround for choosing other language worked just fine (In my case i used Russian). Clean install. Same behaviour as in bug #1153852.

[Julian Andres Klode (juliank)]

There does not seem to be any bug in python-apt here. It returns a str, as it should. In Python 3, everything should work automatically, because str is unicode. In Python 2, you need to decode it yourself if you want to work with unicode strings, as unicode strings are not the default, so we do not use them for return values.

Can we change the status to invalid, or did I miss anything?

[Tiitise (tiitise)]

Hello!
There was a bug in 13.04 but now I'm using 13.10 and everything is working
fine!
Now it's hard to say something more about this bug.
Please change status to invalid
Have a nice day!

On Thu, Oct 24, 2013 at 9:52 AM, Julian Andres Klode <email address hidden>wrote:

> There does not seem to be any bug in python-apt here. It returns a str,
> as it should. In Python 3, everything should work automatically, because
> str is unicode. In Python 2, you need to decode it yourself if you want
> to work with unicode strings, as unicode strings are not the default, so
> we do not use them for return values.
>
> Can we change the status to invalid, or did I miss anything?
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1162454).
> https://bugs.launchpad.net/bugs/1132918
>
> Title:
> Ubiquity crashes after user creation
>
> Status in “python-apt” package in Ubuntu:
> Confirmed
> Status in “ubiquity” package in Ubuntu:
> Fix Released
> Status in “python-apt” source package in Raring:
> Confirmed
> Status in “ubiquity” source package in Raring:
> Fix Released
>
> Bug description:
> After Ubiquity finishes copying files, it creates the users on the
> system (Or a bit later), then crashes.
> I used Grub2 on my USB drive to boot the raring ISO via loopback.
> Installed with a custom partition layout (Reused old Home and formated
> Root).
>
> ProblemType: Bug
> DistroRelease: Ubuntu 13.04
> Package: ubiquity 2.13.12
> ProcVersionSignature: Ubuntu 3.8.0-7.15-generic 3.8.0
> Uname: Linux 3.8.0-7-generic x86_64
> ApportVersion: 2.8-0ubuntu4
> Architecture: amd64
> CasperVersion: 1.330
> Date: Mon Feb 25 20:08:39 2013
> LiveMediaBuild: Ubuntu 13.04 "Raring Ringtail" - Alpha amd64 (20130225)
> MarkForUpload: True
> ProcEnviron:
> TERM=unknown
> PATH=(custom, no user)
> XDG_RUNTIME_DIR=<set>
> LANG=en_US.UTF-8
> SHELL=/bin/bash
> SourcePackage: ubiquity
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/python-apt/+bug/1132918/+subscriptions
>

[Colin Watson (cjwatson)]

I think you miss that the syslog in this bug shows a version of Ubiquity that postdates its switch to Python 3.

[Colin Watson (cjwatson)]

You can reproduce this exception in at least some non-Unicode locales, although I'm not entirely sure why ubiquity would have been in such a locale or why it would have been attempting to render such a large value; this may not have been the original case, but it'll do for a reproducer. The et_EE locale must be generated before running this.

  $ LC_ALL=et_EE python3
  Python 3.3.2+ (default, Oct 9 2013, 14:50:09)
  [GCC 4.8.1] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import locale
  >>> import apt_pkg
  >>> locale.setlocale(locale.LC_ALL, '')
  'et_EE'
  >>> apt_pkg.size_to_str(10000000000000000000000000000)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 2: invalid start byte

[Rolf Leggewie (r0lf)]

raring has seen the end of its life and is no longer receiving any updates. Marking the raring task for this ticket as "Won't Fix".

[Julian Andres Klode (juliank)]

We could fix it in that direction due to PyUnicode_DecodeLocale, but there's nothing we can do for encoding Python strings to the system's locale without a memory leak, so it does not make a lot of sense to support only one direction.

[Julian Andres Klode (juliank)]

Python provides PyUnicode_EncodeLocale(), but that returns a new bytes object. We can get a char* from that, but we'd not be able to delete the bytes object anymore, as we don't know the life time of the char* and our function only returns a char*