yakkety desktop - non-english installation crashes with /plugininstall.py: ValueError: invalid literal for int() with base 10: ''

[Victor gonzalez]

Enviroment: Ubuntu 16.10.5 Yakkety Yak http://cdimage.ubuntu.com/daily-live/current/

HW: Toshiba portege Z30-A12T | Intel i5+intel graphics | SSD hd

Preconditions: W10+Xenial xerus installed, internet connection, 3rd party sw and updates selected

[Impact]
Breaks installation in some non-English locales.

[Test case]
Steps to reproduce:

1º Click something else on Installation type screen>select /dev/sdaX where Xenial is installed, use it to mount / and leave some space to create the /home partition(ext4)
2º Then click back and select erase ubuntu 16.04.1 and install Yakkety
3º Installation starts

Current result: After a little time installing it crashes and a window shows up to let the user know that some info is being gathered to send it to developers, and finally firefox opens on the bug report web on ubiquity package

Expected result: No crashes during installation

[Regression potential]
Very low, we're running this in 1.3 since a long time. The fix depends on another one that restricts the Date parser to accept only UTC time zones in Release files, though (we do not understand the other ones anyway), and thus might reject some repositories it silently accepted in a wrong way previously (we have not seen anyone report a regression yet since uploading that in June).

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: ubiquity 16.10.5
ProcVersionSignature: Ubuntu 4.4.0-33.52-generic 4.4.15
Uname: Linux 4.4.0-33-generic x86_64
ApportVersion: 2.20.3-0ubuntu5
Architecture: amd64
CasperVersion: 1.376
Date: Mon Aug 8 15:50:56 2016
InstallCmdLine: file=/cdrom/preseed/ubuntu.seed boot=casper initrd=/casper/initrd.lz quiet splash --- maybe-ubiquity
LiveMediaBuild: Ubuntu 16.10 "Yakkety Yak" - Alpha amd64 (20160808)
ProcEnviron:
 LANGUAGE=es_ES.UTF-8
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=es_ES.UTF-8
SourcePackage: ubiquity
UpgradeStatus: No upgrade log present (probably fresh install)

[Jean-Baptiste Lallement (jibel)]

From syslog there are several internationalization errors:
Aug 8 13:49:24 ubuntu in-target: Argument "0,0000" isn't numeric in multiplication (*) at /usr/bin/debconf-apt-progress line 168, <STDIN> line 2.

Aug 8 13:49:44 ubuntu /plugininstall.py: Downloads verified successfully
Aug 8 13:49:44 ubuntu /plugininstall.py: Traceback (most recent call last):
Aug 8 13:49:44 ubuntu /plugininstall.py: File "/usr/lib/ubiquity/ubiquity/install_misc.py", line 362, in run
Aug 8 13:49:44 ubuntu /plugininstall.py: self.update_interface()
Aug 8 13:49:44 ubuntu /plugininstall.py: File "/usr/lib/python3/dist-packages/apt/progress/base.py", line 255, in update_interface
Aug 8 13:49:44 ubuntu /plugininstall.py: if float(percent) != self.percent or status_str != self.status:
Aug 8 13:49:44 ubuntu /plugininstall.py: ValueError: could not convert string to float: '0,0000'

and this final error
Aug 8 13:49:47 ubuntu /plugininstall.py: Traceback (most recent call last):
Aug 8 13:49:47 ubuntu /plugininstall.py: File "/usr/share/ubiquity/plugininstall.py", line 1778, in <module>
Aug 8 13:49:47 ubuntu /plugininstall.py: install.run()
Aug 8 13:49:47 ubuntu /plugininstall.py: File "/usr/share/ubiquity/plugininstall.py", line 78, in wrapper
Aug 8 13:49:47 ubuntu /plugininstall.py: func(self)
Aug 8 13:49:47 ubuntu /plugininstall.py: File "/usr/share/ubiquity/plugininstall.py", line 212, in run
Aug 8 13:49:47 ubuntu /plugininstall.py: self.install_language_packs()
Aug 8 13:49:47 ubuntu /plugininstall.py: File "/usr/share/ubiquity/plugininstall.py", line 690, in install_language_packs
Aug 8 13:49:47 ubuntu /plugininstall.py: self.do_install(self.langpacks, langpacks=True)
Aug 8 13:49:47 ubuntu /plugininstall.py: File "/usr/lib/ubiquity/ubiquity/install_misc.py", line 964, in do_install
Aug 8 13:49:47 ubuntu /plugininstall.py: installprogress.finish_update()
Aug 8 13:49:47 ubuntu /plugininstall.py: File "/usr/lib/ubiquity/ubiquity/install_misc.py", line 439, in finish_update
Aug 8 13:49:47 ubuntu /plugininstall.py: self.db.progress('STOP')
Aug 8 13:49:47 ubuntu /plugininstall.py: File "/usr/lib/python3/dist-packages/debconf.py", line 62, in <lambda>
Aug 8 13:49:47 ubuntu /plugininstall.py: lambda *args, **kw: self.command(command, *args, **kw))
Aug 8 13:49:47 ubuntu /plugininstall.py: File "/usr/lib/python3/dist-packages/debconf.py", line 83, in command
Aug 8 13:49:47 ubuntu /plugininstall.py: status = int(status)
Aug 8 13:49:47 ubuntu /plugininstall.py: ValueError: invalid literal for int() with base 10: ''

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jean-Baptiste Lallement (jibel)]

Confirmed in French

[Mathieu Trudel-Lapierre (cyphermox)]

Things aren't simple for this issue.

Looks like when apt prints out its StatusFd messages; it does so (quite rightly) following LC_NUMERIC when doing a straight sprintf(..., "...%.4f...", ...). Many other calls are wrapping decimal values in a call to to_string() to get around this.

The problem with this is that some other tools (debconf's debconf-apt-progress, python-apt) rely on the output being in C.UTF-8 (or some other similar format) to parse the output, and won't be able to parse output where the decimal separator is anything other than '.'. This happens in fr_FR.UTF-8 and es_ES.UTF-8 at least.

I can work around this in ubiquity by setting LC_NUMERIC; but it won't fix the underlying cause being that apt prints out the values that may or may not be parseable by other tools. Apt should be fixed (this appears to be a regression from xenial) or we'll need to modify debconf-apt-progress and python-apt to account for the changes.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1611010

[David Kalnischkies (donkult)]

https://anonscm.debian.org/git/apt/apt.git/commit/?id=0919f1df552ddf022ce4508cbf40e04eae5ef896

[sudodus (nio-wiklund)]

I don't think bug #1616183 is a duplicate of this bug, because it affects the default language too, US English, which I found a few minutes ago while isotesting Lubuntu desktop i386 beta1.

[moma (osmoma)]

Re-hi,
I may agree with @sudodus,

I cannot remember to have changed language or keyboard during the
installation.
I kept all in default: Language = 'en'

My initial bug was #1612570.

On Tue, Aug 23, 2016 at 9:14 PM, sudodus <email address hidden> wrote:

> I don't think bug #1616183 is a duplicate of this bug, because it
> affects the default language too, US English, which I found a few
> minutes ago while isotesting Lubuntu desktop i386 beta1.
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1612570).
> https://bugs.launchpad.net/bugs/1611010
>
> Title:
> yakkety desktop - non-english installation crashes with
> /plugininstall.py: ValueError: invalid literal for int() with base 10:
> ''
>
> Status in apt package in Ubuntu:
> Fix Committed
> Status in ubiquity package in Ubuntu:
> Fix Committed
>
> Bug description:
> Enviroment: Ubuntu 16.10.5 Yakkety Yak http://cdimage.ubuntu.com
> /daily-live/current/
>
> HW: Toshiba portege Z30-A12T | Intel i5+intel graphics | SSD hd
>
> Preconditions: W10+Xenial xerus installed, internet connection, 3rd
> party sw and updates selected
>
> Steps to reproduce:
>
> 1º Click something else on Installation type screen>select /dev/sdaX
> where Xenial is installed, use it to mount / and leave some space to create
> the /home partition(ext4)
> 2º Then click back and select erase ubuntu 16.04.1 and install Yakkety
> 3º Installation starts
>
> Current result: After a little time installing it crashes and a window
> shows up to let the user know that some info is being gathered to send
> it to developers, and finally firefox opens on the bug report web on
> ubiquity package
>
> Expected result: No crashes during installation
>
> ProblemType: Bug
> DistroRelease: Ubuntu 16.10
> Package: ubiquity 16.10.5
> ProcVersionSignature: Ubuntu 4.4.0-33.52-generic 4.4.15
> Uname: Linux 4.4.0-33-generic x86_64
> ApportVersion: 2.20.3-0ubuntu5
> Architecture: amd64
> CasperVersion: 1.376
> Date: Mon Aug 8 15:50:56 2016
> InstallCmdLine: file=/cdrom/preseed/ubuntu.seed boot=casper
> initrd=/casper/initrd.lz quiet splash --- maybe-ubiquity
> LiveMediaBuild: Ubuntu 16.10 "Yakkety Yak" - Alpha amd64 (20160808)
> ProcEnviron:
> LANGUAGE=es_ES.UTF-8
> PATH=(custom, no user)
> XDG_RUNTIME_DIR=<set>
> LANG=es_ES.UTF-8
> SourcePackage: ubiquity
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1611010/+subscriptions
>

--
Sent from my PC, laptop or phone with Ubuntu-Linux.

[Adam Dingle (adam-yorba)]

I believe this bug occurs in any location whose default language is non-English, even if you select English as your language. I saw it while trying to install Yakkety in Spain, even though I selected English for the language and keyboard. As a workaround, I ran the installer again and chose New York rather than Madrid when asked for my time zone. Then it worked fine.

[sudodus (nio-wiklund)]

I'm testing when re-located to the New York time zone, and still used Swedish language ... it works in amd64, and it works in i386. So I can confirm your findings, adam-yorba.

[Julian Andres Klode (juliank)]

Going to upload a fixed APT shortly. There's also some other broken stuff in installation ordering, and missing large file support that should really be fixed now.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubiquity - 16.10.8

---------------
ubiquity (16.10.8) yakkety; urgency=medium

  [ Mathieu Trudel-Lapierre ]
  * Automatic update of included source packages: choose-mirror
    2.69ubuntu1, grub-installer 1.128ubuntu6, netcfg 1.138ubuntu1,
    partman-partitioning 112ubuntu1.
  * bin/ubiquity: Work around apt printing pmstatus messages progress percent
    values following LC_NUMERIC. (LP: #1611010)

  [ Martin Pitt ]
  * scripts/plugininstall.py: Remove fallback creation of /etc/iftab. This
    hasn't existed for a long time (pre-udev).
  * scripts/plugininstall.py: Remove fallback creation of
    /etc/network/interfaces. The ifupdown package already creates one, it
    is not necessary to have a (quasi-)empty file, and desktop systems use
    NetworkManager instead of ifupdown.

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 23 Aug 2016 17:11:45 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.3~rc2ubuntu3

---------------
apt (1.3~rc2ubuntu3) yakkety; urgency=medium

  [ David Kalnischkies ]
  * do dpkg --configure before --remove/--purge --pending (Closes: 835094)
  * prevent C++ locale number formatting in text APIs (try 3) (LP: #1611010)

  [ Julian Andres Klode ]
  * CMake: Add Large File Support (Closes: #834767)
  * prepare-release: Use equivs and gdebi-core for travis deps

 -- Julian Andres Klode <email address hidden> Wed, 24 Aug 2016 08:39:04 +0000

[Nairwolf (nairwolf)]

Thank you for your bug fix ! I've tested yesterday with the daily build of Xubuntu 08/25. Ubiquity was updated to 16.10.8 but I had to update apt to 1.3~rc2ubuntu3.

But I can confirm it works now, with a french keyboard ;)

[Phillip Susi (psusi)]

Apparently this is now also happening in 16.04.1.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dean (pantsdownjedi)]

Problem installing Xenial most recent daily desktop AMD64 builds (3 previous days) while selecting English keyboard and Ecuador timezone.

Same problem of crash during setting language.

Workaround was updating sources.list, update, reinstall apt tzdata language-selector-common language-pack-en-base language-pack-en (everything was up to date so I doubt this did anything) followed by running ubiquity and selecting a US location to match the keyboard.

This went fine after various failed attempts and ubiquity previously crashing every time on setting language. Now location merely needs to be updated on reboot, if it isn't broken.
Perhaps doing this after all other things in the installer...

[Julian Andres Klode (juliank)]

I already started cherry picking all fixes for these kind of bugs to xenial's 1.2 branch last week

I'll try to get it finished this week. Then I'll test it for a few days (via apt 1.2 PPA), so it might take 1 or 2 weeks for the fix to reach proposed (we should get 1.2.15 into updates by then).

[Julian Andres Klode (juliank)]

apt 1.2.16~rc1 just hit the PPA (might take some time for processing, though):

  https://launchpad.net/~deity/+archive/ubuntu/apt-1.2

and should fix this bug. It would be great to have some confirmation of that before uploading that, otherwise I'll go ahead with an SRU tomorrow probably if I don't notice any regression on my system.

[Julian Andres Klode (juliank)]

Uploaded 1.2.16. We should really have a SRU [Test case] thingy, though.

[Phillip Susi (psusi)]

It looks like this has regressed on 17.10, see bug #1638142.

[Phillip Susi (psusi)]

And bug #1637756.

[Julian Andres Klode (juliank)]

These are different bugs.Or rather, we definitely fixed the

if float(percent) != self.percent or status_str != self.status:
Aug 8 13:49:44 ubuntu /plugininstall.py: ValueError: could not convert string to float: '0,0000'

thing here. And to keep our sanity (especially WRT to the SRU fixing that), we do need this bug closed, otherwise everything becomes a mess.

[Julian Andres Klode (juliank)]

Not sure about the ubiquity task, though, I unmerged the other bug reports (which actually are 2 distinct ones), but did not mark it as fixed again.

[Olnei Araujo (olneiaa)]

Thank you! I will not do it.
🙏

Olnei A Araujo
Comunidade ubuntu-br
http://wiki.ubuntu-br/InstalacaoUbuntu
http://ubuntu.com
31-987-120-747
twitter:@olneiaa
linux user # 484050
ubuntu user # 26374-2007

2016-11-30 13:00 GMT-02:00 Julian Andres Klode <email address hidden>:

> Not sure about the ubiquity task, though, I unmerged the other bug
> reports (which actually are 2 distinct ones), but did not mark it as
> fixed again.
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1615273).
> https://bugs.launchpad.net/bugs/1611010
>
> Title:
> yakkety desktop - non-english installation crashes with
> /plugininstall.py: ValueError: invalid literal for int() with base 10:
> ''
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1611010/+subscriptions
>

[Phillip Susi (psusi)]

And yet the end result is still the same:

/plugininstall.py: ValueError: invalid literal for int() with base 10: ''

Are you saying there is a different cause for this now in 17.10?

[Julian Andres Klode (juliank)]

Likely. See, the whole thing happens when the call stack is unwound in response to the first error here. In the other cases, the status update happens elsewhere.

So, what happened here is that we fixed the first error on the apt side, and used this bug to track that (and marked it as such in the SRU). The other bug reports are about ubiquity only.

So what really happened it seems is that:

- the apt side of this bug is closed
- the real bug in ubiquity is not closed, which is a
  bug in the debconf status reporting parsing thing
  - the "fix" just added code to work around the apt issue.

So, let's keep the apt task to track the apt side (thus Fix Released and In Progress), and keep the ubiquity task open to fix that properly there. This way we can still sanely track the bug fixes and SRUs, even if it is a bit weird.

[Phillip Susi (psusi)]

Sounds good.

[Adam Conrad (adconrad)]

Hello Victor, or anyone else affected,

Accepted apt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.2.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Sergey Fedotov (sr-fido)]

Adam, I can confirm, bug fix is worked for me, while Russian keyboard and timezone was selected.

dist: xenial
apt: 1.2.18/xenial-proposed
ubiquity: 2.21.63.3

[Julian Andres Klode (juliank)]

(tag change as per previous comment)

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.2.18

---------------
apt (1.2.18) xenial; urgency=high

  * SECURITY UPDATE: gpgv: Check for errors when splitting files (CVE-2016-1252)
    Thanks to Jann Horn, Google Project Zero for reporting the issue
    (LP: #1647467)
  * gpgv: Flush the files before checking for errors

apt (1.2.17) xenial; urgency=medium

  [ David Kalnischkies ]
  * apt-key: warn instead of fail on unreadable keyrings (LP: #1642386)
  * show apt-key warnings in apt update (Closes: 834973)

  [ Julian Andres Klode ]
  * test-releasefile-verification: installaptold: Clean up before run

apt (1.2.16) xenial; urgency=medium

  [ David Kalnischkies ]
  * avoid changing the global LC_TIME for Release writing
  * use de-localed std::put_time instead rolling our own
  * accept only the expected UTC timezones in date parsing (Closes: 819697)
  * avoid std::get_time usage to sidestep libstdc++6 bug (LP: #1593583)
  * imbue datetime parsing with C.UTF-8 locale (Closes: 828011)
  * prevent C++ locale number formatting in text APIs (try 2) (Closes: 832044)
  * prevent C++ locale number formatting in text APIs (try 3) (LP: #1611010)
    (LP: #1592817)
  * imbue .diff/Index parsing with C.UTF-8 as well

  [ Julian Andres Klode ]
  * Use C locale instead of C.UTF-8 for protocol strings
  * Add shippable.yml for CI on Shippable
  * Revert "if the FileFd failed already following calls should fail, too"
    (LP: #1641905)

 -- Julian Andres Klode <email address hidden> Thu, 08 Dec 2016 15:28:08 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for apt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.