unattended-upgrade --help raises UnicodeEncodeError when stdout encoding is ascii

[errors.ubuntu.com bug bridge]

[Impact]

 * unattended-upgrades --help crashes in apt.systemd.daily script when locale is not in UTF-8.

[Test Case]

In the fixed case no crash is observed.

rbalint@yogi:~$ lxc launch ubuntu:18.04 bb-lp-1806076
Creating bb-lp-1806076
Starting bb-lp-1806076
rbalint@yogi:~$ lxc shell bb-lp-1806076
mesg: ttyname failed: No such device
root@bb-lp-1806076:~# apt install -yqq language-pack-ru-base
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  language-pack-ru
The following NEW packages will be installed:
  language-pack-ru language-pack-ru-base
0 upgraded, 2 newly installed, 0 to remove and 7 not upgraded.
Need to get 2310 kB of archives.
After this operation, 11.8 MB of additional disk space will be used.
Selecting previously unselected package language-pack-ru-base.
(Reading database ... 28536 files and directories currently installed.)
Preparing to unpack .../language-pack-ru-base_1%3a18.04+20180712_all.deb ...
Unpacking language-pack-ru-base (1:18.04+20180712) ...
Selecting previously unselected package language-pack-ru.
Preparing to unpack .../language-pack-ru_1%3a18.04+20180712_all.deb ...
Unpacking language-pack-ru (1:18.04+20180712) ...
Setting up language-pack-ru (1:18.04+20180712) ...
Setting up language-pack-ru-base (1:18.04+20180712) ...
Generating locales (this might take a while)...
  ru_RU.UTF-8... done
  ru_UA.UTF-8... done
Generation complete.

root@bb-lp-1806076:~# echo LANG=ru_RU | tee /etc/default/locale
LANG=ru_RU
root@bb-lp-1806076:~#
/usr/lib/apt/apt.systemd.daily update
Traceback (most recent call last):
  File "/usr/bin/unattended-upgrade", line 2171, in <module>
    (options, args) = parser.parse_args() # type: ignore
  File "/usr/lib/python3.6/optparse.py", line 1387, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "/usr/lib/python3.6/optparse.py", line 1427, in _process_args
    self._process_long_opt(rargs, values)
  File "/usr/lib/python3.6/optparse.py", line 1501, in _process_long_opt
    option.process(opt, value, values, self)
  File "/usr/lib/python3.6/optparse.py", line 785, in process
    self.action, self.dest, opt, value, values, parser)
  File "/usr/lib/python3.6/optparse.py", line 807, in take_action
    parser.print_help()
  File "/usr/lib/python3.6/optparse.py", line 1647, in print_help
    file.write(self.format_help())
UnicodeEncodeError: 'ascii' codec can't encode characters in position 126-133: ordinal not in range(128)

root@bb-lp-1806076:~# echo LANG=ru_RU.UTF-8 | tee /etc/default/locale
LANG=ru_RU.UTF-8
root@bb-lp-1806076:~# /usr/lib/apt/apt.systemd.daily update
root@bb-lp-1806076:~#

[Where problems could occur]

* Nowhere, really. The fix is setting LC_ALL=C.UTF-8 for u-u --help and it is processed by grep then.

[Original Bug Text]

The Ubuntu Error Tracker has been receiving reports about a problem regarding unattended-upgrades. This problem was most recently seen with package version 1.1ubuntu1.18.04.6, the problem page at https://errors.ubuntu.com/problem/b3e3265e302351558260f54ae37c7b4c193dfc95 contains more details, including versions of packages affected, stacktrace or traceback, and individual crash reports.
If you do not have access to the Ubuntu Error Tracker and are a software developer, you can request it at http://forms.canonical.com/reports/.

Also seen in:
 * https://errors.ubuntu.com/problem/936bb1c75c4efe018f968a5773b820bcf52c298a

[Balint Reczey (rbalint)]

Unattended-upgrades does not print to stdout by default thus it can operate properly in the environments triggering the bug, but apt.systemd.daily executes u-u --help that may crash.

To avoid the crash apt could set LC_ALL=C.UTF8 when running unattended-upgrades --help as proposed in the MR below:

https://salsa.debian.org/apt-team/apt/merge_requests/34

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.8.0~alpha3

---------------
apt (1.8.0~alpha3) unstable; urgency=medium

  [ Manuel A. Fernandez Montecelo ]
  * Remove "Ultimedia" config for Acquire::Changelogs
  * Remove "Tanglu" config for Acquire::Changelogs and as vendor

  [ Julian Andres Klode ]
  * Adjust libapt-pkg Breaks aptitude to << 0.8.9
  * Print useful error on "apt changelog" without arguments
  * test-pdiff-usage: make transaction failure test case more robust
  * Provide a "autopurge" shortcut.
    Thanks to Michael Vogt for the initial work
  * Override FileFd copy constructor to prevent copying
  * Add support for /etc/apt/auth.conf.d/*.conf (netrcparts)
  * Set PATH=/usr/sbin:/usr/bin:/sbin:/bin when running dpkg

  [ David Kalnischkies ]
  * Allow setting Referer header for http method
  * Use quoted tagnames in config dumps

  [ Алексей Шилин ]
  * Russian program translation update (Closes: #914096)

  [ Balint Reczey ]
  * Set LC_ALL=C.UTF-8 for unattended-upgrades environment when parsing its --help
    (LP: #1806076)

  [ Milo Casagrande ]
  * [l10n] Update Italian translation

 -- Julian Andres Klode <email address hidden> Tue, 18 Dec 2018 15:02:11 +0100

[Łukasz Zemczak (sil2100)]

Hello errors.ubuntu.com, or anyone else affected,

Accepted apt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.6.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apt (1.6.13) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

apport/2.20.9-0ubuntu7.23 (amd64, i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Julian Andres Klode (juliank)]

I installed 1.6.13 and could verify that the regression wrt --help is fixed. unattended-upgrades still logged a lot of errors because for some reason gettext returned utf8 despite being on non-UTF-8 locale, but that's all from the APT side.

[Julian Andres Klode (juliank)]

Log:

+ /usr/lib/apt/apt.systemd.daily lock_is_held update
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.6/logging/__init__.py", line 996, in emit
    stream.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 29-34: ordinal not in range(128)
Call stack:
  File "/usr/bin/unattended-upgrade", line 2193, in <module>
    sys.exit(main(options))
  File "/usr/bin/unattended-upgrade", line 1750, in main
    " ".join(blacklisted_pkgs))
Message: '\u041f\u0430\u043a\u0435\u0442\u044b \u0432 \u044...

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for apt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.6.13

---------------
apt (1.6.13) bionic; urgency=medium

  [ David Kalnischkies ]
  * Fix incorrect base64 encoding due to int promotion (LP: #1916050)
  * Harden test for no new acquires after transaction abort (Closes: #984966)
    (LP: #1918920)

  [ Julian Andres Klode ]
  * Implement update --error-on=any (Closes: #594813) (LP: #1693900)
  * Include all translations when building the cache (LP: #1907850)
  * Add basic support for the Protected field
  * Do not require force-loopbreak on Important packages
    (Closes: #983014) (LP: #1916725)
  * Protect currently running kernel at run-time (LP: #1615381)
  * Make ADDARG{,C}() macros expand to single statements
  * Improve immediate configuration handling (LP: #1871268)
    - Do not immediately configure m-a: same packages in lockstep
    - Ignore failures from immediate configuration. This does not change the
      actual installation ordering - we never passed the return code to the
      caller and installation went underway anyway if it could be ordered at a
      later stage, this just removes spurious after-the-fact errors.
      (Closes: #973305, #188161, #211075, #649588)
  * Default Acquire::AllowReleaseInfoChange::Suite to "true" (Closes: #931566)
    (LP: #1918907)

  [ Balint Reczey ]
  * Set LC_ALL=C.UTF-8 for unattended-upgrades environment when parsing its --help
    (LP: #1806076)

 -- Julian Andres Klode <email address hidden> Fri, 12 Mar 2021 14:09:15 +0100