Cubic - App fails with invalid text encoding

[mungewell]

Running on Xubuntu 18.04, running the cubic app fails at several points due to the encoding of messages... took me multiple cycles to edit them out sufficiently to complete the process of build a ISO (which actually for another reason).

Not sure what the intent of '\u2022' is... but it's not happy.

Shows as a 'bullet' here:
https://bazaar.launchpad.net/~cubic-wizard/cubic/release/view/head:/usr/share/cubic/utilities/logger.py#L66

Terminal has a default encoding of 'UTF-8'.
--
simon@thevoid:~/livecdtmp$ cubic

Cubic - Custom Ubuntu ISO Creator

Traceback (most recent call last):
  File "/usr/share/cubic/cubic.py", line 53, in <module>
    import navigator
  File "/usr/share/cubic/navigator.py", line 149, in <module>
    from utilities import displayer
  File "/usr/share/cubic/utilities/displayer.py", line 52, in <module>
    from utilities import model
  File "/usr/share/cubic/utilities/model.py", line 46, in <module>
    application.directory = None
  File "/usr/share/cubic/utilities/fields.py", line 71, in __setattr__
    logger.log_value('Set %s %s' % (self.name, key.replace('_', ' ')), value)
  File "/usr/share/cubic/utilities/logger.py", line 66, in log_value
    _log_value_hanging(column_a_text, column_b_text, column_a_initial_indent=' \u2022 ', column_a_subsequent_indent=' ')
  File "/usr/share/cubic/utilities/logger.py", line 232, in _log_value_hanging
    print('{}{}'.format(column_a_line, column_b_line))
UnicodeEncodeError: 'ascii' codec can't encode character '\u2022' in position 2: ordinal not in range(128)
--

--
simon@thevoid:~/livecdtmp$ dpkg --list | grep cubic
ii cubic 2020.10-35-release~202010230044~ubuntu18.04.1 all Wizard to create a customized Ubuntu Live CD image.

simon@thevoid:~/livecdtmp$ lsb_release -rd
Description: Ubuntu 18.04.3 LTS
Release: 18.04

[mungewell (simon-mungewell)]

Rendered as 'triangle' here:
https://bazaar.launchpad.net/~cubic-wizard/cubic/release/view/head:/usr/share/cubic/pages/prepare_page.py#L701
https://bazaar.launchpad.net/~cubic-wizard/cubic/release/view/head:/usr/share/cubic/pages/prepare_page.py#L1124

[Cubic PPA (cubic-wizard)]

I think you are on the right track checking your terminal's unicode encoding (https://bugs.launchpad.net/cubic/+bug/1905105/comments/1).

What is the output of the following commands? (Not all will return results.)

    (a)
        $ echo $LC_CTYPE

    (b)
        $ echo $LANG

    (b)
        $ locale

    (c)
        $ locale charmap

    (d)
        $ cat /etc/default/locale

    (e)
        $ cat /etc/rc.conf

    (f)
        $ cat /etc/locale.conf

    (g)
        $ grep "^[^#;]" /etc/locale.gen

    (h)
        $ dpkg -l locales

    (i)
        $ python3 --version

    (j)
        $ python3
        >>> import sys
        >>> sys.version_info
        >>> sys.stdout.encoding
        >>> exit()

[mungewell (simon-mungewell)]

simon@thevoid:~$ echo $LC_CTYPE

simon@thevoid:~$ echo $LANG
C
simon@thevoid:~$ locale
LANG=C
LANGUAGE=en_CA
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=
simon@thevoid:~$ locale charmap
ANSI_X3.4-1968
simon@thevoid:~$ cat /etc/default/locale
# File generated by update-locale
LANG="en_CA.UTF-8"
simon@thevoid:~$ cat /etc/rc.conf
cat: /etc/rc.conf: No such file or directory
simon@thevoid:~$ cat /etc/locale.conf
cat: /etc/locale.conf: No such file or directory
simon@thevoid:~$ grep "^[^#;]" /etc/locale.gen
en_CA.UTF-8 UTF-8
en_US.UTF-8 UTF-8
simon@thevoid:~$ dpkg -l locales
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii locales 2.27-3ubuntu all GNU C Library: National Language
simon@thevoid:~$ python3 --version
Python 3.6.9
simon@thevoid:~$ python3
Python 3.6.9 (default, Oct 8 2020, 12:12:24)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version_info
sys.version_info(major=3, minor=6, micro=9, releaselevel='final', serial=0)
>>> sys.stdout.encoding
'ANSI_X3.4-1968'
>>> exit()

[Cubic PPA (cubic-wizard)]

There are some obvious issues, but I think this can be remedied easily.

Here is an example of what the output should look like...

xubuntu@xubuntu:~$ echo $LC_CTYPE

xubuntu@xubuntu:~$ echo $LANG
en_CA.UTF-8
xubuntu@xubuntu:~$
xubuntu@xubuntu:~$ locale
LANG=en_CA.UTF-8
LANGUAGE=
LC_CTYPE="en_CA.UTF-8"
LC_NUMERIC="en_CA.UTF-8"
LC_TIME="en_CA.UTF-8"
LC_COLLATE="en_CA.UTF-8"
LC_MONETARY="en_CA.UTF-8"
LC_MESSAGES="en_CA.UTF-8"
LC_PAPER="en_CA.UTF-8"
LC_NAME="en_CA.UTF-8"
LC_ADDRESS="en_CA.UTF-8"
LC_TELEPHONE="en_CA.UTF-8"
LC_MEASUREMENT="en_CA.UTF-8"
LC_IDENTIFICATION="en_CA.UTF-8"
LC_ALL=
xubuntu@xubuntu:~$
xubuntu@xubuntu:~$ locale charmap
UTF-8
xubuntu@xubuntu:~$
xubuntu@xubuntu:~$ cat /etc/default/locale
LANG=en_CA.UTF-8
xubuntu@xubuntu:~$
xubuntu@xubuntu:~$ cat /etc/rc.conf
cat: /etc/rc.conf: No such file or directory
xubuntu@xubuntu:~$
xubuntu@xubuntu:~$ cat /etc/locale.conf
cat: /etc/locale.conf: No such file or directory
xubuntu@xubuntu:~$
xubuntu@xubuntu:~$ grep "^[^#;]" /etc/locale.gen
en_CA.UTF-8 UTF-8
xubuntu@xubuntu:~$
xubuntu@xubuntu:~$ dpkg -l locales
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=============================-===================-===================-===============================================================
ii locales 2.27-3ubuntu1.2 all GNU C Library: National Language (locale) data [support]
xubuntu@xubuntu:~$
xubuntu@xubuntu:~$ python3 --version
Python 3.6.9
xubuntu@xubuntu:~$ python3
Python 3.6.9 (default, Jul 17 2020, 12:50:27)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version_info
sys.version_info(major=3, minor=6, micro=9, releaselevel='final', serial=0)
>>> sys.stdout.encoding
'UTF-8'
>>> exit()

[Cubic PPA (cubic-wizard)]

Follow these instructions, and this should remedy your problem...

(1)

$ sudo dpkg-reconfigure locales

    Use <Page Up>, <Page Down>, <Up Arrow>, <Down Arrow> to navigate lists.
    Use <Space Bar> to select or unselect list items.
    Use <Tab> to switch between lists and buttons.
    Use <Left Arrow>, <Right Arrow> to switch between buttons.
    Use <Enter> to "Click" a selected Button.

(2)

On the 1st page "choose which locales to generate":

    Press the <Space Bar> to select "en_CA.UTF-8 UTF-8"
    Press the <Space Bar> to **UNSELECT** en_"CA ISO-8859-1"
    Continue to page through the list and ensure everything else is **Unselect**.

        [ ] en_CA ISO-8859-1
        [*] en_CA.UTF-8 UTF-8
        [ ] en_US.UTF-8 UTF-8

    (You may be able to select other locales, but do not select anything that doesn't say "UTF-8").

    Select "OK" to go to the next page.

(3)

On the 2nd page, "select the default language for the entire system":

    Scroll down and select "en_CA.UTF-8"

        en_CA.UTF-8

    (You can only select one default locale; make sure it says "UTF-8").

    Select "OK" to go to the next page.

(4)

You will see the following output:

    sxubuntu@xubuntu:~$ sudo dpkg-reconfigure locales
    Generating locales (this might take a while)...
    bn_BD.UTF-8... done
    .
    .
    .
    zh_SG.UTF-8... done
    Generation complete.

(5)

Reboot (or Logout / Login)

(6)

Reinstall Cubic (just to make sure it is a clean install).

    $ sudo apt autoremove --purge cubic
    $ sudo apt-add-repository --remove ppa:cubic-wizard/release
    $ sudo apt-add-repository --remove ppa:cubic-wizard/classic

    $ sudo apt-add-repository ppa:cubic-wizard/release
    $ sudo apt update
    $ sudo apt install cubic

[mungewell (simon-mungewell)]

I will give those a go, however I feel that it is appropriate for Cubic not to break if the user's system is configure as mine.

I would also maybe suggest trying with a Xubuntu LiveCD, because it might just be that _EVERY_ Xubuntu user has a system like this.

Cheers,
Simon.

[Cubic PPA (cubic-wizard)]

I tested using Xubuntu 18.04 Live ISO.
The default ISO is configured with en_US.UTF-8 UTF-8.

Generally, I test with various Ubuntu based distros (Xubuntu, Lubuntu, etc.) and they all come preset with UTF-8 "out of the box."

You has mentioned "I wouldn't consider it a 'clean install'", so perhaps something changed on your system after your initial install?

Hopefully this will resolve your problem.

[mungewell (simon-mungewell)]

Unfortunately I still get the same problem :-(
--
Traceback (most recent call last):
  File "/usr/share/cubic/cubic.py", line 53, in <module>
    import navigator
  File "/usr/share/cubic/navigator.py", line 149, in <module>
    from utilities import displayer
  File "/usr/share/cubic/utilities/displayer.py", line 52, in <module>
    from utilities import model
  File "/usr/share/cubic/utilities/model.py", line 46, in <module>
    application.directory = None
  File "/usr/share/cubic/utilities/fields.py", line 71, in __setattr__
    logger.log_value('Set %s %s' % (self.name, key.replace('_', ' ')), value)
  File "/usr/share/cubic/utilities/logger.py", line 66, in log_value
    _log_value_hanging(column_a_text, column_b_text, column_a_initial_indent=' \u2022 ', column_a_subsequent_indent=' ')
  File "/usr/share/cubic/utilities/logger.py", line 232, in _log_value_hanging
    print('{}{}'.format(column_a_line, column_b_line))
UnicodeEncodeError: 'ascii' codec can't encode character '\u2022' in position 2: ordinal not in range(128)
--

The other thought on my system, is that my '/home' is probably many distributions old, as I tend to re-install periodically.

[Cubic PPA (cubic-wizard)]

Do you think there is a locale setting in a file in your home folder?
Perhaps in your .profile or .bashrc files ?

[Cubic PPA (cubic-wizard)]

Re-execute the commands in comment #5, and share the output.
Let’s see if something has changed.
This may give a clue as to where the character encoding is coming from?

[Cubic PPA (cubic-wizard)]

I was able to simulate your issue on my system by explicitly setting the character encoding:

    $ PYTHONIOENCODING=ascii cubic

I got the same error as you did:

UnicodeEncodeError: 'ascii' codec can't encode character '\u2022' in position 2: ordinal not in range(128)

(1)
I wonder is a similar approach to explicitly set the encoding when running cubic would help with your situation?...

    $ PYTHONIOENCODING=utf8 cubic

(2)
To make testing your terminal easier, instead of running cubic each time, just cut and paste the following:

    $ echo -e "This is a UTF-8 bullet character: \\u2022"

It should output:

    "This is a UTF-8 bullet character: •"

(3)
Assuming the system defaults are set correctly...

    Can you create a new "terminal profile" in Xubuntu (like Gnome Terminal allows)?
    May be a "clean" terminal profile would help?

(4)
If the issue is due to your current logged-in environment...

    Could you create a new user on your system (with sudo privileges, of course)?
    May be a "clean" user without the legacy /home directory would help?

[mungewell (simon-mungewell)]

2.
simon@thevoid:~/livecdtmp$ echo -e "This is a UTF-8 bullet character: \\u2022"
This is a UTF-8 bullet character: \u2022
simon@thevoid:~/livecdtmp$
simon@thevoid:~/livecdtmp$ echo -e "This is a UTF-8 bullet character: \u2022"
This is a UTF-8 bullet character: \u2022

4.
"sudo adduser cubic"

I was able to build a custom ISO (will test shortly) when logged in as the new user.

So it does appear to be something in my particular user setting, perhaps a 'dot-file' somewhere. My wife has account on this machine since first install I'll try that one...

[Cubic PPA (cubic-wizard)]

Perhaps a quick search for files in your home directory may help pinpoint where the local is set...
$ cd ~
$ grep -I -r ^C$
$ grep -I -r ^en_CA$
$ grep -I -r '"en_CA"'

(I'll be closing this bug report shortly).

[Cubic PPA (cubic-wizard)]

If you encounter an error like:

    UnicodeEncodeError: 'ascii' codec can't encode character '\u2022'

This means you do not have UTF-8 support setup on your system.

Cubic requires UTF-8.