22.04 idn -a domain.com idn: could not convert from ASCII to UTF-8

[Jaap Marcus]

[ Impact ]

idn cannot handle non-ASCII characters

[ Test Plan ]

Existing state in 22.04:

$ echo ü | idn
idn: could not convert from ASCII to UTF-8

Fixed output:

$ echo ü | idn
xn--tda

[ Where problems could occur ]

The patch enables an extra autotools features, so the most likely problem is a build failure. Currently non-ASCII characters are entirely unhandled, so there's no risk of regression there, but this could potentially turn clean failures into crashes or incorrect output.

[Original report]

On Ubuntu 22.04 Jammy:

idn -a "xxx-tést.eu"
idn: could not convert from ASCII to UTF-8

And

idn -u "xn--xxx-tst-fya.eu"
idn: could not convert from UTF-8 to ASCII

Expected:
Successfully conversion from ASCII to UTF-8

Ubuntu 20.04 / 18.04 it works fine

Also with idn2 from UTF-8 to ASCII works as expected how ever support lacks from ASCII to UTF-8

Description: Ubuntu 22.04 LTS
Release: 22.04

root@test:~# echo $LC_ALL
C.UTF-8

root@test:~# apt-cache policy idn
idn:
  Installed: 1.38-4build1
  Candidate: 1.38-4build1
  Version table:
 *** 1.38-4build1 500
        500 https://mirror.hetzner.com/ubuntu/packages jammy/universe amd64 Packages
        100 /var/lib/dpkg/status

[Jaap Marcus (jaapmarcus)]

For conversion backwards use idn2 -d domain.com

How ever it doesn't stop why it doesn't work for idn package it self

[Simon Josefsson (simon-josefsson)]

It is an upstream bug, see:
https://lists.gnu.org/archive/html/help-libidn/2022-06/msg00000.html

Patch here:
https://git.savannah.gnu.org/gitweb/?p=libidn.git;a=blobdiff;f=configure.ac;h=14a39972b59ff1d2f223a2fbd03db0d872b85826;hp=49c30f6e486f3430188d4b52601c861faad2e70b;hb=8c0f2aca9e026bdd846d930fc1bed0866c7383f2;hpb=3d3818eb5f6e0712595ba32d74f620d1ce0450e3

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steve Langasek (vorlon)]

Hello Jaap, or anyone else affected,

Accepted libidn into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libidn/1.38-4ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Nikita Radchenko (nradchenko)]

Hi,

I confirm that the patch fixes input encoding detection issue for me (22.04.2 LTS)

Current package:

free1 : ~ [0] # echo $LC_ALL
en_US.UTF-8
free1 : ~ [0] # dpkg -l idn libidn12
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=============================================================
ii idn 1.38-4build1 amd64 Command line interface to GNU Libidn
ii libidn12:amd64 1.38-4build1 amd64 GNU Libidn library, implementation of IETF IDN specifications
free1 : ~ [0] # idn xxx-tést.eu
idn: could not convert from ASCII to UTF-8
free1 : ~ [1] # idn -u xxx-tést.eu
idn: could not convert from ASCII to UTF-8
free1 : ~ [1] # idn -a xxx-tést.eu
idn: could not convert from ASCII to UTF-8
free1 : ~ [1] # echo ü | idn
idn: could not convert from ASCII to UTF-8
free1 : ~ [1] #

Package from jammy-proposed:

free1 : ~ [130] # echo $LC_ALL
en_US.UTF-8
free1 : ~ [0] # dpkg -l idn libidn12
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=============-============-=============================================================
ii idn 1.38-4build1 amd64 Command line interface to GNU Libidn
ii libidn12:amd64 1.38-4ubuntu1 amd64 GNU Libidn library, implementation of IETF IDN specifications
free1 : ~ [0] # idn xxx-tést.eu
xn--xxx-tst-fya.eu
free1 : ~ [0] # idn -u xxx-tést.eu
xxx-tést.eu
free1 : ~ [0] # idn -a xxx-tést.eu
xn--xxx-tst-fya.eu
free1 : ~ [0] # echo ü | idn
xn--tda
free1 : ~ [0] #

No problems after upgrade so far.

[Le hottest of doge (le-hottest-of-doge)]

Hello, patch build resolved my issue as well

juggler@circus:/tmp$ dpkg -l idn libidn12
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=============================================================
ii idn 1.38-4build1 amd64 Command line interface to GNU Libidn
ii libidn12:amd64 1.38-4build1 amd64 GNU Libidn library, implementation of IETF IDN specifications
juggler@circus:/tmp$ idn корюшка.любовь
idn: could not convert from ASCII to UTF-8

juggler@circus:/tmp$ sudo dpkg -i ~/Downloads/idn_1.38-4ubuntu1_amd64.deb
juggler@circus:/tmp$ sudo dpkg -i ~/Downloads/libidn12_1.38-4ubuntu1_amd64.deb

juggler@circus:/tmp$ dpkg -l idn libidn12
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=============-============-=============================================================
ii idn 1.38-4ubuntu1 amd64 Command line interface to GNU Libidn
ii libidn12:amd64 1.38-4ubuntu1 amd64 GNU Libidn library, implementation of IETF IDN specifications
juggler@circus:/tmp$ idn корюшка.любовь
xn--80atapk2c0b.xn--90aczn5ei

[Launchpad Janitor (janitor)]

This bug was fixed in the package libidn - 1.38-4ubuntu1

---------------
libidn (1.38-4ubuntu1) jammy; urgency=medium

  * debian/patches/initialise_langinfo_codeset.patch:
    - Fix handling of non-ASCII characters (LP: #1971715)

 -- Christopher James Halse Rogers <email address hidden> Tue, 01 Nov 2022 16:40:24 +0100

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for libidn has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.