can't connect to networks with non-unicode essids

[Bogdan Butnaru]

Binary package hint: network-manager
(Hardy, up-to-date at the moment, last week of Mars 2008.)

Hello!

I've been traveling last week and I stayed at a hotel that had WiFi. I noticed a weird problem. The access points had ESSIDs containing French diacritics—"Hôtel de la Gare 1", etc. (it was a French hotel)—and I couldn't connect to them using nm-applet.

In fact, nm-applet didn't display those ESSIDs at all. The problem might be with wireless-tools or with the driver, though: I've had time to do a bit of looking around. I used "iwlist scan" to check what was going on, and it turns out that it could see the access points, but the 'ô' character in the ESSID was displayed with an "unknown code-point" symbol. However, my terminal is correctly configured to UTF-8 (I can type the character, and 'cat'ing Unicode files with 'ô' works correctly), so I suspected some encoding problems.

I did some logging (I'll attach the files next), which included redirecting the output of the scan to a file. If I open the file with Geany it detects the encoding as ISO-8859-1. I imagine "iwlist" (or maybe the driver or some other component) doesn't do some charset conversion it should do at some point. (Note that Windows laptops in the same room could detect, display and connect to the APs.)

I could connect to the network by using "iwconfig ap [address] && iwconfig essid any", so the APs were probably working correctly.

This should be easy to test by setting the ESSID of your access-point to "Hôtel" (encoded in various charsets, at least ISO-8859-1). Note that the first file I attach was obtained by simply redirecting output of commands while I was typing. So it should contain exactly the bytes outputed by the applications.

(There are two attempts at connection in the log, the first time I mistakenly connected to another AP I didn't have access to. There's some screwing around with dhclient, too.)

[Bogdan Butnaru (bogdanb)]

If it's not obvious from the lshw output, I have a Dell Latitude D620 laptop, with an Intel wifi card, using the iwl3945 driver. I don't know which package is responsible for the bug, I just linked everything I thought might be related.

Note that I can't perfectly reproduce this, obviously, as I'm not staying at that hotel anymore. It was perfectly deterministic there, of course.

I imagine it can be easily tested by changing an access-point's ESSID to the one described above (try some others, too), but it needs to be done by a developer to make sure the encoding is the same as that above. (They might want to try a few others for good measure.)

[Leann Ogasawara (leannogasawara)]

Hi Bogdan,

Care to confirm this is still an issue with the latest Alpha for the upcoming Intrepid Ibex 8.10. It is based on a 2.6.26 kernel and should have updated version of the iwl3945 driver. You should be able to test via a LiveCD - http://www.ubuntu.com/testing. Please let us know your results.

Also, beginning with the Intrepid Ibex 8.10 development cycle the linux-ubuntu-modules package was actually merged with the linux kernel package. Going forward, bugs that would have been reported against linux-ubuntu-modules should not just be reported against linux. I'm going to automatically open the 'linux' task for this bug to make sure we carry it forward. However, against linux-ubuntu-modules-2.6.24 this will be closed as this does not qualify for a Stable Release Update - http://wiki.ubuntu.com/StableReleaseUpdates . Thanks

[Leann Ogasawara (leannogasawara)]

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.

[Alexander Sack (asac)]

i doubt that this is a NM issue. invalidating network manager task.

[Bogdan Butnaru (bogdanb)]

Hello again.

I've been doing a bit more testing. I have a SpeedTouch ST706 router/ADSL modem from my ISP and I've tried entering accented letters into its ESSID with only partial success. (The router's web interface itself handles them inconsistently.)

I've tried é, è and ô, but the router converts them randomly to ASCII letters, so I can't test how the network manager would deal with them.

I've also tried others including ș, ț. At one point the router interpreted these as something weird; the terminal, upon when calling "iwlist scan", showed "ș*țnng", with a strange sign instead of the asterisk (pasting in Firefox leads to a "hidden" character, i.e. it's not displayed in this form but when moving the cursor over it with the arrows there's a pause there. I'll paste it here to see what launchpad makes of it: "ș
țnng") On the other hand, the network manager applet prints the character as [001C]-in-a-box. Again something "interesting" must be happening with charset conversions.

On the one hand, network manager successfully connected itself to these networks.

On the other hand, since I can't set arbitrary byte strings for my AP's ESSID, I have no idea if it works in general (especially for byte strings that are certainly not valid for the encoding used on the host computer, in my case UTF-8).

If anyone has an access point that can set arbitrary byte strings for the ESSID, they should try several random values, including some that contain invalid Unicode characters (null bytes, illegal byte sequences, etc.)

Unless someone triggers a bug that way (or I encounter another weird AP), my bug is considered solved.

[Steve Langasek (vorlon)]

Alexander,

Why do you say that this is not a network-manager issue? essids are just raw values with no concept of encoding, so it doesn't make sense for (e.g.) wireless-tools to try to do a charset conversion on them. Instead, tools that expect UTF8 input (which is likely the case for network-manager, since GNOME libs are heavily UTF8-oriented) should take care to convert non-UTF8 bytes to some representable form (without assuming anything about the encoding).

I think the network-manager task should be reopened here...

[Andreas Gnau (rondom)]

This is definitely also a network-manager bug. Look at the screenshot, there's no sense in displaying HTML-entities in a menu!

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 210484

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Mathieu Trudel-Lapierre (cyphermox)]

This bug is almost certainly not a kernel issue. NetworkManager has grown support for complex ESSIDs by saving them in a string of bytes; later than the release for which this was reported.

I think the changes are likely in Lucid and later release. Regardless, I'll consider this closed: please file a new bug report for the release on which you can observe this issue, making sure that bug is reported via 'ubuntu-bug network-manager' to ensure we have as much information as possible to debug the issue. Then, just comment back here with the bug number so that developers can reach it :)

Thanks!