Calamares fails to parse GeoIP results correctly in JSON

[Chris Guiver]

STEPS TO REPRODUCE
 0. Ensure Internet is connected
 1. Boot Focal or Groovy Lubuntu or Groovy Ubuntu Studio
 2. Run the installer
 3. Advance to the time zone selection map

EXPECTED RESULTS
The actual time zone is selected.

ACTUAL RESULTS
New York is selected no matter what.

NOTES
A similar bug was fixed with 3.2.14-0ubuntu1
https://bugs.launchpad.net/ubuntu/+source/calamares/+bug/1845424

It seems that between then and Focal (3.2.20), there was another similar regression affecting JSON handling.

Log files show something like this:
virtual QString CalamaresUtils::GeoIP::GeoIPJSON::rawReply(const QByteArray&)
    WARNING:Invalid YAML data for GeoIPJSON
setGeoIP(Config*, const QVariantMap&)::<lambda()>
    GeoIP result for welcome= ""
void setCountry(Config*, const QString&, CalamaresUtils::GeoIP::Handler*)
    Unusable country code ""
.. Obtained from "https://ipapi.co/json" ( 1 "country" )

The ultimate effect on the user was resolved by changing to XML (and subsequently Canonical's GeoIP services):
https://bugs.launchpad.net/ubuntu/+source/calamares-settings-ubuntu/+bug/1899854

However, this bug remains to track changes to JSON parsing, since it may be something we will need to use in the future.

Given that this doesn't really have an impact on users, its Importance has been set to low.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1898501

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Leó Kolbeinsson (leok)]

I can confirm this.

Firefox reports my location as :

64.1360 N 21.9203 W
Reykjavik Iceland

calamares has selected 'New York' as my location,
Region:America
Zone: New York

I then manually selected Iceland as location and continued with the install.

After installation and reboot the location is correct.

[Raman Sarda (theloudspeaker)]

here's the calamares session log: https://paste.ubuntu.com/p/cbMcQrr998/

[Chris Guiver (guiverc)]

Lubuntu groovy QA-test install (install alongside with web connected); 2020-10-06 ISO

Calamares is reporting me as in New York,

https://ipapi.co/json however reports the following
---
ip "203.214.60.52"
version "IPv4"
city "Prahran"
region "Victoria"
region_code "VIC"
country "AU"
country_name "Australia"
country_code "AU"
country_code_iso3 "AUS"
country_capital "Canberra"
country_tld ".au"
continent_code "OC"
in_eu false
postal "3181"
latitude -37.8517
longitude 145.0007
timezone "Australia/Melbourne"
utc_offset "+1100"
country_calling_code "+61"
currency "AUD"
currency_name "Dollar"
languages "en-AU"
country_area 7686850
country_population 21515754
asn "AS7545"
org "TPG Telecom Limited""

[Leó Kolbeinsson (leok)]

This bug also affects Lubuntu 20.04.1 tested dail ISO 20201005

[Chris Guiver (guiverc)]

Lubuntu groovy daily QA-install (d755-8)
Calamares is reporting me as in New York,
https://ipapi.co/json however reports the following
---

ip "203.214.60.52"
version "IPv4"
city "Prahran"
region "Victoria"
region_code "VIC"
country "AU"
country_name "Australia"
country_code "AU"
country_code_iso3 "AUS"
country_capital "Canberra"
country_tld ".au"
continent_code "OC"
in_eu false
postal "3181"
latitude -37.8517
longitude 145.0007
timezone "Australia/Melbourne"
utc_offset "+1100"
country_calling_code "+61"
currency "AUD"
currency_name "Dollar"
languages "en-AU"
country_area 7686850
country_population 24992369
asn "AS7545"
org "TPG Telecom Limited"

(looks pretty much the same as #6, but Aussie population appears to be growing faster than rabbits... 3M in 3 days???)

[Chris Guiver (guiverc)]

(I forgot to append in last comment)

[Bill (franksmcb) (franksmcb)]

Tested on Groovy daily and getting New York as timezone.

Made the following edit in /etc/calamares/modules/locale.conf

From:

geoip:
   style: "json"
   url: "https://ipapi.co/json"

To:

geoip:
   style: "xml"
   url: "https://ipapi.co/xml"

System was able to correctly identify my location

[Bill (franksmcb) (franksmcb)]

session.log

[ԜаӀtеr Ⅼарсһуnѕkі (wxl)]

Though it does not fix this issue directly (which seems to be related to parsing JSON), switching to XML resolves its symptoms, so there's work in progress to do that and to use Canonical's GeoIP service at that:
https://bugs.launchpad.net/ubuntu/+source/calamares-settings-ubuntu/+bug/1899854

[Leó Kolbeinsson (leok)]

- Tested lubuntu Groovy daily ISO - http://cdimage.ubuntu.com/lubuntu/daily-live/20201015/groovy-desktop-amd64.iso -

- Calamares now selects correct location - in my instance - Iceland -

- Tested on InWin BL641 i5-10400 and Lenovo V14 IIL i3-1005G1 -

Looks like this bug is squashed.

[ԜаӀtеr Ⅼарсһуnѕkі (wxl)]

Glad to hear it, Leó. You might want to comment on that other bug since ultimately you're seeing its effects. It would still be nice to get the JSON handling fixed, so let's keep *this* bug open for that particular purpose.

[ԜаӀtеr Ⅼарсһуnѕkі (wxl)]

Looks like we were mistaken after all. Upstream discovered the issue was apparently ipapi.co rate limiting requests from the default user agent, which is why `curl`/Firefox never showed any issues where Calamares did. I flipped locale.conf over to use https://geoip.kde.org/v1/calamares and the time_zone selector, keeping the json style and everything worked fine.

[Leó Kolbeinsson (leok)]

This bug still ocurring on Focal daily ISO http://cdimage.ubuntu.com/lubuntu/focal/daily-live/20201022/focal-desktop-amd64.iso

Tested on 2 seperate installs/machines.

[Brian Murray (brian-murray)]

I imagine this fix was not SRU'ed to Focal and that is why you still see it there.