com.canonical.scopes.nominatim does not provide localized address

[Kyle Nitzsche]

com.canonical.scopes.nominatim scope in hanloon project is aggregated by NearBy scope.

It provides the user's current address.

A bug was filed against NearBy saying the address is not localized.

This can only be fixed in the nominatim child scope.

[Kyle Nitzsche (knitzsche)]

original bug description:

On my phone the location in "where are you" is like:

$house-number,$street,$part-of-city,$city,$district,$state,$zip,$country,$continent

a proper german format would be:

$street,$house-number,$zip,$city,$part-of-city,$district,$state,$country,$continent

i would actually expect $LC_ADDRESS to make the scope use the proper format, either we do not read/respect the environment variable or we do not set it properly on phones.

[Kyle Nitzsche (knitzsche)]

assigned to yc so he can reassign as appropriate

[Jin (jindallo)]

Hello Enwei,

Please have a look on this bug,
thank you.

[Kyle Nitzsche (knitzsche)]

Here's how we may be able to provide localized addresses inside golang.

You can run C code in golang. So we can get the localized address format (which shows the order of address components).

Include these lines near the top of the nominatum .go file:
//#include <locale.h>
//#include <libintl.h>
//#include <langinfo.h>
import "C"

Now we can access those C functions in go. Also need to convert C strings to go strings as below.

We can use the following to get the current locale's address format in your go:

        //ensure C knows the current locale. convert it to go string, and log it (don't log it in released code though)
        log.Printf("ADDY. locale: %q\n", C.GoString(C.setlocale(C.LC_ALL, C.CString(""))))

        //get the current locale's address format and covert it to a golang string
        addFmt := C.GoString(C.nl_langinfo(C._NL_ADDRESS_POSTAL_FMT))

I logged addFmt and ran my sample scope twice, in English/US locale and in Spanis/Spain locale and logged the result:
        log.Printf("ADDY: %q\n", addFmt)

Here's what was in the log, en/USA and es/Spain:

2015/11/03 14:23:22 ADDY. locale: "es_ES.utf8"
2015/11/03 14:23:22 ADDY: "%f%N%a%N%d%N%b%N%s %h %e %r%N%z %T%N%c%N"

2015/11/03 14:28:29 ADDY. locale: "en_US.utf8"
2015/11/03 14:28:29 ADDY: "%a%N%f%N%d%N%b%N%h %s %e %r%N%T, %S %z%N%c%N"

Note the format strings are different.

To understand the format string components, check http://www.eki.ee/itstandard/2000/LC_ADDRESS.shtml

Notice for example that %a means C/O Address, which in en/USA comes *first* but in es_ES comes *third*.

With this, we should be able to determine the order of address components for the current locale. We would then need to split the address returned from the web API into the corresponding parts and reconstruct an address from those parts using the order as returned above, in theory :) (and of course we'd need to handle invalid format string returned from the C code too, perhaps in that case using the address from the web API without modification.)

[Zhang Enwei (zhangew401)]

Thank you very much, Kyle.
I met other obstacles and need your advice.
1. The returned values from Nominatim API seems not standard and they used a lot of items to specify one location. I took some examples for your reference. What I want to use is the value of "address".
{"place_id":"2576701262","licence":"Data © OpenStreetMap contributors, ODbL 1.0. http:\/\/www.openstreetmap.org\/copyright","osm_type":"way","osm_id":"334026024","lat":"52.5714773","lon":"19.8971761","display_name":"Płocka, Radzanowo, gmina Radzanowo, powiat płocki, Masovian Voivodeship, 09-451, Poland","address":{"road":"Płocka","village":"Radzanowo","county":"gmina Radzanowo","state":"Masovian Voivodeship","postcode":"09-451","country":"Poland","country_code":"pl"}}

{"place_id":"66454177","licence":"Data © OpenStreetMap contributors, ODbL 1.0. http:\/\/www.openstreetmap.org\/copyright","osm_type":"way","osm_id":"48121662","lat":"45.011207","lon":"20.018092","display_name":"126, Indjija Municipality, Syrmia District, Vojvodina, 22321, Serbia","address":{"road":"126","city":"Indjija Municipality","county":"Syrmia District","state":"Vojvodina","postcode":"22321","country":"Serbia","country_code":"rs"}}

{"place_id":"801104","licence":"Data © OpenStreetMap contributors, ODbL 1.0. http:\/\/www.openstreetmap.org\/copyright","osm_type":"node","osm_id":"271373068","lat":"52.5167951","lon":"13.38312","display_name":"S+U Brandenburger Tor, Under the linden trees, Mitte, Berlin, 10117, Germany","address":{"bus_stop":"S+U Brandenburger Tor","road":"Under the linden trees","suburb":"Mitte","city_district":"Mitte","city":"Berlin","state":"Berlin","postcode":"10117","country":"Germany","country_code":"de"}}

{"place_id":"2144230998","licence":"Data © OpenStreetMap contributors, ODbL 1.0. http:\/\/www.openstreetmap.org\/copyright","lat":"35.8620926077602","lon":"-84.4592530653989","display_name":"199, Little Johnston Valley Road, Lawnville, Roane County, Tennessee, 37763, United States of America","address":{"house_number":"199","road":"Little Johnston Valley Road","hamlet":"Lawnville","county":"Roane County","state":"Tennessee","postcode":"37763","country":"United States of America","country_code":"us"}}

{"place_id":"56837758","licence":"Data © OpenStreetMap contributors, ODbL 1.0. http:\/\/www.openstreetmap.org\/copyright","osm_type":"way","osm_id":"20206256","lat":"51.6479763","lon":"-0.3947587","display_name":"Wiggenhall Road, Watford Heath, Watford, Hertfordshire, East of England, England, United Kingdom","address":{"road":"Wiggenhall Road","suburb":"Watford Heath","town":"Watford","county":"Hertfordshire","state_district":"East of England","state":"England","country":"United Kingdom","country_code":"gb"}}
I didn't find any document from Nominatim about the dictionary they used. It seems difficult to map this variety of items into the standard locale address format if I cannot get the rule for transformation.

2. In http://man7.org/linux/man-pages/man5/locale.5.html, it seems "town" has the same meaning with "city"
but in fact, "town" means an administrative unit below county(which is below city).

3. It seems the locale address format returned from C is not correct at least for Ch...

[Kyle Nitzsche (knitzsche)]

Please note: for more up-to-date description of LC_ADDRESS fields use this command:
$ man 5 locale

For example, that ^ shows State, whereas the web page I linked above does not show State.

I'll comment more shortly.

[Kyle Nitzsche (knitzsche)]

reply to point 1.

1. Can we map the nominatum "address" keys to LC_ADDRESS fields and use the ones we do have?

For example, all four examples you show have the following key/values, here is how you might map them to LC_ADDRESS fields

LCADDRESS field | nominatum key
%s Street or block (e.g., Japanese) name. | "road"
%T Town, city ||| "town" or "village" or "city" or "hamlet" (use the one they provide)
%S State, province, or prefecture. | state
%c Country, as taken from data record. | country

In addition some have, which can be added to our address if available:
%h House number or designation. | "house_number"
%z Zip number, postal code. | "postal_code"

[Kyle Nitzsche (knitzsche)]

reply to comment 2.

True that "town" and "city" do not have the *exact* same meaning, but they may be close enough to use the one nominatum provides (or "village" or "hamlet" as noted above).

[Kyle Nitzsche (knitzsche)]

reply to point 3.

If the Chinese address format is incorrect, you can simply handle that case (locale is equal to zh_CN.UTF-8) using your own corrected format.

For extra credit file a bug against glibc source package, perhaps here: http://www.gnu.org/software/libc/bugs.html

Note that the overall goal here is not to display an address sufficient for mailing a package. Instead, the goal is to provide a readable statement of the user's location in an address-like format.

[Zhang Enwei (zhangew401)]

Hi Kyle,
Thank you and I am much appreciated of your advice.
Yes, I am using latest format from locale, there are 15 items instead of 14 on the webpage(http://www.eki.ee/itstandard/2000/LC_ADDRESS.shtml).

I think this info you give me is very critical.
"Note that the overall goal here is not to display an address sufficient for mailing a package. Instead, the goal is to provide a readable statement of the user's location in an address-like format."

Based on this purpose, I will delete many other items like "residential", "bus_stop", "district", "sub_district", "company", "corporation", "kindergarten", "hospital" , etc, etc, etc. And will use only what we care about(mainly in the format string returned from C library), just as you have listed above in the answer to my Point 1.
Two points I need to say,
1. for German case, in "$street,$house-number,$zip,$city,$part-of-city,$district,$state,$country,$continent", part-of-city and district will be ignored, is this okay?
2. Is the format returned from C library correct for all countries except only China? what about Indian? If we cannot test all countries for now, we need to test before we launch products in those countries.

I will begin to implement the change asap and reply you the test result.
Thank you so much.

[Zhang Enwei (zhangew401)]

Hi Kyle,
I have implemented this patch. At the same time I got answer from OSM(OpenStreetMap), they suggested we use Open Cage API(http://geocoder.opencagedata.com/api.html#formatte) which uses the same database of OSM but the address is formatted.
Below is a comparison between the two solutions. IMHO, Open Cage works well.
the value of address is got from OSM which is the basis for me to format.
the value of display_name is used by current solution, also from OSM.
the value of formated_name is the result after I adapted address according to LC_ADDRESS.
the value of opencage is I got by using Open Cage API.
German
"%f%N%a%N%d%N%b%N%s %h %e %r%N%z %T%N%c%N"
"address":{"bus_stop":"S+U Brandenburger Tor","road":"Under the linden trees","suburb":"Mitte","city_district":"Mitte","city":"Berlin","state":"Berlin","postcode":"10117","country":"Germany","country_code":"de"}
display_name: S+U Brandenburger Tor, Unter den Linden, Mitte, Berlin, 10117, Deutschland
formated_name: Unter den Linden, 10117, Berlin, Deutschland
opencage: S+U Brandenburger Tor, Unter den Linden, 10117 Berlin, Deutschland

American
"%a%N%f%N%d%N%b%N%h %s %e %r%N%T, %S %z%N%c%N"
"address":{"house_number":"199","road":"Little Johnston Valley Road","hamlet":"Lawnville","county":"Roane County","state":"Tennessee","postcode":"37763","country":"United States of America","country_code":"us"}
display_name: 199, Little Johnston Valley Road, Lawnville, Roane County, Tennessee, 37763, United States of America
formated_name: 199, Little Johnston Valley Road, Lawnville, Tennessee, 37763, United States of America
opencage: 199 Little Johnston Valley Road, Lawnville, TN 37763, United States of America

Japan
"%z%c%T%s%b%e%r"
"address":{"park":"Shinjuku Central Park","pedestrian":"水の広場","city":"Shinjuku","state":"Tokyo","region":"Kanto","postcode":"163-8001","country":"Japan","country_code":"jp"}
display_name: 新宿中央公園, 水の広場, 新宿区, 東京都, 関東地方, 163-8001, 日本
formated_name: 163-8001, 東京都, 新宿区, 日本
opencage: 新宿中央公園, 水の広場, 163-8001 新宿区, 日本

Indian
LC_ADDRESS=en_IN.UTF-8
"%z%c%T%s%b%e%r"//this is wrong
display_name: NH 744, Kollam, Kerala, 691001, India
formated_name: 691001, India, Kollam, NH 744
opencage: NH 744, Kollam - 691001, Kerala, India // this is correct

China
display_name: 金牛镇，青县，沧州/cangzhou，中国
formated_name: 青县，中国
opencage: 金牛镇，青县， 中国

[Kyle Nitzsche (knitzsche)]

Hi Enwei. This is in response to your comment #10

1: You wrote: "1. for German case, in "$street,$house-number,$zip,$city,$part-of-city,$district,$state,$country,$continent", part-of-city and district will be ignored, is this okay?"

Yes. If the data you receive does not have "part-of-city" or "district", I think you will not include them in the displayed result in the scope.

2: You asked whether all LC_ADDRESS formats are correct for every part of the planet, solar system, galaxy and universe. :) I don't know. You might want to ask Penk to have someone test particularly the Indian and Russian addresses. (I was surprised the Chinese format is wrong: LC_ADDRESS has been around for a long time.)

[Zhang Enwei (zhangew401)]

Hi Kyle,
Thank you.
1. Probably I didn't express clearly. I mean I will take the LC_ADDRESS string as a standard.
I will try to map the info I got from OSM to the string. Rules as follows.
%f: one of company or corporation that is not empty.
%d: department if it is not empty.
%b: building if it is not empty.
%h: one of housenumber or house_number that is not empty.
%s: one of street, road or block that is not empty
%b: one of building or door that is not empty.
%c: country if is not empty.
%S: one of state, province or prefecture that is not empty.
%T: one of town, city, village, hamlet or county that is not empty.
suppose the data I received contains "village", "county", "part-of-city" and "city", I don't know how to arrange their positions in the string, so I will abandon "part-of-city" and choose one of "village", "county" and "city".
2. According to Comment 11, I found the LC_ADDRESS for Indian is also wrong.
Thank you very much.

[Zhang Enwei (zhangew401)]

Hi Kyle,
Thank you so much,
I explained more about the first point, please help check the comments.
https://bugs.launchpad.net/savilerow/+bug/1508169

Br
Enwei

On Fri, Nov 6, 2015 at 6:27 AM, Kyle Nitzsche <email address hidden>
wrote:

> Hi Enwei. This is in response to your comment #10
>
> 1: You wrote: "1. for German case, in "$street,$house-number,$zip,$city
> ,$part-of-city,$district,$state,$country,$continent", part-of-city and
> district will be ignored, is this okay?"
>
> Yes. If the data you receive does not have "part-of-city" or "district",
> I think you will not include them in the displayed result in the scope.
>
> 2: You asked whether all LC_ADDRESS formats are correct for every part
> of the planet, solar system, galaxy and universe. :) I don't know. You
> might want to ask Penk to have someone test particularly the Indian and
> Russian addresses. (I was surprised the Chinese format is wrong:
> LC_ADDRESS has been around for a long time.)
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1508169
>
> Title:
> com.canonical.scopes.nominatim does not provide localized address
>
> Status in Canonical System Image:
> Triaged
> Status in The Savilerow project:
> In Progress
>
> Bug description:
> com.canonical.scopes.nominatim scope in hanloon project is aggregated
> by NearBy scope.
>
> It provides the user's current address.
>
> A bug was filed against NearBy saying the address is not localized.
>
> This can only be fixed in the nominatim child scope.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/canonical-devices-system-image/+bug/1508169/+subscriptions
>

[Kyle Nitzsche (knitzsche)]

Hi Enwei,

Regarding using Open Cage API, that may make sense. they probably have a better sense of how to localize addresses than we do.

 http://geocoder.opencagedata.com/pricing.html

They are in "beta". They say the first 2500 requests per day are free. What happens if we exceed 2500? They do arrange for paid for service. But I do not see a price.

Can you ask YC whether switching makes sense to him.

Let's assume for now we do not switch to Open Cage API:
1) Please include "postcode" (%z Zip number, postal code)
2) You seem to know the correct Indian and Chinese address formats, so please use them instead of LC_ADDRESS.
3) It is not clear to me why the country names in "formated_name" are translated but they are not translated in "address". Please note that if you need translated country names, you can get them using the function DGettext("iso_3166", COUNTRYNAME).

[Zhang Enwei (zhangew401)]

Thank you, Kyle.
I am in contact with Open Cage, after I get price info, I will talk to YC.
1) Yes, %z is included.
2) Yes, I will do this adjustment.
3) Please ignore this issue because the address value is got from my PC. It is localized on the phone.
So do you agree with the mapping rules in Comment 13?

Although the code to use Open Cage is ready, I am not sure when we the whole thing will be hammered out.
I think if this issue is urgent, we should use current patch on OSM, do you agree?

Thanks a lot.

[Zhang Enwei (zhangew401)]

Hi Kyle and YC,
Here is the response from Open Cage.

a. a free level for testing
b. a small package priced at $100/month (or $1000/year if paid in
advance) and allowing 20,000 queries/day
c. medium at $500/month ($5,000/year) up to 100,000/day
d. large at $1,000/month or $10,000/year more than 100,000/day
e. custom solutions and dedicate servers for large enterprise
customers - price really depends on the solution.

Our website still says 'beta'. That's not the geocoding system itself,
that can (and does) handle millions of requests per day, but the
billing system: You'd need to tell us your billing address, we
create an invoice/subscription which you can pay with the usual
credit cards or Paypal (powered by Stripe). We haven't integrated
that sub-system into the rest of our website yet.

[Kyle Nitzsche (knitzsche)]

Hi Enwei,

On 11/05/2015 05:44 PM, Zhang Enwei wrote:
> Hi Kyle,
> Thank you.
> 1. Probably I didn't express clearly. I mean I will take the LC_ADDRESS string as a standard.
> I will try to map the info I got from OSM to the string. Rules as follows.
> %f: one of company or corporation that is not empty.
> %d: department if it is not empty.
> %b: building if it is not empty.
> %h: one of housenumber or house_number that is not empty.
> %s: one of street, road or block that is not empty
> %b: one of building or door that is not empty.
> %c: country if is not empty.
> %S: one of state, province or prefecture that is not empty.
> %T: one of town, city, village, hamlet or county that is not empty.
> suppose the data I received contains "village", "county", "part-of-city" and "city", I don't know how to arrange their positions in the string, so I will abandon "part-of-city" and choose one of "village", "county" and "city".
Regarding %T. I think city is the preferred one. So if you have to
choose between city and any of the others, please choose city.
> 2. According to Comment 11, I found the LC_ADDRESS for Indian is also wrong.
> Thank you very much.
Use the correct format, if you have it.

Thanks!

[Zhang Enwei (zhangew401)]

Hi Kyle,
Thank you very much for helping me figuring out the patch.

Br
Enwei

[Zhang Enwei (zhangew401)]

Got it, Thank you Kyle.

On Fri, Nov 6, 2015 at 9:07 PM, Kyle Nitzsche <email address hidden>
wrote:

> Hi Enwei,
>
> On 11/05/2015 05:44 PM, Zhang Enwei wrote:
> > Hi Kyle,
> > Thank you.
> > 1. Probably I didn't express clearly. I mean I will take the LC_ADDRESS
> string as a standard.
> > I will try to map the info I got from OSM to the string. Rules as
> follows.
> > %f: one of company or corporation that is not empty.
> > %d: department if it is not empty.
> > %b: building if it is not empty.
> > %h: one of housenumber or house_number that is not empty.
> > %s: one of street, road or block that is not empty
> > %b: one of building or door that is not empty.
> > %c: country if is not empty.
> > %S: one of state, province or prefecture that is not empty.
> > %T: one of town, city, village, hamlet or county that is not empty.
> > suppose the data I received contains "village", "county", "part-of-city"
> and "city", I don't know how to arrange their positions in the string, so I
> will abandon "part-of-city" and choose one of "village", "county" and
> "city".
> Regarding %T. I think city is the preferred one. So if you have to
> choose between city and any of the others, please choose city.
> > 2. According to Comment 11, I found the LC_ADDRESS for Indian is also
> wrong.
> > Thank you very much.
> Use the correct format, if you have it.
>
> Thanks!
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1508169
>
> Title:
> com.canonical.scopes.nominatim does not provide localized address
>
> Status in Canonical System Image:
> Triaged
> Status in The Savilerow project:
> In Progress
>
> Bug description:
> com.canonical.scopes.nominatim scope in hanloon project is aggregated
> by NearBy scope.
>
> It provides the user's current address.
>
> A bug was filed against NearBy saying the address is not localized.
>
> This can only be fixed in the nominatim child scope.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/canonical-devices-system-image/+bug/1508169/+subscriptions
>

[Zhang Enwei (zhangew401)]

Please QA team use this click for testing.
Thanks.

[Elsa Wang (elsawang)]

@Evan, could you pls help to verify this issue? thanks.

[Yuan-Chen Cheng (ycheng-twn)]

I think use LC_ADDRESS and some built-in table can be the first step.
And we can submit patch to glibc for area that we think glibc's result is not good enough.

[Zhang Enwei (zhangew401)]

A new bug reported to glibc, https://sourceware.org/bugzilla/show_bug.cgi?id=19220

[Zhang Enwei (zhangew401)]

latest click

[Evan Wang (wsy324)]

Respectively verified each click of locations for Germany,London,Beijing,Spain,India,Russia,American,France,Italy,and Switzerland provided by Enwei, the address formats are displayed as expected.

formated_name: Unter den Linden, 10117, Berlin, Deutschland

formated_name: Blue Fin Building, Southwark Street, SE1 0NZ, London, United Kingdom

Beijing：lat=39.9235&lon=116.4034
formated_name: 中国, 北京市, 五四大街

> lat=40.452587;lon=-3.693831//Spain

formated_name: Avenida del General Perón, 36, 28020, Madrid, España

> lat, lon = 22.755954, 77.740116//indian

formated_name: SH22, Hoshangābād, Madhya Pradesh, India

> lat=64.269278;lon=100.217653//Russia

formated_name: Колхозная улица, сельское поселение Тура, Россия

> lat=38.459216;lon=-98.539594//american

formated_name: Northeast 130 Avenue, Barton County, Kansas, 67525,
United States of America

> lat=46.645122;lon=2.218117//France

formated_name: D 70, 18170, Maisonnais, France

> lat=41.888844;lon=12.538363//Italy

formated_name: Via Renzo Da Ceri, 00177, Roma, Italia

> lat=46.928585;lon=7.426990//Switzerland

formated_name: Heimweg, 9, 3097, Köniz, Schweiz, Suisse, Svizzera,
Svizra

[Zhang Enwei (zhangew401)]

Click submitted.

The change list is submitted as rev. 915,
below is the detail information.
  https://bazaar.launchpad.net/~hanloon-team/hanloon/clicks/revision/915

Information:
  com.canonical.scopes.nominatim_1.0.9_armhf.click

[Jin (jindallo)]

Hello Penk,

Please help us to push this .click released,
thank you.

[Pat McGowan (pat-mcgowan)]

seems this change was reverted the following day with 1.0.10

[Penk Chen (penk)]

com.canonical.scopes.nominatim 1.0.11 will be released in OTA-9 as described in bug #1515182, mark this as fix committed.

[Kyle Nitzsche (knitzsche)]

Since this is fix-released in canonical-devices-system-images, it should also be marked fixed released in savilerow.