Data loss of postal addresses between Evolution and Ubuntu One's Funambol exchange/web UI

[Rodrigo Moya]

The freedesktop specification for contacts records (http://www.freedesktop.org/wiki/Specifications/desktopcouch/contact) had, for postal addresses, one field for "street", containing a multiline text, while the Funambol exchange (phone sync) the web UI uses "address1" and "address2" for the same, which results in data loss for users that edit in both Evolution and the Funambol exchange+web UI, and this has been happening since Karmic.

We need several things:
* change the Freedesktop specification to use "address1" and "address2"
* change evolution-couchdb in both Karmic and Lucid to switch from using a multiline "street" field to using "address1" and "address2"
* change the Funambol exchange and the web UI to support records with a multiline "street", so as not to lose data from not yet upgraded clients

So this needs a SRU for couchdb-glib for both Karmic and Lucid and a synchronized change on the server side

[Rodrigo Moya (rodrigo-moya)]

https://code.launchpad.net/~rodrigo-moya/ubuntu/lucid/couchdb-glib/fix-571286 branch has the fix for lucid. Fix for karmic package coming soon

[Scott Kitterman (kitterman)]

Instead of changing everything (including an FDO spec) to match the Ubuntu One web site, wouldn't it make more sense just to change the Ubuntu One web site? There are additional Ubuntu One clients in development and so this seems like a lot of changes on the client.

[Nicola Larosa (teknico)]

Scott, the main problem is not with the web UI, but with the Funambol component that manages syncing with phones: the web UI part is just a reflection of that. The Funambol data definition cannot be changed, because it interacts with hundreds of different phone kinds out there, that's why we need two fields rather than one for the "street" part of the address.

I changed the description to better clarify this, thanks for asking.

[Scott Kitterman (kitterman)]

This really should not be an SRU until after there is general consensus that this is the way free desktops are going.

[Rodrigo Moya (rodrigo-moya)]

Scott, we need 2 fields for the street information anyway, and we first added a 'ext_street' one, but since the only application following the standard is Evolution, and we had all the phone syncing using already address1/address2, we decided to use those instead. So, the freedesktop standard is now changed to match that. Also, code has been added to both evolution-couchdb and the U1 server to deal with old records using 'street' field only, so it's safe to do the SRU, once I've got the karmic package ready.

As for U1 clients being developed, for contacts there's only Akonadi, and it doesn't support postal addresses yet anyway, AFAIK (I might be wrong).

[Scott Kitterman (kitterman)]

Harald,

Would you please comment on how this change would impact Akonadi in Maverick?

[Harald Sitter (apachelogger)]

Well. We just discussed this a bit on IRC and I would like to highlight the following:

a) vCard is an established standard that defines free-form address data without ext
b) as far as we (that is me and a KDE gentoo developer) can see, akonadi will support phone sync via syncml (an established standard), and syncml enforces exchange of contact data via vCard (an even more established standard). (that said I wonder why the fdo spec cant implement vcard to begin with)
c) I do not see how implementations that by design do not have an ext field can support this without actually following the change
d) each mail client and groupware and essentially any addressbook software on the market can handle vcard, and vcard does not have such a silly ext thingy

In a more specific scenario:
http://api.kde.org/4.4-api/kdepimlibs-apidocs/kabc/html/classKABC_1_1Addressee.html
KDE's contact implementation does not implement any kind of ext field so options are:
1. change KDE to support ext
2. not show that data
3. mess with the data to squeeze it into KDE's free-form street label and then extract it again for sync

1 is simply stupid since it basically suggest that every contact application that would want to use coudb/u1 and does not have an ext field, would also have to change around. 2 is really not much different from loss of data. 3 is a major headache and bound to end up in data loss from bugs.

We agreed that the only viable options are:
* Have u1 implement abstraction, instead of doing it for each client - syncml can, so why should u1 not be able to
* fix Funambol

[Harald Sitter (apachelogger)]

<apachelogger> also I find it concerning how quickly they are on changing the spec
<apachelogger> ...what if a company internally implemented that couchdb stuff using the spec...

[Rodrigo Moya (rodrigo-moya)]

Packages for Karmic and Lucid uploaded, waiting for approval

[Rodrigo Moya (rodrigo-moya)]

Harald, so KDE just stores addresses in one big field, right? If so, you are having this problem of more-structured data in the fdo spec, since we are already storing city, postcode, street, etc in separate fields. If that's the case then, what are you planning to do? Or are you talking just about 'street' information? If so, the change you need to do is as easy as concatening address1 and address2 in one string when reading from CouchDB, and splitting it (first line for address1, the rest for address2) when writing. That's exactly what the patch for Karmic and Lucid for couchdb-glib does, to support the old API but the new format.

Also, does this interest mean someone is working on Akonadi to make it read from desktopcouch?

[Nicola Larosa (teknico)]

> d) each mail client and groupware and essentially any addressbook software
> on the market can handle vcard, and vcard does not have such a silly ext thingy

See RFC 2426, par. 3.2.1, "ADR Type Definition" <http://tools.ietf.org/html/rfc2426#section-3.2.1>:

"The structured type value corresponds, in sequence, to the post office box; the extended address; the street address; the locality (e.g., city); the region (e.g., state or province); the postal code; the country name."

Also par. 4, "Formal grammar":

"adr-value = 0*6(text-value ";") text-value
        ; PO Box, Extended Address, Street, Locality, Region, Postal
        ; Code, Country Name"

[Rodrigo Moya (rodrigo-moya)]

> In a more specific scenario:
> http://api.kde.org/4.4-api/kdepimlibs-apidocs/kabc/html/classKABC_1_1Addressee.html
> KDE's contact implementation does not implement any kind of ext field so options are:
>
KDE indeed does:

http://api.kde.org/4.4-api/kdepimlibs-apidocs/kabc/html/classKABC_1_1Address.html#abb3c1e4c0ae5cc9cc0c3f14ad48e604f

[Harald Sitter (apachelogger)]

re #10 no, KDE does not store everything in one big field, it only stores the street data in that field. What's more is that I do not see how Evoluation does use the ext data visually...
About concatening though, I find it horribly horribly horribly awful. How do you sync that if user changed the ext on his phone and the ext part on his desktop in different ways, I suspect it will be difficult to detach the ext data from the street without any guidance where to split. That said, what to do if the user changed the address data in such a way that the ext data is unrecognizable? Taking the whole data and stuff it into address1 is probably no option because the user might be ext aware and have provided an ext and expects it to show up as such on his phone. I really do not think that glueing things together in the implementation is a good idea, especially if you consider that at some point more implements are around. If there is a common flaw in all of them, they all need to be fixed instead of fixing it once for all.

And yes, someone is (or in this case will be) working on Akoandi support https://wiki.kubuntu.org/GSoC/2010/HaraldSitter
+ I think you should keep those options in mind eitherway.

re #12 I seem to have missed that. Anyhow, the problem is that no UI exposes the ext field, most likely because the addressee class does not directly expose this interface. Time to poke upstream I suppose...

[Martin Pitt (pitti)]

I do not want to accept this SRU. Changing data storage formats post-release is a no-go area for me. I think we need to work around this in evo/funambol and just support both the old and new data format. (It's also pretty much a "no way") for further Ubuntu releases).

The current transition code is in couchdb-glib, but how can we be sure that there are no other applications which use those address fields in couchdb without couchdb-glib?

[Steve Langasek (vorlon)]

This had been added to https://wiki.ubuntu.com/LucidLynx/TechnicalOverview, but needs to be documented in https://wiki.ubuntu.com/LucidLynx/ReleaseNotes instead. The note that had been written there was:

 * Street address information might get lost when editing/viewing contacts in Evolution, the Ubuntu One contacts web interface and synced phones. (Bug:571286)

This needs to be expanded upon to more clearly explain to users what steps they should take to work around or avoid this bug.

[Jonathan Riddell (jr)]

rejected upload from lucid-proposed and karmic-proposed unapproved queue in line with pitti's comment above

[Rodrigo Moya (rodrigo-moya)]

I already explained to pitti why we are doing it this way. For Maverick we are changing the on-disk format anyway, and these packages I uploaded prepare Karmic and Lucid users for the change, while also fixing the bug they are already seeing, which is that street information gets lost if they edit contacts in evolution and see them in the web UI and viceversa.

So, with these packages having been rejected, karmic and lucid users won't ever see this bug fixed.

[Scott Kitterman (kitterman)]

I think the reasons this is a bad idea were pretty clearly explained. It would be a good idea to consider them and not just to press on blindly.

[Rodrigo Moya (rodrigo-moya)]

Scott, I haven't heard any reason this is a bad idea apart from "changing the on-disk format is not a good idea" and "how can we be sure there are no other apps using it", because the KDE discussion is out of context, given KDE does not use this couchdb database.

So, about changing the on disk format, I understand that from a stable distro point of view, it sounds scary, but we are going to have to do it several times, as we add new fields, like adding photos for contacts, etc. That's why we provide a detailed field-by-field API in couchdb-glib and are adding one for desktopcouch (see lp:~teknico/desktopcouch/contacts-detailed-api/), so that apps don't have to deal with the on-disk format changes. So, changing it in Evolution is impossible, since Evolution uses that couchdb-glib API and thus doesn't know about the on-disk format at all.

About apps using the contacts database, we have Evolution, which doesn't need any change at all given the above mentioned reasons, and macaco-contacts, which is been changed as soon as this fix goes in, and which is not even part of Karmic or Lucid.

[Robert Collins (lifeless)]

@rodrigo - there are several strands here that interact.

* end users write apps to code in the distro. So when we offer them a release, they know they have a *fixed* basis on which to use the API's present in the distro. Anything which would break a third party user using a shipped API is something that an SRU cannot do except under very specific circumstances - such as 'only way to fix a local root exploit' or 'there is no way to use it without this fix anyway, because it depends on server XYZ which has changed its protocol'.

* Code changes are risky, SRU's are meant to be auditable and understandable; on disk format changes to a database sound (to me) to be unlikely to be easily audited and understood. I may be misunderstanding here, but I'd be expecting the pure distro folk to be hearing what I'm hearing here ;)

* It's not clear whether anyone can directly depend on your on-disk format or not - I mean, you have an API, and you have a disk format, but can anyone open up one of the couchdb's with couch itself, and query? if so, then you may have third party things you don't know about starting to use the shiny you've provided.

[Rodrigo Moya (rodrigo-moya)]

Forgot to talk about 3rd party apps, so here it is.

First of all, we don't know what 3rd party apps are using it, so not sure if there are any at all. Then, we haven't yet agreed on API/on-disk format stability yet for this contacts database, it's still an unstable API/specification, so there is not API/on-disk format stability assurance for this that 3rd party apps can rely on. If a 3rd party app uses this, it already knows (or should know) that this is unstable and can change.

Also, let me explain why we need, not only for this change but for other on-disk format changes we are going to do, the karmic/lucid packages to be upgraded. Whenever we deploy some change on the server, users in those distros will get the new database format automatically on their next synchronization between U1's couchdb and their local couchdb. So, if we are not allowed to do on-disk format changes, this means we need to freeze the format used in Karmic for years and not improve / evolve it at all. Provided, as mentioned above, this is an unstable API/spec, that would mean having to commit to a work-in-progress for years. So this is not about changing on-disk format for an old distro, it is about making it support stuff that they will be getting from the server, which is not tied to any distro, for the next few years.

[Scott Kitterman (kitterman)]

You do know there's a KDE client for Ubuntu One being developed that's expected to land in this cycle? It's true that there's no KDE user of the database today, but that's hardly the only relevant data point.

The problem isn't with adding new information to the database, it's with modifying the existing address format.

I thought it was quite clear that the correct answer to the address question was for your web app to convert between the two formats. Patching all potential users of address data to match the format used by your proprietary back end is not the solution.

[Martin Pitt (pitti)]

Rodrigo Moya [2010-05-26 8:56 -0000]:
> So, if we are not allowed to do on-disk format changes, this means
> we need to freeze the format used in Karmic for years

Well, isn't that the point of stable releases?

> and not improve / evolve it at all.

We did not say that, though. The problem is right now that the way we
use couchdb now does not provide any kind of versioning. If there was
a version field which client apps could check, and which would be
bumped on format changes, it would at least be possible for client
apps to handle underlying format changes. Right now it's not possible
at all, and if you mix different couchdb sources from different
releases/sources/etc. you'd just create a mess. As long as this point
does not get addressed, I do not want to accept such kind of
unversioned on-disk format changes as an SRU. Once there is proper
support for versioning, this can be revisited, of course.

[Scott Kitterman (kitterman)]

Additionally, I still think distro unique MUA patches to change the data format are a mistake.

[Martin Albisetti (beuno)]

Bumping to low as the solution is still unclear.

[Martin Pitt (pitti)]

rodrigo_ | pitti, bug #571286 was an issue for karmic, I think, it's not anymore

[Joshua Hoover (joshuahoover)]

Marked invalid since Funambol is no longer supported.

[Rolf Leggewie (r0lf)]

lucid has seen the end of its life and is no longer receiving any updates. Marking the lucid task for this ticket as "Won't Fix".