Cut off messages

[Krzysztof Tataradziński]

Hello,

My girlfriend send me some SMS with text: 'Umiem. Skoro się pytam to chcę (emoticon) bo miałeś wczoraj rację, że dawno nie wychodziliśmy', but I've received 'nie wychodzilismy'.
We tested that exact text few times - all the time I'm receiving 'nie wychodzilismy'. Could anyone test that text? Please look at 'ś' that disappeared in my phone.
We test that without emoticon and - magic, I've received all text, but without polish special signs (like ł ś ę, etc.). Inserted other emoticon - received 'ie wychodzilismy'.
I must also admin that I'm getting 's', 'e', etc. instead of 'ś', 'ę', etc.

[Michał Sawicz (saviq)]

Seems to be working fine when looping to my phone, see attachment. Will try with another phone in a while.

[Krzysztof Tataradziński (ktatar156)]

Maybe this is related to the fact that messages was sent from phone with android (Hangouts app) and by Swype keyboard (and emoticons was taken from them).

[Michał Sawicz (saviq)]

I've tried a longer text, too.

Krzysztof, any reason to consider that your girlfriend's phone is actually sending it without diacritics? Have you tried to send that text to another phone?

[Michał Sawicz (saviq)]

Sent the exact text you pasted from a Nokia N9 and received it as shown...

[Michał Sawicz (saviq)]

Sorry, totally can't reproduce, sent two messages from Android hangouts, and got everything as expected, see attachment.

[Michał Sawicz (saviq)]

Also, Krzysztof, I'm not sure if I completely understand the issue - are you actually getting only part of the message, or is the problem the lack of diacritics?

[Tiago Salem Herrmann (tiagosh)]

If you are getting characters replaced, I would say the operator is incorrectly parsing/editing the message before sending.

Could you run "dbus-monitor > session.log", and in another shell "dbus-monitor --system > system.log", and then receive the message and attach the logs here?

Also note that the files might contain phone numbers, feel free to remove them before attaching the logs.

Thanks.

[Krzysztof Tataradziński (ktatar156)]

@Michał, I've got two problems: one that message is cut off. Second is related to diacritics.
About first one: could anyone test that with sending message by entered by Swype keyboard (and add some emoticons from it)?
I think that second problem isn't related to Hangouts/Swype or even android. Sometimes I receive with diacritics, sometimes not.
@Tiago, that logs are really big. How can I find some personal info and hide them?

[Krzysztof Tataradziński (ktatar156)]

Sorry, my mistake - about second one: it's swiftkey keyboard, not swype. Maybe bug is related to that I can't receive any MMS messages?

[Krzysztof Tataradziński (ktatar156)]

Another mistake - not about second, it's about first one :)

[Krzysztof Tataradziński (ktatar156)]

Please look at attachments.
dia-session is related to missing polish diacritics.
There is "a e c s z z", instead of 'ą ę' etc....

swiftkey-hangout-to-ubuntu-session is related to cut off message. I sent message (04.12.2015) from LG F Fino with Android (by Hangouts SMS + Swiftkey) with text: "To jest drobny test wiadomości <emoticon-from-swiftkey>, a więc jak widać nie wszystko działa", but received only "o dziala".

If you look at both files, you can see that sometimes there are polish diacritics, sometimes not.

[Tiago Salem Herrmann (tiagosh)]

The two possibilities here are that ofono is truncating the message after it's received, or the carrier is already sending it truncated to the phone.
The fact that characters like "ą ę" are being replaced by "a e" already indicates that something in the middle (carrier) is trying to replace/remove utf-8 characters from the message.
My opinion is that the carrier is editing the message and the swiftkey emoticon is making their parser break.

If possible, insert the same sim card on another phone with another OS and try sending the very same message. If it arrives truncated, than it's a bug in the carrier side, otherwise if might be something in ofono.

Thanks

[Krzysztof Tataradziński (ktatar156)]

Dear All,
I've tested that "To jest drobny test wiadomości <custom-emoticon-from-swiftkey>, a więc jak widać nie wszystko działa" with same SIM card on android 4.2 Lenovo A820 - everything works fine, whole message received.
Guys, if someone can, please test above string to confirm (use android phone with Swiftkey and put some randomly emoticons from it (can't confirm that bug appears with all or only some emoticons from Swiftkey))

[Tiago Salem Herrmann (tiagosh)]

In this case I believe the bug is in ofono/ril, since the IncomingMessage signal we get from ofono already contains the message cut off.

[Krzysztof Tataradziński (ktatar156)]

Okey, so should I report that bug anywhere else or leave here?

[Tony Espy (awe)]

@Krystoff

So to summarize:

1. If SwiftKey is used on an Android device to send a message containing diacritics, the message is displayed on Ubuntu minus the diacritics.

2. If a SwiftKey emoticon is included in the sent message, the received message is truncated ( and still lacks the diacritics ).

So far, the traces to date have shown that the IncomingMessage DBus signals from ofono show the same text as displayed by the messaging app.

The next check is to get traces of the raw contents of the incoming SMS messages. To do this:

 - as root, edit the file /etc/init/ofono.override on the phone
 - add the line "env OFONO_RIL_TRACE=y" to the file, just below the line "env OFONO_RIL_NUM_SIM_SLOTS=1"
 - reboot the phone
 - re-try sending the various messages to the phone
 - when finished, please attach the file /var/log/syslog to this bug

Also, if it's possible to send the same message with diacritics and no emoticon via the Hangouts app, please give this a try. If this works, then the blame probably lies with the encoding specified by SwiftKey.

[Krzysztof Tataradziński (ktatar156)]

I'm getting 'error writing /etc/... : Read-only file system' while trying to save file. Could anyone with writable file system test that?
Regarding 1. I think that is not SwiftKey related - if you see at my logs, sometimes there are diacritics, sometimes not, maybe randomly?

[Tony Espy (awe)]

@Krzysztof

The error you're seeing is that by default, the Ubuntu filesystem is read-only. Sorry for not mentioning this. There is a way to make your filesystem write-able, but an easier way to get the traces is to run the following command as root:

# restart ofono OFONO_RIL_DEVICE=mtk OFONO_RIL_NUM_SIM_SLOTS=2 OFONO_RIL_TRACE=y

Now, re-run your tests and then pull /var/log/syslog from the device and attach to the bug.

Whenever you finish, re-run the above command but leave off "OFONO_RIL_TRACE=y" ( or just reboot ).

Regarding (1), you see the problem with SwiftKey ( ie. sometimes the diacritics are missing ). Have you tried the exact same messages that fail to pass diacritics with Hangouts? If you haven't tried this, could you please?