win32 :libofx bad utf-8 char name to payee import corrupt xml file

[dleigh]

This is a strange one and may be hard to track down since it was happening, then it stopped happening and now it's happening again.

First of all the environment:
HomeBank for Windows 5.1.2
Running on Windows 10
My homebank file is on OneDrive so that I can access it (at different times) on two different computers. I've tried to be very careful to close HomeBank on one computer before opening it on another. It's version 5.1.2 on both computers.

What happens:
I enter transactions, and do various tasks in HomeBank. I save the file in HomeBank and close it. When I open HomeBank again, the accounts are there, but when I open an account, there are no transactions. When I edit the xhb file, all the "ope" records are still there, but it's like HomeBank can't see them. I have 11 accounts. Only the basic account information and payees remain. If I go into categories or automatic assignments or any individual account, there is nothing there in the dialogs. All of that information is still in the xhb file.

I can reenter the transactions (I do all transactions via OFX/QFX import), save and close again and it happens again.

I'm just starting to use HomeBank as my principal financial tool to enable me to get rid of Quicken once and for all. So at the moment, I can "recreate" my world somewhat easily by reimporting files. Soon this will be very difficult as the files and transactions accumulate. Also I have a ton of categories and automatic assignments and I can't see redoing those.

After trying to track down what was happening, it stopped behaving like this suddenly and everything was good for several days. Then suddenly, today, it started doing it again.

I tried using a file that was not on OneDrive - just a file on the desktop - but I still had the same problem.

Not quite sure where to turn at this point. Any ideas of what I should look at? Any secure way to send you my file for inspection?
Many thanks!

[Maxime DOYEN (mdoyen)]

Strange indeed.
The good news is that txn seems to be there, so there is a filter somewhere that hide it.
I don't think it is necessary to provide your file (BTW you have a anonimizer for that if at a point it make sense to do)

Have you checked with all filter off under the accoutn window ?
All date
Any Type
Any status

And that no other preference option could produce this ?
There might be something you change when this occurs

Let's start with this

[dleigh (david-leighweb)]

Ok - I went into every account and clicked on the "reset filters" button and still nothing showed up. The transactions are still not recognized.

Also the fact that the categories and the automatic assignments don't show up either (but they're both in the file) makes me think that it's not a filter since there isn't any filter for those.

I went through all the preferences and didn't SEE anything that I think would cause the problem. The problem happens on both computers that use this file, so it makes me think there's something wrong in the file, but I don't see any problems with it (like all the XML seems good - all tags closed, etc.).

The ONLY thing that is recognized is the accounts. It's as if everything after the <account records is ignored.

[Maxime DOYEN (mdoyen)]

Let's investigate the file then
please anonimize your file using tools/anonimize
save it to another name
send it to the email you will find on support page
http://homebank.free.fr/support.php

[dleigh (david-leighweb)]

Ok, I've sent you the email. By the way, I uninstalled 5.1.2 and then tried 5.1.1 with the same problem and then back to 5.1.2 with the same problem so I guess it's somehow in the file.

Thanks for your help.

[Maxime DOYEN (mdoyen)]

If you encounter some trouble again, et me know.

[dleigh (david-leighweb)]

Maxime,
It HAS happened again...twice, and without me going into the XML file
outside of Homebank. I DO still have the file in OneDrive, but I have
found one consistent aspect of the problem. It's ALWAYS happening on a
"<pay" record, where the name is "CVS/PHARMACY". I think it must have
something to do with the "/" in the middle of the name field. The line
ends up looking like:

<pay key="1140" name="CVS/PHARMACY ,<pay key="1141" name="VIR SEPA DIME
LEIGH POUR JOSUE" />

instead of:

<pay key="1140" name="CVS/PHARMACY" />
<pay key="1141" name="VIR SEPA DIME LEIGH POUR JOSUE" />

So, my thought is that there may be a real bug - either in Homebank code
or in some library that you're using to parse and write the XML.
Manually fixing the XML file makes it work again.

Hopefully that will help a bit to find the problem.
Thanks,
David

------ Original Message ------
From: "Maxime Doyen" <email address hidden>
To: <email address hidden>
Sent: 1/22/2017 9:04:41 AM
Subject: [Bug 1655626] Re: transactions disappear

>If you encounter some trouble again, et me know.
>
>** Changed in: homebank
> Status: New => Fix Released
>
>--
>You received this bug notification because you are subscribed to the
>bug
>report.
>https://bugs.launchpad.net/bugs/1655626
>
>Title:
> transactions disappear
>
>Status in HomeBank:
> Fix Released
>
>Bug description:
> This is a strange one and may be hard to track down since it was
> happening, then it stopped happening and now it's happening again.
>
> First of all the environment:
> HomeBank for Windows 5.1.2
> Running on Windows 10
> My homebank file is on OneDrive so that I can access it (at different
>times) on two different computers. I've tried to be very careful to
>close HomeBank on one computer before opening it on another. It's
>version 5.1.2 on both computers.
>
> What happens:
> I enter transactions, and do various tasks in HomeBank. I save the
>file in HomeBank and close it. When I open HomeBank again, the accounts
>are there, but when I open an account, there are no transactions. When
>I edit the xhb file, all the "ope" records are still there, but it's
>like HomeBank can't see them. I have 11 accounts. Only the basic
>account information and payees remain. If I go into categories or
>automatic assignments or any individual account, there is nothing there
>in the dialogs. All of that information is still in the xhb file.
>
> I can reenter the transactions (I do all transactions via OFX/QFX
> import), save and close again and it happens again.
>
> I'm just starting to use HomeBank as my principal financial tool to
> enable me to get rid of Quicken once and for all. So at the moment, I
> can "recreate" my world somewhat easily by reimporting files. Soon
> this will be very difficult as the files and transactions accumulate.
> Also I have a ton of categories and automatic assignments and I can't
> see redoing those.
>
> After trying to track down what was happening, it stopped behaving
> like this suddenly and everything was good for several days. Then
> suddenly, today, it started doing it again.
>
> I tried using a file that was not o...

[Maxime DOYEN (mdoyen)]

We must look into the file you import, because naming a payee CSV/TOTO do not lead to problem, so can you:
- tell me in which format you load file into HB
- do any preprocessing on those files
- have a look into one of the file containing CVS/PHARMACY, the char just after the Y should be the one that mixing things

or better send me a file with such a payee into it !

[dleigh (david-leighweb)]

Maxime,
I just reproduced the error and I'm sending you the files that did it
for me.

You may want to compare LEIGH_BEFORE.xhb and LEIGH_AFTER.xhb to see the
difference in the files before running the test.

These are the steps I followed to see the error:
Open LEIGH_BEFORE.xhb in HomeBank 5.1.2 for Windows.Import the following
files (I don't override the importing dialog - I let it select what
transactions are to be
imported):20170206CM.OFX20170206MB.OFX20170206DN.QFX (this one will say
that all the transactions are rejected but I went ahead and clicked NEXT
all the way through - this is the one that contains "CSV/PHARMACY" as
well, so it's probably important to go all the way with it).Save the
file in HomeBank and then exit.Go back into HomeBank with the
LEIGH_BEFORE.xhb fileOpen the register detail for Crédit Mutuel and if
it does the same to you as it does to me, you'll see that there are no
transactions (even though they're still in the file)
Hope this helps,
David

------ Original Message ------
From: "Maxime Doyen" <email address hidden>
To: <email address hidden>
Sent: 2/4/2017 9:55:46 PM
Subject: [Bug 1655626] Re: transactions disappear

>We must look into the file you import, because naming a payee CSV/TOTO
>do not lead to problem, so can you:
>- tell me in which format you load file into HB
>- do any preprocessing on those files
>- have a look into one of the file containing CVS/PHARMACY, the char
>just after the Y should be the one that mixing things
>
>or better send me a file with such a payee into it !
>
>** Changed in: homebank
> Importance: Undecided => Medium
>
>** Changed in: homebank
> Milestone: None => 5.1.4
>
>** Changed in: homebank
> Assignee: (unassigned) => Maxime Doyen (mdoyen)
>
>--
>You received this bug notification because you are subscribed to the
>bug
>report.
>https://bugs.launchpad.net/bugs/1655626
>
>Title:
> transactions disappear
>
>Status in HomeBank:
> Incomplete
>
>Bug description:
> This is a strange one and may be hard to track down since it was
> happening, then it stopped happening and now it's happening again.
>
> First of all the environment:
> HomeBank for Windows 5.1.2
> Running on Windows 10
> My homebank file is on OneDrive so that I can access it (at different
>times) on two different computers. I've tried to be very careful to
>close HomeBank on one computer before opening it on another. It's
>version 5.1.2 on both computers.
>
> What happens:
> I enter transactions, and do various tasks in HomeBank. I save the
>file in HomeBank and close it. When I open HomeBank again, the accounts
>are there, but when I open an account, there are no transactions. When
>I edit the xhb file, all the "ope" records are still there, but it's
>like HomeBank can't see them. I have 11 accounts. Only the basic
>account information and payees remain. If I go into categories or
>automatic assignments or any individual account, there is nothing there
>in the dialogs. All of that information is still in the xhb file.
>
> I can reenter the transactions (I do all transactions via OFX/QFX
> import), save and close again and it happens again.
>
> I'm ju...

[Maxime DOYEN (mdoyen)]

can you provide me the OFX option you have into the preferences import/export ?

[dleigh (david-leighweb)]

Here's a screenshot of my options:

------ Original Message ------
From: "Maxime Doyen" <email address hidden>
To: <email address hidden>
Sent: 2/6/2017 1:57:58 PM
Subject: [Bug 1655626] Re: transactions disappear

>can you provide me the OFX option you have into the preferences
>import/export ?
>
>--
>You received this bug notification because you are subscribed to the
>bug
>report.
>https://bugs.launchpad.net/bugs/1655626
>
>Title:
> transactions disappear
>
>Status in HomeBank:
> Incomplete
>
>Bug description:
> This is a strange one and may be hard to track down since it was
> happening, then it stopped happening and now it's happening again.
>
> First of all the environment:
> HomeBank for Windows 5.1.2
> Running on Windows 10
> My homebank file is on OneDrive so that I can access it (at different
>times) on two different computers. I've tried to be very careful to
>close HomeBank on one computer before opening it on another. It's
>version 5.1.2 on both computers.
>
> What happens:
> I enter transactions, and do various tasks in HomeBank. I save the
>file in HomeBank and close it. When I open HomeBank again, the accounts
>are there, but when I open an account, there are no transactions. When
>I edit the xhb file, all the "ope" records are still there, but it's
>like HomeBank can't see them. I have 11 accounts. Only the basic
>account information and payees remain. If I go into categories or
>automatic assignments or any individual account, there is nothing there
>in the dialogs. All of that information is still in the xhb file.
>
> I can reenter the transactions (I do all transactions via OFX/QFX
> import), save and close again and it happens again.
>
> I'm just starting to use HomeBank as my principal financial tool to
> enable me to get rid of Quicken once and for all. So at the moment, I
> can "recreate" my world somewhat easily by reimporting files. Soon
> this will be very difficult as the files and transactions accumulate.
> Also I have a ton of categories and automatic assignments and I can't
> see redoing those.
>
> After trying to track down what was happening, it stopped behaving
> like this suddenly and everything was good for several days. Then
> suddenly, today, it started doing it again.
>
> I tried using a file that was not on OneDrive - just a file on the
> desktop - but I still had the same problem.
>
> Not quite sure where to turn at this point. Any ideas of what I
>should look at? Any secure way to send you my file for inspection?
> Many thanks!
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/homebank/+bug/1655626/+subscriptions

[Maxime DOYEN (mdoyen)]

Hi David,

I was a bit lost at first because I was not able to reproduce this in my Linux but then remind of a fix I was forced to do just for windows and libofx library, the one I use to read OFX/QFX file. The windows version do not manage some UTF-8 characters very well and this is the case there: the £ seems to be problematic.
I did not pay attention when I added the ability to import NAME OFX field to HB payee, and did not safely ensure what libOFX return is well UTF-8 (what I have done for memo).

To be fixed in 5.1.4.

[dleigh (david-leighweb)]

Wow - good problem-solving! Glad you could dig that one out! Thanks.

------ Original Message ------
From: "Maxime Doyen" <email address hidden>
To: <email address hidden>
Sent: 2/7/2017 9:08:29 PM
Subject: [Bug 1655626] Re: transactions disappear

>Hi David,
>
>I was a bit lost at first because I was not able to reproduce this in
>my Linux but then remind of a fix I was forced to do just for windows
>and libofx library, the one I use to read OFX/QFX file. The windows
>version do not manage some UTF-8 characters very well and this is the
>case there: the £ seems to be problematic.
>I did not pay attention when I added the ability to import NAME OFX
>field to HB payee, and did not safely ensure what libOFX return is well
>UTF-8 (what I have done for memo).
>
>To be fixed in 5.1.4.
>
>** Changed in: homebank
> Status: Incomplete => In Progress
>
>--
>You received this bug notification because you are subscribed to the
>bug
>report.
>https://bugs.launchpad.net/bugs/1655626
>
>Title:
> libofx for windows bad utf-8 char import corrupt xml file
>
>Status in HomeBank:
> Fix Committed
>
>Bug description:
> This is a strange one and may be hard to track down since it was
> happening, then it stopped happening and now it's happening again.
>
> First of all the environment:
> HomeBank for Windows 5.1.2
> Running on Windows 10
> My homebank file is on OneDrive so that I can access it (at different
>times) on two different computers. I've tried to be very careful to
>close HomeBank on one computer before opening it on another. It's
>version 5.1.2 on both computers.
>
> What happens:
> I enter transactions, and do various tasks in HomeBank. I save the
>file in HomeBank and close it. When I open HomeBank again, the accounts
>are there, but when I open an account, there are no transactions. When
>I edit the xhb file, all the "ope" records are still there, but it's
>like HomeBank can't see them. I have 11 accounts. Only the basic
>account information and payees remain. If I go into categories or
>automatic assignments or any individual account, there is nothing there
>in the dialogs. All of that information is still in the xhb file.
>
> I can reenter the transactions (I do all transactions via OFX/QFX
> import), save and close again and it happens again.
>
> I'm just starting to use HomeBank as my principal financial tool to
> enable me to get rid of Quicken once and for all. So at the moment, I
> can "recreate" my world somewhat easily by reimporting files. Soon
> this will be very difficult as the files and transactions accumulate.
> Also I have a ton of categories and automatic assignments and I can't
> see redoing those.
>
> After trying to track down what was happening, it stopped behaving
> like this suddenly and everything was good for several days. Then
> suddenly, today, it started doing it again.
>
> I tried using a file that was not on OneDrive - just a file on the
> desktop - but I still had the same problem.
>
> Not quite sure where to turn at this point. Any ideas of what I
>should look at? Any secure way to send you my file for inspection?
> Many thanks!
>
>To ma...

[Maxime DOYEN (mdoyen)]

5.1.4 is privately available here for last testing before release:
http://homebank.free.fr/public/

[dleigh (david-leighweb)]

Sorry I wasn't able to get to pre-release testing but just to say that it appears to be functioning correctly now. Many thanks for your help!