libofx crashes on OFX import in gnucash

[Zach Sadecki]

=== SRU Justification ===

Impact: Certain kinds of OFX files used to work in 10.04 but started crashing in 10.10. And still do in 11.04.

Fixed upstream in 0.9.4, by applying a different version of the patch here.

To reproduce:
 0. Download attached test.ofx.xml
 1. $ sudo apt-get install gnucash
 2. $ gnucash
 3. Set up an account, just keep clicking Forward
 4. File->Import->OFX/QFX... and pick test.ofx.xml
 5. You should now see a crash

Patch:
Because Ubuntu only has 0.9.0 but upstream has released four bugfix releases (up to 0.9.4) that address this and other outstanding issues and Debian hasn't been responsive for years, it is felt that just updating to 0.9.4 is best. I've attached https://launchpadlibrarian.net/70440554/libofx.debian.diff which is just the debian/ changes for such an update.

If this is considered too intrusive, a more targeted patch could be prepared. This was just easier and seemed low-risk as there are no new features.

Full branch: https://code.launchpad.net/~mterry/ubuntu/natty/libofx/libofx.new-upstream-benoit-sru

See also bug 769554 and bug 771233, fixed by this same update.

Testing:
1:0.9.4-0ubuntu0.1 has been pushed to natty-proposed. Please test and verify the fix here. See https://wiki.ubuntu.com/QATeam/PerformingSRUVerification

=========================

I recently upgraded to Maverick from Lucid. OFX imports used to work fine, now they cause gnucash to crash.

I see this error on the console:
....
LibOFX INFO: Created OfxDummyContainer to hold unsupported aggregate CCSTMTTRNRS
(Above message occured on Line 1, Column 279)
terminate called after throwing an instance of 'std::out_of_range'
  what(): basic_string::substr

and a gdb stack trace shows the following:
#0 0x00007fffef09aba5 in raise () from /lib/libc.so.6
#1 0x00007fffef09e6b0 in abort () from /lib/libc.so.6
#2 0x00007fffe7ad66bd in __gnu_cxx::__verbose_terminate_handler() ()
   from /usr/lib/libstdc++.so.6
#3 0x00007fffe7ad4906 in ?? () from /usr/lib/libstdc++.so.6
#4 0x00007fffe7ad4933 in std::terminate() () from /usr/lib/libstdc++.so.6
#5 0x00007fffe7ad4a3e in __cxa_throw () from /usr/lib/libstdc++.so.6
#6 0x00007fffe7a7f38e in std::__throw_out_of_range(char const*) ()
   from /usr/lib/libstdc++.so.6
#7 0x00007fffdf4b71dd in ofxdate_to_time_t(std::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from /usr/lib/libofx.so.4
#8 0x00007fffdf4c4388 in OfxBalanceContainer::add_attribute(std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from /usr/lib/libofx.so.4
#9 0x00007fffdf4ce799 in OFXApplication::endElement(SGMLApplication::EndElementEvent const&) () from /usr/lib/libofx.so.4
#10 0x00007fffdf1a8842 in OpenSP::GenericEventHandler::endElement(OpenSP::EndElementEvent*) () from /usr/lib/libosp.so.5
#11 0x00007fffdf215a4c in OpenSP::Parser::implyCurrentElementEnd(OpenSP::Location const&) () from /usr/lib/libosp.so.5
#12 0x00007fffdf215db6 in OpenSP::Parser::acceptEndTag(OpenSP::EndElementEvent*) () from /usr/lib/libosp.so.5
#13 0x00007fffdf218d63 in OpenSP::Parser::doContent() ()
   from /usr/lib/libosp.so.5
#14 0x00007fffdf1becf8 in OpenSP::Parser::parseAll(OpenSP::EventHandler&, int const volatile*) () from /usr/lib/libosp.so.5
#15 0x00007fffdf1c1b0e in OpenSP::ParserApp::parseAll(OpenSP::SgmlParser&, OpenSP::EventHandler&, int const volatile*) () from /usr/lib/libosp.so.5
#16 0x00007fffdf1c3161 in OpenSP::ParserEventGenerator::run(SGMLApplication&)
    () from /usr/lib/libosp.so.5
#17 0x00007fffdf4cdf92 in ofx_proc_sgml(LibofxContext*, int, char**) ()
   from /usr/lib/libofx.so.4
#18 0x00007fffdf4bae57 in ofx_proc_file () from /usr/lib/libofx.so.4
#19 0x00007fffdf4b80ab in libofx_proc_file () from /usr/lib/libofx.so.4
#20 0x00007fffdf6e2abf in gnc_file_ofx_import ()
   from /usr/lib/gnucash/gnucash/libgncmod-ofx.so
#21 0x00007ffff0397a6e in g_closure_invoke () from /usr/lib/libgobject-2.0.so.0
#22 0x00007ffff03ad4d7 in ?? () from /usr/lib/libgobject-2.0.so.0
#23 0x00007ffff03ae996 in g_signal_emit_valist ()
   from /usr/lib/libgobject-2.0.so.0
#24 0x00007ffff03aef53 in g_signal_emit () from /usr/lib/libgobject-2.0.so.0
#25 0x00007ffff2b48a23 in ?? () from /usr/lib/libgtk-x11-2.0.so.0
#26 0x00007ffff0397a6e in g_closure_invoke () from /usr/lib/libgobject-2.0.so.0
#27 0x00007ffff03acda1 in ?? () from /usr/lib/libgobject-2.0.so.0
#28 0x00007ffff03ae996 in g_signal_emit_valist ()

[Zach Sadecki (zsadecki)]

Certain OFX files still work. I've also seen one give the out of range error while calling basic_string::copy. These files all used to work before upgrading to Maverick...

[Zach Sadecki (zsadecki)]

I've discovered what was causing the error to happen with certain OFX files. An empty tag that should contain a date seem to be the cause.

For example, this <LEDGERBAL> entry works fine:
<LEDGERBAL>
 <BALAMT>2518.57
 <DTASOF>20101016085012[-7:MST]
</LEDGERBAL>

While this one causes the out_of_range crash:
<LEDGERBAL>
 <BALAMT>2518.57
 <DTASOF />
</LEDGERBAL>

removing the DTASOF line altogether works fine, too..

[Luka Renko (lure)]

I have very similar issue, reported with skrooge:
https://bugs.kde.org/show_bug.cgi?id=255133

Root cause is wrong format of date in my bank's ofx file:

Sample dates:
          <DTSTART>201009099.9</DTSTART>

          <DTEND>201009099.9.</DTEND>

libofx code does not seems to be well written for such cases - from lib/ofx_utilities.cpp:

    if(ofxdate.size()>8) {
    /* if exact time is specified */
exact_time_specified = true;
      time.tm_hour=atoi(ofxdate.substr(8,2).c_str());
      time.tm_min=atoi(ofxdate.substr(10,2).c_str());
      time.tm_sec=atoi(ofxdate.substr(12,2).c_str());
    }

As a workaround, will fix this dates by hand...

[Zach Sadecki (zsadecki)]

Submitting a patch that will better handle malformed dates in OFX files.

[Zach Sadecki (zsadecki)]

After submitting a patch upstream I discovered that this problem has been in libofx for ~4 years, I guess they're just a little slow to fix it... http://sourceforge.net/tracker/?func=detail&aid=1454059&group_id=61170&atid=496353

[Benoit Grégoire (benoitg)]

Maybe so, but it HAS been fixed upstream.

[Benoit Grégoire (benoitg)]

To clarify:

- Original issue (empty date) has been resolved upstream a while ago, probably in November 2010.

- The one with malformed dates, originally reported in https://bugs.kde.org/show_bug.cgi?id=255133 has been fixed upstream today, with a patch inspired by this one.

- https://sourceforge.net/tracker/?func=detail&aid=1454059&group_id=61170&atid=496353 reported upstream is a different issue, and not a bug.

[Benoit Grégoire (benoitg)]

The merge request above would fix this bug.

[Michael Terry (mterry)]

This is just the debian/ changes for an SRU. Original work by Benoit.

[Martin Pitt (pitti)]

Accepted libofx into natty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Nobuto Murata (nobuto)]

libofx 1:0.9.4-0ubuntu0.1/natty-proposed works fine. It does not crash during importing test.ofx.xml.

[Zach Sadecki (zsadecki)]

I can also confirm that libofx 1:0.9.4-0ubuntu0.1 fixes the original problem I had with empty date strings.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libofx - 1:0.9.4-0ubuntu0.1

---------------
libofx (1:0.9.4-0ubuntu0.1) natty-proposed; urgency=low

  * New upstream release. Since 0.9.4, all changes were bugfixes and
    changes requested by packagers
    - Fixes "libofx crashes on OFX import in gnucash" (LP: #661809)
    - Fixes bad regression from previous patch for 629996 that truncated
      all strings after the first non-ascii character (LP: #769554)
  * Fix various lintian reported problems
  * Remove previous patch for 629996 which causes regressions.
    A better patch was integrated upstream.
  * Install the man pages for ofxdump and ofxconnect (LP: #771233)
 -- Benoit Gregoire <email address hidden> Tue, 26 Apr 2011 09:33:38 -0400

[Martin Pitt (pitti)]

Copied to oneiric as well.