Kupfer reads default Firefox bookmarks

[Andy Holmes]

Using c19.1 on Ubuntu's Karmic Koala, Kupfer only displays the default Firefox bookmarks (ie. "Get Bookmark Add-ons", "Getting Started", etc). I delete those bookmarks, and yet my bookmarks are not displayed at all.

[X (u78qir8a9-deactivatedaccount)]

Thank you for reporting bugs in Kupfer!

If you quit kupfer properly (run the Quit object in Kupfer), then start kupfer anew from a terminal, in debug output mode (kupfer --debug), you will see more output that can help us understand this issue. Please do this if you can: find firefox, select the action Rescan (only available in debug mode), and paste the debug output here. Just paste all of it if you want.

I have a guess about the issue though, and it is that it is partly a firefox bug. The problem is that firefox writes JSON data to its bookmarks file that none of the two JSON modules that Kupfer supports can read. Firefox writes a new bookmarks file every day, so it might change from day to day if it's invalid or not! Here is debug output from my test setup, on an occation when it fails:

D [kupfer.plugin.firefox] BookmarksSource: Parsing /home/ulrik/.mozilla/firefox/u7ssei2l.default/bookmarkbackups/bookmarks-2010-01-09.json
Error [kupfer.plugin.firefox] BookmarksSource: unterminated array starting at position 329
D [kupfer.plugin.firefox] BookmarksSource: Parsing /home/ulrik/.mozilla/firefox/u7ssei2l.default/bookmarks.html
[kupfer.plugin.firefox] BookmarksSource: Loaded 644 items

The error 'unterminated array starting at position 329' that you see is a JSON parsing error in one of Kupfer's support libraries. I blame Firefox for this invalid data. Maybe you have almost the same kind of output on your setup.

What happens in Kupfer is that it tries to fall back to the bookmarks.html file. However, you must configure Firefox to write bookmarks to the bookmarks.html file as backup. Maybe google or Karol can tell you how to do that (I don't know off hand).

[X (u78qir8a9-deactivatedaccount)]

2010/1/9, Andrew Holmes <email address hidden>:
> Yes, I think you're probably right, but my error output is a little
> different:
>
> So I assume this is a bug in a python module? Should I/we file a bug
> upstream to have this fixed or is the bookmarks.html workaround the best fix
> right now?
>
> Regards,
> Andy

I really think Firefox is outputting JSON that most libraries reject
as invalid. It commonly uses trailing commas or unterminated array
[...] or dict {..} objects.

See for example: http://code.google.com/p/simplejson/issues/detail?id=44

Here is a comparison of Python JSON modules and their behavior on
common invalid JSON:

http://blog.hill-street.net/?p=7 (Unavailable.. google cache link below)
http://74.125.77.132/search?q=cache:kOIOawEQNjIJ:blog.hill-street.net/%3Fp%3D7+python+json+%22Expecting+object%22&hl=en&strip=1

The trailing comma in array is rejected by 4 out of 6 listed modules.
Kupfer uses 'cjson' (column 1 in the table, see above link) if
available, then tries 'json' (almost the same as 'simplejson', column
3)

Workaround 0

Use bookmarks.html. We could output an error message (better, in the
future, an error dialog?) to instruct the user about this.

Workaround 1

So it might be possible to test out a Python json module that works
with Firefox' output habits. It must be tested out, and then Kupfer
can support only that module for firefox bookmarks. One module is
included with Python 2.6's standard library, called 'json'; it fails
to parse this output. The other Kupfer-supported alternative 'cjson'
likewise.

Workaround 2

Parse successively older .json files. There is usually five of them,
and one of them might work.

[Andy Holmes (andyholmes)]

Two ways to make Firefox save HTML instead of JSON:

1) Open the bookmark manager (Ctrl-Shift-O) and select "Import and Backup => Export HTML", then save to "~/.mozilla/firefox/*.default/bookmarks.html

2) Type about:config in the address bar, then use the filter bar to search for "browser.bookmarks.autoExportHTML" and set this to "true" by double clicking or selecting and pressing enter

[Oier Mees (oier)]

I can confirm this issue and also that Andrews workaround works.
I got the following error:
D [kupfer.plugin.firefox] BookmarksSource: Reading history from /home/oier/.mozilla/firefox/k8cbgs0a.default/places.sqlite
D [kupfer.plugin.firefox] BookmarksSource: Parsing /home/oier/.mozilla/firefox/k8cbgs0a.default/bookmarkbackups/bookmarks-2010-05-22.json
Exception in [kupfer.plugin.firefox] BookmarksSource:
Traceback (most recent call last):
  File "/usr/share/kupfer/kupfer/plugin/firefox.py", line 94, in get_items
    json_bookmarks = list(self._get_ffx3_bookmarks(fpath))
  File "/usr/share/kupfer/kupfer/plugin/firefox.py", line 65, in _get_ffx3_bookmarks
    bookmarks = firefox3_support.get_bookmarks(fpath)
  File "/usr/share/kupfer/kupfer/plugin/firefox3_support.py", line 21, in get_bookmarks
    root = json_decoder(content)
  File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/json/decoder.py", line 319, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.6/json/decoder.py", line 336, in raw_decode
    obj, end = self._scanner.iterscan(s, **kw).next()
  File "/usr/lib/python2.6/json/scanner.py", line 55, in iterscan
    rval, next_pos = action(m, context)
  File "/usr/lib/python2.6/json/decoder.py", line 183, in JSONObject
    value, end = iterscan(s, idx=end, context=context).next()
  File "/usr/lib/python2.6/json/scanner.py", line 55, in iterscan
    rval, next_pos = action(m, context)
  File "/usr/lib/python2.6/json/decoder.py", line 219, in JSONArray
    raise ValueError(errmsg("Expecting object", s, end))
ValueError: Expecting object: line 1 column 10117 (char 10117)
D [kupfer.plugin.firefox] BookmarksSource: Parsing /home/oier/.mozilla/firefox/k8cbgs0a.default/bookmarks.html
[kupfer.plugin.firefox] BookmarksSource: Loaded 42 items
Because of Andrews workaround you can see that it loads 42 items.
Another problem was that the first 29 items were from the history and my bookmarks where the last items. I think that the plugin should give you the option to disable history, my workaround has been to comment the hisoty related function and calls in the firefox plugin and reset the history list in the get_items function (self._history = []).

[Neosano (neosanor)]

What about recently used sites? Why kupfer doesn't show them?