Unable to Connect to Database Error with Kobo

[Scott Brown]

Whenever I open Calibre after connecting to my Kobo Aura 3.18.0 using USB on OS X 10.11.1 I get the error below and am unable to communicate with the device.

calibre, version 2.45.0
ERROR: Error: Error communicating with device

unable to open database file

Traceback (most recent call last):
  File "site-packages/calibre/gui2/device.py", line 88, in run
  File "site-packages/calibre/gui2/device.py", line 504, in _books
  File "site-packages/calibre/devices/kobo/driver.py", line 1683, in books
OperationalError: unable to open database file

[Kovid Goyal (kovid)]

Changing the component for this bug.

 assignee davidfor
 tag kobo-driver
 status triaged

[David Forrester (davidfor)]

That sort of error usually means the there is a problem with the database on the device. It is corrupted in such a way that the it cannot be opened and read properly. If this happens, the device can keep working as it is using an in memory copy of the data. But, it won't be able to save the data and restarting the device will revert to old data. This is usually seen as losing the reading position or old books showing on the home screen. You can check the database using any SQLite database management tool, or the Kobo Utilities plugin in calibre.

If the database is corrupt, unless you have a backup of the database, you will need to signout of your Kobo account on the device. This will generate a new database and take you through the setup process.

But, as this is on a Mac, there is another possibility. For some reason, Macs can occasionally mount the devices readonly. I have no idea why, but it happens.

[Jeff Walker (jeffwalker99)]

I'm having the exact same problem with my Kobo Aura HD. It started right after updating to El Capitan. Even directly after restarting the Kobo I get the "unable to open database file" error message.

[Jeff Walker (jeffwalker99)]

I should add that the problem got worse after the El Capitan 10.11.2 update. Prior to that update Calibre could occasionally open the database, but after the 10.11.2 update I've not yet been able to get Calibre to open the database.

[David Forrester (davidfor)]

That suggests that something in the El Captain update has changed how the USB drives are handled. Unfortunately, I don't have a Mac to test with. But, does the Kobo desktop application work with the device? Can you access the database on the device using an SQLite database manager?

[Jeff Walker (jeffwalker99)]

I think the problem might be caused by the Mac not ejecting the kobo device and the micro SD card properly. When you eject one, the other gets ejected as well and the Mac complains that the device hasn't been ejected properly.

The Kobo desktop application seems to work okay. Today I used it to add a book to my library directly after Caliber said it couldn't connect to the database. Also the Kobo desktop application synced with the device okay.

When I used the Kobo application's eject button, the Micro SD card ejected okay but the Mac put up a message saying the Kobo device itself hadn't been properly ejected.

The book I added to my library is now showing up on my Kobo.

Directly after quitting the Kobo desktop device I reconnected the Kobo and ran Calibre. I got the same "unable to open database file" error message.

Directly after quitting Calibre I used the Firefox SQLite Manager to open the' KoboReader.sqlite' database directly from the Kobo. It did that without any error. I ran its "Check Integrity" option and when it finished it said 'Checked Integrity of the database using "PRAGMA integrity_check". Result: OK'.

It's a puzzler.

[Jeff Walker (jeffwalker99)]

Okay, I've deleted the 'KoboReader.sqlite' file on the Kobo and copied across an old backup.

Even though when I ejected the device and card the Mac complained as usual that the Kobo device itself hadn't been properly ejected, directly afterwards I reconnected the Kobo, ran Caliber and it successfully opened the database. This I think pretty much confirms that the problem is a corrupted database.

I then used Calibre to copy one new book to the Kobo, quit Calibre when it had finished doing that, and ejected the card and device. As usual, the Mac complained that the Kobo device itself hadn't been properly ejected.

I can see that the book I copied to the Kobo is there.

So I immediately reconnected the Kobo, ran Calibre and ... wait for it ... "Error communicating with device: unable to open database file".

To me, this suggests that the database is being corrupted by Calibre. What say you?

[Jeff Walker (jeffwalker99)]

Or perhaps Calibre isn't understanding the latest 3.19.x database structure properly? (The old backup database I used was from the previous 3.18.x firmware.)

[Jeff Walker (jeffwalker99)]

It gets even more puzzling. I just replaced the 'KoboReader.sqlite' file on the Kobo with the same backup as before, and now Calibre can't open it. It's the exact same file.

[Jeff Walker (jeffwalker99)]

The Epic Saga of the Kobo Aura HD continues...

Starting at the position where Calibre can't open the database, today I tried a different procedure for copying a book to the Micro SD card.

First I switched off the Kobo and removed the Micro SD card.

Using an adapter I then plugged the card into my iMac (it has a card slot).

Then I copied the book from my iMac to the Micro SD card.

Then I ejected and removed the Micro SD card from the iMac.

Then I inserted the Micro SD card back into the Kobo and switched on the Kobo.

The Kobo booted up and the book I copied across was there.

Then I connected the Kobo to my iMac and ran Calibre and ... wait for it ... Calibre was able to open the database.

So I used Calibre to remove the new book I copied to the Micro SD card, quit Calibre and ejected the Micro SD card from the iMac. As usual, the Kobo device also ejected itself and the iMac complained about it not being ejected properly.

Then I reconnected the Kobo to the iMac, waited for the device and the card to mount, ran Calibre and ... wait for it ... "Error communicating with device: unable to open database file".

So I immediately used the Firefox SQLite Manager to open the' KoboReader.sqlite' database directly from the Kobo and ran the "Check Integrity" option; when it finished it said 'Checked Integrity of the database using "PRAGMA integrity_check". Result: OK'.

I think this suggests that the problem is not caused by a corrupted database. I think it is all now pointing towards the problem being caused by the iMac not ejecting the Kobo device properly.

[Kovid Goyal (kovid)]

If I were you, I'd try resetting the Kobo and trying different USB
cables/ports. Kobo firmware has a reputation for flakiness. You should
also try using the eject device function inside calibre to eject your
device (right the device icon in the calibre toolbar and choose eject).

[Jeff Walker (jeffwalker99)]

Thanks for your feedback.

I should have mentioned before that I also tried different USB cables and different USB ports in order to rule out those as a possible cause.

The eject device function in Calibre results in the same problem with the iMac complaining about a device not being ejected properly.

The results of my experimentation suggest that a factory reset isn't going to make any difference. With >12,000 books nicely sorted into collections and series over the past three years (thank you Project Gutenberg) it's not something I want to do on the off chance that it might help.

I think I'll try taking this issue to Kobo support, but as the normal way of adding books to a Micro SD card works with no problems, they may just send me back here or to Apple.

Thanks again for your feedback.

[David Forrester (davidfor)]

12000 books? Exactly how big is the database? For about 2000 books, I'm at about 85MB. Six times that puts it over 400MB. Maybe it us something related to the size.

You can test the factory reset safely if you take a full copy of the user partition from the device. After that, do the factory reset and test with a smaller number of books. Then you can decide whether to restore the saved data or start again. If you do this, make sure you take the SD card out. If it has most of your books, you don't want to have them processed into the library before doing the tests.

Another test is to connect without the card. The database gets opened twice, once for the main memory and once for the card. I would need to see a debug log to work out which it was. They shouldn't be different, but there is a chance that the the first open has left something in a bad state.

And have you tried another machine? Checking with an older iMac would help to see if it was this version of El Capitan. A Windows or Linux machine would help to see if it was the device.

Also, trying immediately after rebooting the iMac and the ereader might give the best chance of everything working.

[Jeff Walker (jeffwalker99)]

Thanks for your feedback, I'll try some of that and get back to you.

You'll appreciate that the library didn't go from 0 to 12,000 in 6 seconds :) it has been gradually built up over a period of some years. In fact, it has been bigger in the past (>14K) but I recently pruned it. Currently the database is just over 1GB.

Prior to the 3.19.x firmware update (i.e. with the 3.18.x firmware) the problem was intermittent but could be fixed by doing something on the Kobo that changed the database, such as adding a book to a collection for example. After doing that and reconnecting the device to the iMac, Calibre could open the database again. But after the 3.19.x firmware update that little trick no longer works.

I think the best clue is that when I add a book to the Micro SD card via the slot in the iMac, the Kobo spots this and thus adds the book to then database, and directly after that Calibre can open the database once more. But then ejecting the card from the iMac causes the mounted device to be ejected too, followed by the error message about it not being ejected properly, and directly after that Calibre can no longer open the database.

I don't think the problem will happen without a Micro SD card inserted because all the evidence says the problem is related to how the iMac is ejecting the mounted card and device. But I'll try it to make sure.

I'll also try some of your other suggestions and let you know how it goes.

[Jeff Walker (jeffwalker99)]

Here's the result of one of the tests you suggested.

Directly prior to test = Calibre cannot open the database.

Test...

Turn off Kobo.
Remove Micro SD card.
Turn on Kobo and wait for it to boot up.
Connect Kobo to iMac.
Run Calibre.

Result = No problem, Calibre opens the database.

Extended test...

Quit Calibre.
Eject mounted Kobo device from iMac. (Ejects properly, no complaints from the iMac.)
Wait for Kobo to display its home page.
Turn off Kobo.
Insert Micro SD card.
Turn on Kobo and wait for it to boot up.
Connect Kobo to iMac.
Run Calibre.

Result = Calibre cannot open the database.

I should emphasise that the content of the Micro SD card has not changed at all between the test and the extended test.

My conclusions...

There's something about the current state of the Micro SD card that Calibre doesn't like. I don't know what that might be. I can read from and write to the card without problems.

This is why I think the best clue is that when I add a book to the Micro SD card via the slot in the iMac, upon inserting it back into the Kobo it spots this and thus adds the book to the database, and directly after that Calibre can open the database once more.

I can't currently test this with fewer books on the Micro SD card; I don't have a spare one so I'll need to buy another before I can do that test.

[Jeff Walker (jeffwalker99)]

Okay. I popped down the road and bought a brand new 32GB Micro SD card.

Nuvver test...

Turn off Kobo.
Remove Micro SD card that has 12K books on it.
Insert brand new totally blank Micro SD card.
Turn on Kobo and wait for it to boot up.
Do a factory reset.
Connect Kobo to iMac.
Run Calibre.

Result = Calibre cannot open the database.

Quit Calibre.
Eject card and Kobo device from iMac.
Turn off Kobo.
Remove brand new totally blank Micro SD card.
Connect Kobo to iMac.
Run Calibre.

Result = No problem, Calibre opens the database.

End of Nuvver test.

I think this confirms that this is a Calibre issue?

[Kovid Goyal (kovid)]

That error comes from the sqlite library failing to open the database
file, not calibre code. The issue is most definitely not in calibre
code. Given that the sqlite library is one of the most stable peices of
software around, the issue is almost certainly in device firmware.

Copy the KoboReader.sqlite off the device, and do the following:

Run:

calibre-debug -c "import sqlite3; sqlite3.connect('/path/to/copied/KoboReader.sqlite')"

This is the exact code that is failling in the driver.

You will either

1) get an error, in which case you will not be able to
open the sqlite file with other software either, indicating the file is
corrupted. Presumably the kobo firmware rebuilds/repainrs the file on
disconnect.

2) Or it will open, indicating that the problem is with reading the file
over the USB connection -- which means the device firmware does not
implement the USB Mass storage spec properly.

[Jeff Walker (jeffwalker99)]

I don't think you have understood the issue.

In a nutshell, when the Kobo Aura HD has a Micro SD card inserted Calibre can't open the database. Remove the Micro SD card and Calibre can open the database.

In a situation where Calibre says it cannot open the database, using Firefox SQLite Manager I can open the database directly on the device. There has been no disconnection when I do this.

Why can Firefox SQLite Manager open the database when Calibre cannot?

Are you saying this must be a fault in the device firmware?

[Kovid Goyal (kovid)]

yes.

[David Forrester (davidfor)]

Now, you've triggered a memory. Someone reported a similar problem on MR last month. For some reason the Mac was reporting the SD card as the main memory. Could you run calibre in debug mode and post the log? From that, we should be able to see if the same thing is happening.

[Jeff Walker (jeffwalker99)]

Here you go...

calibre Debug log
calibre 2.46 isfrozen: True is64bit: True
Darwin-15.2.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '15.2.0', 'Darwin Kernel Version 15.2.0: Fri Nov 13 19:56:56 PST 2015; root:xnu-3248.20.55~2/RELEASE_X86_64')
Python 2.7.9
OSX: ('10.11.2', ('', '', ''), 'x86_64')
Successfully initialized third party plugins: Kobo Utilities
Starting up...
DEBUG: 0.0 KoboUtilites::action.py - loading translations
DEBUG: 0.0 KoboUtilites::dialogs.py - loading translations
DEBUG: 0.0 KoboUtilites::action.py - loading translations
DEBUG: 2.2 No Kobo Touch, Glo or Mini appears to be connected
DEBUG: 2.2 rebuild_menus - self.supports_ratings=None, self.supports_tiles=None
unable to open database file

Traceback (most recent call last):
  File "site-packages/calibre/gui2/device.py", line 87, in run
  File "site-packages/calibre/gui2/device.py", line 503, in _books
  File "site-packages/calibre/devices/kobo/driver.py", line 1683, in books
OperationalError: unable to open database file

[David Forrester (davidfor)]

How did you get that? There should be a lot more messages than that. You can either run "calibre-debug -g" from the command line and capture the output, or restart calibre in debug mode. That is done by right clicking (or equivalent on a Mac) on the preferences button and selecting "restart in debug mode". Calibre restarts and displays a message about the log. Then plugin the device and wait for the error. When that happens, close calibre and the log should be displayed. That should have a lot of debug messages.

[Jeff Walker (jeffwalker99)]

I connected the Kobo to the iMac (which has the Micro SD card inserted) and ran Calibre. As usual it said it "Error communicating with device". Then I selected "Restart in debug mode" from the Preferences menu. Calibre restarted and its spinner at bottom right span until is said "Error communicating with device" at which point I quitted Calibre and what I posted was the complete content of the error log that opened after Calibre had quit.

Later I'll try a different procedure. I'll not connect the Kobo to begin with. I'll run Calibre, select "Restart in debug mode" from the Preferences menu, and when Calibre has opened again I'll then connect the Kobo. I'll let you know how that goes.

[Jeff Walker (jeffwalker99)]

Here you go.

BTW, to get this to work I had to delete the 'driveinfo.calibre' and 'metadata.calibre' files on the Micro SD card, otherwise I just got the "Error communicating with device" message all the time and that short debug log.

calibre Debug log
calibre 2.47 isfrozen: True is64bit: True
Darwin-15.2.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '15.2.0', 'Darwin Kernel Version 15.2.0: Fri Nov 13 19:56:56 PST 2015; root:xnu-3248.20.55~2/RELEASE_X86_64')
Python 2.7.9
OSX: ('10.11.2', ('', '', ''), 'x86_64')
Successfully initialized third party plugins: Kobo Utilities
Starting up...
DEBUG: 0.0 KoboUtilites::action.py - loading translations
DEBUG: 0.0 KoboUtilites::dialogs.py - loading translations
DEBUG: 0.0 KoboUtilites::action.py - loading translations
DEBUG: 2.2 No Kobo Touch, Glo or Mini appears to be connected
DEBUG: 2.2 rebuild_menus - self.supports_ratings=None, self.supports_tiles=None
Started up in 6.27 seconds with 12329 books
DEBUG: 24.2 KoboUtilities:_on_device_connection_changed - self.plugin_device_connection_changed.__class__: <type 'PyQt5.QtCore.pyqtBoundSignal'>
DEBUG: 24.2 Methods for self.plugin_device_connection_changed: ['__call__', '__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'connect', 'disconnect', 'emit', 'signal']
Job: 1 Get device information started
DeviceJob: 1 Get device information done, calling callback
DeviceJob: 1 Get device information callback returned
Job: 2 Set library information started
DeviceJob: 2 Set library information done, calling callback
DeviceJob: 2 Set library information callback returned
Job: 3 Get list of books on device started
DEBUG: 0.0 KoboTouch:books - oncard='None'
DEBUG: 0.0 KoboTouch:books - oncard='None', prefix='/Volumes/KOBOeReader/'
DEBUG: 0.0 Kobo device: Kobo Aura HD
DEBUG: 0.0 Version of driver: (2, 1, 11) Has kepubs: True
DEBUG: 0.0 Version of firmware: (3, 19, 5613) Has kepubs: True
DEBUG: 0.0 Firmware supports cover image tree: True
DEBUG: 0.1 KoboTouch:books - opts.extra_customization= [u'#koboshelves', True, False, True, False, True, False, False, False, True, True, False, u'']
DEBUG: 0.1 KoboTouch:books - prefs['manage_device_metadata']= on_send
DEBUG: 0.1 KoboTouch:books - set_debugging_title to ''
DEBUG: 0.1 KoboTouch:books - length bl=0
DEBUG: 0.1 KoboTouch:books - length bl after sync=5
DEBUG: 0.1 KoboTouch:books - reading device database
DEBUG: 0.2 Database Version=125
DEBUG: 0.2 KoboTouch:books - shelf list: [u'Reference', u'Fiction', u'Non-Fiction']
DEBUG: 0.2 KoboTouch:books - query= SELECT Title, Attribution, DateCreated, ContentID, MimeType, ContentType, ImageID, ReadStatus, ___ExpirationStatus, FavouritesIndex, Accessibility, IsDownloaded, Series, SeriesNumber, ___UserID, ExternalId FROM content where BookID is Null and ((Accessibility = -1 and IsDownloaded in ('true', 1 )) or (Accessibility in (1,2) and IsDownloaded in ('true', 1)) ) and not ((___ExpirationStatus=3 or ___ExpirationStatus is Null)...

[Jeff Walker (jeffwalker99)]

So I've discovered what I need to do in order for Calibre to be able to open the database when the Kobo has a Micro SD card inserted.

1. Delete the 'metadata.calibre' file on the Micro SD card.

2. Eject the mounted Kobo device and Micro SD card from the iMac.

3. Reboot the iMac.

4. Connect the Kobo (with Micro SD card inserted) to the iMac.

5. Run Calibre.

And now Calibre will open the database.

Subsequently, if I quit Calibre and eject the mounted Kobo device and Micro SD card from the iMac, when I re-connect the Kobo and run Calibre, then Calibre cannot open the database.

To get it to once again open the database I have to go through the above procedure, exactly.

The deleting of the the 'metadata.calibre' file on the Micro SD card might be a clue; if I go through the same procedure but don't delete the 'metadata.calibre' file from the Micro SD card then when I run Calibre it cannot open the database.

Also, the iMac must be rebooted else the procedure doesn't work either.

Merry Christmas!

[David Forrester (davidfor)]

That's "interesting".

The metadata.calibre file is read just before database is opened. That shouldn't have an affect the database opening, but that's the only thing I can think of. Something about reading this file is preventing the database from being opened. The problem is, I can't think of what. If the file couldn't be read, the error should have been raised earlier. I can only think that reading the file is leaving the driver in a state that the next file opened, the database, can't be opened. But, I can't see how that could happen. The Kobo devices are just presenting the SD card's partition for access, so it would almost have to be in the OSes MSD driver or FAT32 support.

Things that come to mind while writing this are:

- How big are the metadata.calibre and KoboReader.sqlite files?
- How big is the SD card?
- Have you mounted the card directly in a computer and checked it was OK?

The only other thing I can think of is that the debug logs look to come from restarting in debug mode. Can you try running "calibre-debug -g" from the terminal? I don't know if the output will be different, but I'm hoping it will.

[Jeff Walker (jeffwalker99)]

After the Calibre v2.48 update, so far this problem has not manifested itself. Fingers crossed.

[Jeff Walker (jeffwalker99)]

Spoke too soon. It happened again today. Sigh.

[David Forrester (davidfor)]

I knew I should have kept my fingers crossed :)

Something I just thought of is to run the "Debug device detection" and to know exactly what calibre is seeing. This on the "Miscellaneous" page of the calibre preferences. But, you will need to disable the KoboTouch first.

p.s. Each time I see your name, I add "Jerry" in front of it. And seeing it yesterday reminded me that I hadn't listened to "Mr. Bojangles" for a while.

[Jeff Walker (jeffwalker99)]

I'm going to speak too soon again. :)

After the El Capitan 10.11.3 update this problem has gone away.

Fingers crossed.

[Jeff Walker (jeffwalker99)]

...and it happened again today.

Ah well, it was nice while it lasted. :'(

[Jeff Walker (jeffwalker99)]

Yahay! Look what I see in the Calibre v2.56 release notes...

• Fix for OS X upgrade 10.11.4 breaking detection of Kobo devices on some systems

[Jeff Walker (jeffwalker99)]

Nope. The problem remains.

But the problem is intermittent so I think it must be an El Capitan thing to do with how the devices are being ejected.

Ho hum.

[Julian (myfakeacc-4)]

This is definitly a problem with the latest OSX (10.11.4 or even 10.11.3) and Calibre.
I have my Aura HD since ~fall 2013 and I never upgraded its firmware but I always upgraded OSX and Calibre. I have never seen this error before and suddenly (after roughly 5 months of not using it) I can no longer connect my Aura HD to Calibre because of this error.

To be honest I have no idea what do to now. As I managed my collection (mainly comics and manga in self-created CBZ format) completely with Calibre I'm totally lost now and unable to update it with further stuff to read.

[Julian (myfakeacc-4)]

I'd like to add that before these ~5 months of not using it I frequently and periodically connected my Aura HD to Calibre, roughly every 1-2 weeks, always using the latest OSX and Calibre but always with the old firmware and it is only now that it no longer works.

[Julian (myfakeacc-4)]

I can confirm that the workaround in #25 works for me even without step 3.) on my MacBookPro running 10.11.4.
However, Calibre looses a lot of information: around half of the books on my ebook reader are no longer recognized as "in Library" even though they are the exact same version (unmodified) from the library. It looks totally random: even from the same series/collection some books are recognized and some are not.

[Jeff Walker (jeffwalker99)]

For me the problem is intermittent. For example...

1. I'll connect the Aura HD to the Mac and wait for the devices to show on the desktop.

2. Then I run Calibre.

3. Calibre tries to connect to the database, but fails.

So I unmount the devices, quit Caliber and go to step 1 again.

Sometimes I need to do this three or four times before Caliber eventually manages to connect to the database.

[Jeff Walker (jeffwalker99)]

This is interesting...

I uninstalled the Kobo Utilities plugin because I wanted to remove it as a possible cause of my problem.

After I uninstalled the Kobo Utilities plugin I didn't have any problems. It was a couple of weeks problem free.

I reinstalled the Kobo Utilities plugin and pretty soon the same intermittent problem was back; sometimes I had the problem every day, other times it was two or three days before it happened again.

So I uninstalled the Kobo Utilities plugin again and the problem went away again. Just over a week now problem free.

I think all I was using the Kobo Utilities plugin for was to backup the database every day. I can live without that.

[Julian (myfakeacc-4)]

As for me, I don't have and never had this plugin installed though.