akonadi dead

[SA]

I rebooted my computer (at which point it was fine) on reboot nothing that requires akonadi works.

I cannot access any of my email or my calendar which is pretty important to me.

Everything hangs on "starting Akonadi server..."

In the akonadi server conf I eventually get a load of errors (below):

I'm guessing the DB is screwed and that likely means I'm screwed? whatever the problem the consequence seems pretty drastic and the fix beyond what is reasonable so this should (at the very least) be more robust!

Any suggestions as to what I do next?

Akonadi Server Self-Test Report
===============================

Test 1: SUCCESS
--------

Database driver found.
Details: The QtSQL driver 'QMYSQL' is required by your current Akonadi server configuration and was found on your system.

File content of '/home/XXXX/.config/akonadi/akonadiserverrc':
[%General]
Driver=QMYSQL
SizeThreshold=4096
ExternalPayload=false
...

Test 2: SUCCESS
Test 3: SUCCESS
Test 4: SUCCESS
Test 5: ERROR
--------

MySQL server log contains errors.
Details: The MySQL server error log file &apos;<a href='/home/XXXX/.local/share/akonadi/db_data/mysql.err'>/home/XXXX/.local/share/akonadi/db_data/mysql.err</a>&apos; contains errors.

File content of '/home/XXXX/.local/share/akonadi/db_data/mysql.err':
130703 0:34:02 [Note] Plugin 'FEDERATED' is disabled.
130703 0:34:02 InnoDB: The InnoDB memory heap is disabled
130703 0:34:02 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130703 0:34:02 InnoDB: Compressed tables use zlib 1.2.7
130703 0:34:02 InnoDB: Using Linux native AIO
130703 0:34:02 InnoDB: Initializing buffer pool, size = 80.0M
130703 0:34:02 InnoDB: Completed initialization of buffer pool
130703 0:34:02 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 2607872694
130703 0:34:02 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 2613115392
InnoDB: Doing recovery: scanned up to log sequence number 2613499840
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 1480 row operations to undo
InnoDB: Trx id counter is C6ED00
130703 0:34:02 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 8031, file name ./mysql-bin.000021
InnoDB: Cleaning up trx with id C6BF11
130703 0:34:03 InnoDB: Waiting for the background threads to start
130703 0:34:04 InnoDB: 5.5.31 started; log sequence number 2613499840
130703 0:34:04 InnoDB: Assertion failure in thread 140228725671680 in file trx0purge.c line 840
InnoDB: Failing assertion: purge_sys->purge_trx_no <= purge_sys->rseg->last_trx_no
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
23:34:04 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=256
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 568168 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
130703 0:34:04 [ERROR] Native table 'performance_schema'.'events_waits_current' has the wrong structure
130703 0:34:04 [ERROR] Native table 'performance_schema'.'events_waits_history' has the wrong structure
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7f899caf2e59]
130703 0:34:04 [ERROR] Native table 'performance_schema'.'events_waits_history_long' has the wrong structure
130703 0:34:04 [ERROR] Native table 'performance_schema'.'setup_consumers' has the wrong structure
130703 0:34:04 [ERROR] Native table 'performance_schema'.'setup_instruments' has the wrong structure
130703 0:34:04 [ERROR] Native table 'performance_schema'.'setup_timers' has the wrong structure
130703 0:34:04 [ERROR] Native table 'performance_schema'.'performance_timers' has the wrong structure
130703 0:34:04 [ERROR] Native table 'performance_schema'.'threads' has the wrong structure
130703 0:34:04 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_thread_by_event_name' has the wrong structure
130703 0:34:04 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_instance' has the wrong structure
/usr/sbin/mysqld(handle_fatal_signal+0x3d8)[0x7f899c9eb8c8]
130703 0:34:04 [ERROR] Native table 'performance_schema'.'events_waits_summary_global_by_event_name' has the wrong structure
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfbd0)[0x7f899b772bd0]
130703 0:34:04 [ERROR] Native table 'performance_schema'.'file_summary_by_event_name' has the wrong structure
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f899adca037]
130703 0:34:04 [ERROR] Native table 'performance_schema'.'file_summary_by_instance' has the wrong structure
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f899adcd698]
130703 0:34:04 [ERROR] Native table 'performance_schema'.'mutex_instances' has the wrong structure
130703 0:34:04 [ERROR] Native table 'performance_schema'.'rwlock_instances' has the wrong structure
130703 0:34:04 [ERROR] Native table 'performance_schema'.'cond_instances' has the wrong structure
130703 0:34:04 [ERROR] Native table 'performance_schema'.'file_instances' has the wrong structure
130703 0:34:04 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.31-0ubuntu0.13.04.1' socket: '/home/XXXX/.local/share/akonadi/socket-lunesta/mysql.socket' port: 0 (Ubuntu)
/usr/sbin/mysqld(+0x596835)[0x7f899cb9b835]
/usr/sbin/mysqld(+0x596c39)[0x7f899cb9bc39]
/usr/sbin/mysqld(+0x6586ef)[0x7f899cc5d6ef]
/usr/sbin/mysqld(+0x64f15b)[0x7f899cc5415b]
/usr/sbin/mysqld(+0x597d89)[0x7f899cb9cd89]
/usr/sbin/mysqld(+0x5898bc)[0x7f899cb8e8bc]
/usr/sbin/mysqld(+0x58dad3)[0x7f899cb92ad3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e)[0x7f899b76af8e]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f899ae8ce1d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Test 6: SUCCESS
--------
Test 9: SUCCESS
Test 10: ERROR
--------

Akonadi control process not registered at D-Bus.
Details: The Akonadi control process is not registered at D-Bus which typically means it was not started or encountered a fatal error during startup.

Test 11: ERROR
--------

Akonadi server process not registered at D-Bus.
Details: The Akonadi server process is not registered at D-Bus which typically means it was not started or encountered a fatal error during startup.

Test 12: SUCCESS
Test 13: SUCCESS
Test 14: SKIP
Test 15: ERROR
Test 16: ERROR
--------

Current Akonadi server error log found.
Details: The Akonadi server reported errors during its current startup. The log can be found in <a href='/home/XXXX/.local/share/akonadi/akonadiserver.error'>/home/XXXX/.local/share/akonadi/akonadiserver.error</a>.

File content of '/home/XXXX/.local/share/akonadi/akonadiserver.error':
Database process exited unexpectedly during initial connection!
executable: "/usr/sbin/mysqld"
arguments: ("--defaults-file=/home/XXXX/.local/share/akonadi/mysql.conf", "--datadir=/home/XXXX/.local/share/akonadi/db_data/", "--socket=/home/XXXX/.local/share/akonadi/socket-lunesta/mysql.socket")
stdout: ""
stderr: ""
exit code: 1
process error: "Process operation timed out"

0: akonadiserver() [0x418114]
1: akonadiserver() [0x418541]
2: /lib/x86_64-linux-gnu/libc.so.6(+0x370b0) [0x7fb4a4c1e0b0]
3: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7fb4a4c1e037]
4: /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7fb4a4c21698]
5: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_Z17qt_message_output9QtMsgTypePKc+0x122) [0x7fb4a66d15c2]
6: akonadiserver() [0x41a3eb]
7: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_ZN9QIODevice5writeEPKcx+0xb4) [0x7fb4a676b9f4]
8: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(+0x11662f) [0x7fb4a677662f]
9: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_ZN11QTextStreamD1Ev+0x3b) [0x7fb4a677ebcb]
10: akonadiserver() [0x486d2a]
11: akonadiserver() [0x41b0b7]
12: akonadiserver() [0x41ce65]
13: akonadiserver() [0x41e487]
14: akonadiserver() [0x411f43]
15: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fb4a4c08ea5]
16: akonadiserver() [0x4128d1]

Test 17: ERROR
same as 16
18: SUCCESS
19: SUCCESS

[SA (superaorta)]

Now kmail won't even start:

The error was: Failed to fetch the resource collection

Failed to fetch the resource collection."
kmail2(9986)/libakonadi Akonadi::SpecialCollectionsRequestJobPrivate::resourceScanResult: Failed to request resource "akonadi_mixedmaildir_resource_0" : "Failed to fetch the resource collection."
kmail2(9986)/libakonadi Akonadi::SpecialCollectionsRequestJob::slotResult: Failed SpecialCollectionsRequestJob::slotResult "Failed to fetch the resource collection."

WTF am I supposed to do now? Id really rather not lose everything

[Daniel Letzeisen (dtl131)]

You may want to try this before giving up completely:

http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html

[SA (superaorta)]

I saw this but...

Even if I knew what database was corrupted (and where it was), what would I do with the data once the tables had been dumped? I mean it isn't apparent that I would be able to reconstruct the akonadi databases in any meaningful way to get kmail and kpim working again with my data.

[SA (superaorta)]

I've resorted to deleting everything in my home directory containing "akonadi" once all the directories were gone stuff started to work again. I had to reconfigure kmail to find my mail folders and I think I've lost my calender (since I'm using it as a front end for google calender I think I can get most of this back).

I have no idea what the problem was but it is a bug that so much could be so fragile and that the recovery is so difficult.