backup_gmail.py fails with "No such file or directory: '/data/gmail-backup/(added/by'"

[Thomas Tanghus]

Hi

First: Thanks for a great tool, it's really needed and I'm sure the following is either a fault from my side or an easy fixable bug.

First I ran a full backup using gui.py to a writable partition with settings noincremental and not to keep read/unread (maybe give a little warning if people doesn't check that - it was a mistake that I didn't check it. Well, my bad ;)

The backup ran without problems so I let it write the config to a file:

$ cat .gmail-backup.cfg
[Main]
username = <email address hidden>
password = xxxxxxxx
backup_dir = /data/gmail-backup
keep_read = True
incremental = True

Then a ran:

~/bin/backup-gmail/backup_gmail.py -c ~/.gmail-backup.cfg

which after running for a while and backing up several folders gave this error:

Traceback (most recent call last):11/12346490]
  File "/home/tol/bin/backup-gmail/backup_gmail.py", line 556, in <module>
    doBackup(options, TerminalProgress())
  File "/home/tol/bin/backup-gmail/backup_gmail.py", line 524, in doBackup
    backup.incrementalBackupTo([options.start_date, options.end_date], include_labels, exclude_labels)
  File "/home/tol/bin/backup-gmail/backup_gmail.py", line 362, in incrementalBackupTo
    self.doBackup(date_range, include_labels, exclude_labels)
  File "/home/tol/bin/backup-gmail/backup_gmail.py", line 338, in doBackup
    self.fetchAllMail(date_range)
  File "/home/tol/bin/backup-gmail/backup_gmail.py", line 310, in fetchAllMail
    self.fetchAllLabel(date_range, exclude_labels)
  File "/home/tol/bin/backup-gmail/backup_gmail.py", line 299, in fetchAllLabel
    self.fetchMailByLabel(l, date_range)
  File "/home/tol/bin/backup-gmail/backup_gmail.py", line 225, in fetchMailByLabel
    self.addToMBox(label, self.mails[mid].getMail(self.dest))
  File "/home/tol/bin/backup-gmail/backup_gmail.py", line 38, in getMail
    with open('%s/%s/%s' % (dest, self.folder, self.hash_value)) as f:
IOError: [Errno 2] No such file or directory: '/data/gmail-backup/(added/by'

I ran it again with the setting "incremental = False" and pressed yes to overwriting the folder. That went fine, but running it afterwards again with "incremental = True" gave the above error.

I'm using version 0.1.2.1

Best Regards

Thomas Olsen

[Matthias Gehre (m-gehre)]

The problems comes from emails with a Message-ID in the header like

Message-ID: <email address hidden> (added by <email address hidden>)

which are parsed wrongly. I'll try to write a patch.

[Thomas Tanghus (tanghus)]

Thanks! I was just thinking about Gmail Backup today after all the fuss about suspended profiles on Google. It would be great to have it in my crontab :-)

[Matthias Gehre (m-gehre)]

Please try to replace
at the beginning of
the processMail function the line
mid = mail.get('message-id')
with
mid = getMID(mail.get('message-id'))

If the error is already happening, then the 'labels' file in your backup-directory is broken
and you will have to rerun once with incremental = false. Afterwards the fix above should
keep the error from happening again.

Can you confirm that this works?

[Thomas Tanghus (tanghus)]

Hi Matthias

Sorry for the long response time. I kept forgetting to test it but now it has become more relevant unfortunately.

First I took the code I had already, made the change suggested and ran it with incremental = False. That gave some regexp error.
Then I created a fresh directory and tried the same which failed with:

  File "./gui.py", line 358, in run
    self.func(self.options, self.prog, True)
  File "/home/tol/Dropbox/bin/backup-gmail/backup_gmail.py", line 527, in doBackup
    backup.backupTo([options.start_date, options.end_date], overwrite, include_labels, exclude_labels)
  File "/home/tol/Dropbox/bin/backup-gmail/backup_gmail.py", line 352, in backupTo
    self.doBackup(date_range, include_labels, exclude_labels)
  File "/home/tol/Dropbox/bin/backup-gmail/backup_gmail.py", line 339, in doBackup
    self.fetchAllMail(date_range)
  File "/home/tol/Dropbox/bin/backup-gmail/backup_gmail.py", line 311, in fetchAllMail
    self.fetchAllLabel(date_range, exclude_labels)
  File "/home/tol/Dropbox/bin/backup-gmail/backup_gmail.py", line 300, in fetchAllLabel
    self.fetchMailByLabel(l, date_range)
  File "/home/tol/Dropbox/bin/backup-gmail/backup_gmail.py", line 222, in fetchMailByLabel
    self.fetchMail(uid, seen, label, size)
  File "/home/tol/Dropbox/bin/backup-gmail/backup_gmail.py", line 279, in fetchMail
    self.flushFetchMailRequest()
  File "/home/tol/Dropbox/bin/backup-gmail/backup_gmail.py", line 270, in flushFetchMailRequest
    self.processMail(rfc[1], *self.fetchBuffer[i])
  File "/home/tol/Dropbox/bin/backup-gmail/backup_gmail.py", line 240, in processMail
    mid = getMID(mail.get('message-id'))
  File "/home/tol/Dropbox/bin/backup-gmail/backup_gmail.py", line 45, in getMID
    tmp = re.findall('(<[^>]+>)', env)
  File "/usr/lib/python2.7/re.py", line 177, in findall
    return _compile(pattern, flags).findall(string)
TypeError: expected string or buffer

Then I downloaded the newest (?) with 'bzr branch lp:backup-gmail', added the suggested changes, created a fresh backup dir again and it failed with:

  File "/home/tol/backup-gmail/backup_gmail.py", line 562, in <module>
    doBackup(options, TerminalProgress())
  File "/home/tol/backup-gmail/backup_gmail.py", line 532, in doBackup
    backup.backupTo([options.start_date, options.end_date], overwrite, include_labels, exclude_labels)
  File "/home/tol/backup-gmail/backup_gmail.py", line 355, in backupTo
    self.doBackup(date_range, include_labels, exclude_labels)
  File "/home/tol/backup-gmail/backup_gmail.py", line 342, in doBackup
    self.fetchAllMail(date_range)
  File "/home/tol/backup-gmail/backup_gmail.py", line 314, in fetchAllMail
    self.fetchAllLabel(date_range, exclude_labels)
  File "/home/tol/backup-gmail/backup_gmail.py", line 303, in fetchAllLabel
    self.fetchMailByLabel(l, date_range)
  File "/home/tol/backup-gmail/backup_gmail.py", line 222, in fetchMailByLabel
    self.fetchMail(uid, seen, label, size)
  File "/home/tol/backup-gmail/backup_gmail.py", line 282, in fetchMail
    self.flushFetchMailRequest()
  File "/home/tol/backup-gmail/backup_gmail.py", line 273, in flushFetchMailRequest
    self.processMail(rfc[1], *self.fetchBuffer[i])
  Fi...

[Thomas Tanghus (tanghus)]

BTW: I tried lp:~cfraire/backup-gmail/devel but it fails with:

Traceback (most recent call last):
  File "/home/tol/devel/backup_gmail.py", line 778, in <module>
    gbackup.execute()
  File "/home/tol/devel/backup_gmail.py", line 438, in execute
    include_labels = self.__fetchDefaultLabels()
  File "/home/tol/devel/backup_gmail.py", line 387, in __fetchDefaultLabels
    allMail = specials['AllMail'] # no space
KeyError: 'AllMail'

I tried to replace 'AllMail' with 'All' and 'all' just to try something but it still failed with a KeyError.

I then commented out that section:

 # Fetch default Gmail folders excluding the special \Drafts and \Trash
 def __fetchDefaultLabels(self):
  labels = self.fetchLabelNames()
  specials = self.fetchSpecialLabels()
  #allMail = specials['AllMail'] # no space
  #if allMail not in labels:
  # labels.append(allMail)
  ignore = [ specials['Drafts'], specials['Trash'] ]
  labels = filter(lambda x : x not in ignore , labels)
  return labels

Then the initial and afterwards incremental backup succeeded but created no .mbox files for the labels.

[C Fraire (cfraire)]

Thomas, my version does not create mbox files by default, because I wanted a fast incremental backup. Also with my version, full backup and incremental backup are the same operation, and the program no longer deletes any backup data.

You can run with an additional flag to export mbox files:

    -m <mbox directory>

Regarding the 'AllMail' problem, I'm curious why this fails for your account. I had put in handling for Gmail's XLIST command in order to map special folders for internationalized clients.

If you wouldn't mind, would you respond back with your Gmail XLIST result. You can print this with this short python script:

#!/usr/bin/python

import imaplib
g = imaplib.IMAP4_SSL('imap.gmail.com', 993)
g.login('<username>', '<password>')
g.xatom('XLIST', '', '*')
print g.response('XLIST')[1]

[C Fraire (cfraire)]

OK, I see now that Gmail lets users hide any label from IMAP, which likely explains Thomas's problem with the 'AllMail' key.

I updated lp:~cfraire/backup-gmail/devel to treat the special folders it looks up ('\\AllMail', '\\Drafts', and '\\Trash') as possibly hidden.

[Thomas Tanghus (tanghus)]

Yes I do hide AllMail from IMAP because it takes forever to load it initially. In KMail 1 at least. I will be closing my Gmail account soon I wont be able to do any further debugging.
Sorry for not responding to your previous comment - it has been (still is) a hectic time so the mail must have slipped by without me noticing.

>>> g.xatom('XLIST', '', '*')
('OK', ['Success'])
>>> print g.response('XLIST')[1]
['(\\HasNoChildren \\Inbox) "/" "Inbox"', '(\\HasChildren) "/" "Lists"', '(\\HasNoChildren) "/" "Lists/amarok"', '(\\HasNoChildren) "/" "Lists/css-d"', '(\\HasNoChildren) "/" "Lists/debian-kde"', '(\\HasNoChildren) "/" "Lists/diaspora-discuss"', '(\\HasNoChildren) "/" "Lists/kde"', '(\\HasNoChildren) "/" "Lists/kde-bindings"', '(\\HasNoChildren) "/" "Lists/kde-buildsystem"', '(\\HasNoChildren) "/" "Lists/kde-core-devel"', '(\\HasNoChildren) "/" "Lists/kde-devel"', '(\\HasNoChildren) "/" "Lists/kde-linux"', '(\\HasNoChildren) "/" "Lists/kde-pim"', '(\\HasNoChildren) "/" "Lists/kdepim-users"', '(\\HasNoChildren) "/" "Lists/koffice"', '(\\HasNoChildren) "/" "Lists/kubuntu-devel"', '(\\HasNoChildren) "/" "Lists/kubuntu-users"', '(\\HasNoChildren) "/" "Lists/kwin"', '(\\HasNoChildren) "/" "Lists/plasma-active"', '(\\HasNoChildren) "/" "Lists/plasma-devel"', '(\\HasNoChildren) "/" "Lists/pyqt"', '(\\HasNoChildren) "/" "Lists/quanta"', '(\\HasNoChildren) "/" "Lists/rekonq"', '(\\HasNoChildren) "/" "Lists/ubuntu-users"', '(\\HasNoChildren) "/" "Lists/xorg-ati"', '(\\HasChildren) "/" "Notifications"', '(\\HasNoChildren) "/" "Notifications/Diaspora"', '(\\HasNoChildren) "/" "Notifications/Facebook"', '(\\HasNoChildren) "/" "Notifications/Google+"', '(\\HasNoChildren) "/" "Notifications/currency-converter"', '(\\HasNoChildren) "/" "Notifications/google-translator"', '(\\HasNoChildren) "/" "Notifications/identi.ca"', '(\\HasNoChildren) "/" "Notifications/twitter"', '(\\HasNoChildren) "/" "Receipts and Invoices"', '(\\Noselect \\HasChildren) "/" "[Gmail]"', '(\\HasNoChildren \\Drafts) "/" "[Gmail]/Drafts"', '(\\HasNoChildren \\Sent) "/" "[Gmail]/Sent Mail"', '(\\HasNoChildren \\Spam) "/" "[Gmail]/Spam"', '(\\HasNoChildren \\Starred) "/" "[Gmail]/Starred"', '(\\HasChildren \\HasNoChildren \\Trash) "/" "[Gmail]/Trash"', '(\\HasChildren) "/" "bug-reports"', '(\\HasNoChildren) "/" "bug-reports/Diaspora"', '(\\HasNoChildren) "/" "memberships"', '(\\HasNoChildren) "/" "snippets"', '(\\HasNoChildren) "/" "hidden_for_privacy_reasons"']