"bzr: ERROR: exceptions.OverflowError: long int too large to convert" when committing large files

[Carsten Conrad]

running bzr 2.2 (today even 2.3b3) on Windows 7 I tried to create a local repository by typing

cd \Users\myuser\Videos
bzr init
bzr ignore desktop.ini
bzr commit

This directory contains several large .mpg and divx-files, with the largest being ca. 4,7 GB large.

This repeatedly makes bzr crash with the following output:

aborting commit write group: OverflowError('long int too large to convert',)
bzr: ERROR: exceptions.OverflowError: long int too large to convert

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 917, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1117, in run_bzr
  File "bzrlib\commands.pyo", line 691, in run_argv_aliases
  File "bzrlib\commands.pyo", line 710, in run
  File "bzrlib\cleanup.pyo", line 135, in run_simple
  File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
  File "bzrlib\builtins.pyo", line 3241, in run
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\workingtree_4.pyo", line 197, in commit
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\mutabletree.pyo", line 201, in commit
  File "bzrlib\commit.pyo", line 286, in commit
  File "bzrlib\cleanup.pyo", line 131, in run
  File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
  File "bzrlib\commit.pyo", line 415, in _commit
  File "bzrlib\commit.pyo", line 650, in _update_builder_with_changes
  File "bzrlib\repository.pyo", line 659, in record_iter_changes
  File "bzrlib\commit.pyo", line 671, in _filter_iter_changes
  File "_dirstate_helpers_pyx.pyx", line 1439, in bzrlib._dirstate_helpers_pyx.P
rocessEntryC.__next__ (bzrlib\_dirstate_helpers_pyx.c:12084)
  File "_dirstate_helpers_pyx.pyx", line 1712, in bzrlib._dirstate_helpers_pyx.P
rocessEntryC._iter_next (bzrlib\_dirstate_helpers_pyx.c:14846)
  File "_dirstate_helpers_pyx.pyx", line 1811, in bzrlib._dirstate_helpers_pyx.P
rocessEntryC._loop_one_block (bzrlib\_dirstate_helpers_pyx.c:15715)
  File "_dirstate_helpers_pyx.pyx", line 1136, in bzrlib._dirstate_helpers_pyx.P
rocessEntryC._process_entry (bzrlib\_dirstate_helpers_pyx.c:8198)
  File "_dirstate_helpers_pyx.pyx", line 872, in bzrlib._dirstate_helpers_pyx._u
pdate_entry (bzrlib\_dirstate_helpers_pyx.c:5834)
  File "_dirstate_helpers_pyx.pyx", line 824, in bzrlib._dirstate_helpers_pyx._p
ack_stat (bzrlib\_dirstate_helpers_pyx.c:5492)
OverflowError: long int too large to convert

bzr 2.3b3 on python 2.6.6 (Windows-7-6.1.7600)
arguments: ['bzr', 'commit', '-m', 'klappt nicht']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  bzrtools C:\Program Files (x86)\Bazaar\plugins\bzrtools [2.3.0]
  colo C:\Program Files (x86)\Bazaar\plugins\colo [0.2.1dev]
  explorer C:\Program Files (x86)\Bazaar\plugins\explorer [1.1.2dev]

  fastimport C:\Program Files (x86)\Bazaar\plugins\fastimport [0.10.0d
ev]
  launchpad C:\Program Files (x86)\Bazaar\plugins\launchpad [2.3b3]
  loom C:\Program Files (x86)\Bazaar\plugins\loom [2.2.1dev]
  netrc_credential_store C:\Program Files (x86)\Bazaar\plugins\netrc_credential_
store [2.3b3]
  news_merge C:\Program Files (x86)\Bazaar\plugins\news_merge [2.3b3]
  pipeline C:\Program Files (x86)\Bazaar\plugins\pipeline [unknown]
  qbzr C:\Program Files (x86)\Bazaar\plugins\qbzr [0.20.0dev1]
  rewrite C:\Program Files (x86)\Bazaar\plugins\rewrite [0.6.2dev]
  svn C:\Program Files (x86)\Bazaar\plugins\svn [1.0.5dev]
  upload C:\Program Files (x86)\Bazaar\plugins\upload [1.0.0dev]
  xmloutput C:\Program Files (x86)\Bazaar\plugins\xmloutput [0.8.7.de
v]

Obviously the large file sizes might contribute to the problem. I know this is not the typical workspace setup for a VCS, but it would add greatly to the versatility of bzr (like a distributed OS-crossing backup of ALL user files) if it could handle larger files.

It would be great if you could make bzr handle this problem.

Thanks in advance!

[Martin Packman (gz)]

If you could add any more information about what you were doing when you get this error in the existing bug, that would be great. In particular, is it only when you commit? Or do you also get it with other operations, and if so does the 4.7GB file have to be added first, or does it sitting in the directory being ignored still break things for you?

[Carsten Conrad (pinpoint)]

The list of steps i performed was missing the "add"-command.

The original error occurs when I type:
cd \Users\myuser\Videos
bzr init
bzr ignore desktop.ini // after the first try I just left the existing .bzrignore in place
bzr add .
bzr commit

The "add"-command adds the large files and probably causes the crash.

To answer your question, I just deleted the ".bzr"-directory and started all over again, this time adding only a small file (.bzrignore) but leaving the large files in the directory. this works:

"scenario 2":
C:\Users\carsten\Videos>dir
 Volume in Laufwerk C: hat keine Bezeichnung.
 Volumeseriennummer: AED8-C42B

 Verzeichnis von C:\Users\carsten\Videos

06.01.2011 18:28 <DIR> .
06.01.2011 18:28 <DIR> ..
05.01.2011 18:34 13 .bzrignore
05.01.2011 18:26 <DIR> 1987-04-26 - Kinderkommunion Carsten
05.01.2011 18:25 <DIR> 1991-09-25 - Kim
05.01.2011 18:27 <DIR> 1998-11-13 - Goldene Hochzeit
05.01.2011 18:26 <DIR> 19xy - Meran RTL und ARD
05.01.2011 18:25 <DIR> 2001-09 - Cala Ratjada
               1 Datei(en), 13 Bytes
               7 Verzeichnis(se), 208.388.423.680 Bytes frei

C:\Users\carsten\Videos>bzr init
Created a standalone tree (format: 2a)

C:\Users\carsten\Videos>bzr status
unknown:
  .bzrignore
  1987-04-26 - Kinderkommunion Carsten/
  1991-09-25 - Kim/
  1998-11-13 - Goldene Hochzeit/
  19xy - Meran RTL und ARD/
  2001-09 - Cala Ratjada/

C:\Users\carsten\Videos>bzr add .bzrignore
adding .bzrignore

C:\Users\carsten\Videos>bzr commit -m "..."
bzr: warning: The commit message is a file name: "...".
(use --file "..." to take commit message from that file)
Committing to: C:/Users/carsten/Videos/
added .bzrignore
Committed revision 1.

So, nothing bad happens when I leave the large files unadded.

"scenario 2":
To discover what happens with other operations with the large files added, I did this (after first deleting the ".bzr"-directory):

C:\Users\carsten\Videos>bzr init
Created a standalone tree (format: 2a)

C:\Users\carsten\Videos>bzr add "19xy - Meran RTL und ARD"
adding "19xy - Meran RTL und ARD"
adding "19xy - Meran RTL und ARD/Meran.MPG"
adding "19xy - Meran RTL und ARD/Meran.divx"

C:\Users\carsten\Videos>bzr status
bzr: ERROR: exceptions.OverflowError: long int too large to convert

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 917, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1117, in run_bzr
  File "C:/Program Files (x86)/Bazaar/plugins\loom\commands.py", line 203, in ru
n_argv_aliases
  File "bzrlib\commands.pyo", line 691, in run_argv_aliases
  File "bzrlib\commands.pyo", line 710, in run
  File "bzrlib\cleanup.pyo", line 135, in run_simple
  File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
  File "bzrlib\commands.pyo", line 1132, in ignore_pipe
  File "bzrlib\builtins.pyo", line 301, in run
  File "bzrlib\status.pyo", line 168, in show_tree_status
  File "bzrlib\status.pyo", line 72, in report_changes
  File "bzrlib\tree.pyo", line 94, in changes_from
  File "bzrlib\decorators.pyo", line 140, in read_locked
  File "bzrlib\tree.pyo", line 1002,...