0.6.25->0.7.05 fatal performance regression
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Duplicity |
New
|
Undecided
|
Unassigned |
Bug Description
Running on updated Fedora 23 x86_64:
duplicity-
21:57:20 lstat("
21:57:20 readlink(
21:57:24 lstat("
21:57:24 readlink(
21:57:28 lstat("
21:57:28 readlink(
21:57:33 lstat("
21:57:33 readlink(
- it was running for 2 hours and reading each file for 4 seconds is unusable.
26.69% duplicity libpython2.7.so.1.0 [.] PyEval_EvalFrameEx
4.73% duplicity libc-2.22.so [.] vfprintf
3.70% duplicity libpython2.7.so.1.0 [.] lookdict_string
3.70% duplicity libpython2.7.so.1.0 [.] PyUnicodeUCS4_
3.47% duplicity libpython2.7.so.1.0 [.] PyString_Format
3.11% duplicity libpython2.7.so.1.0 [.] _PyObject_
3.09% duplicity libpython2.7.so.1.0 [.] PyEval_EvalCodeEx
1.85% duplicity libpython2.7.so.1.0 [.] convertitem
1.85% duplicity libpython2.7.so.1.0 [.] PyFrame_New
After downgrading just the package duplicity to 0.6.25 the backup was successfuly done in 4 minutes, like always before:
duplicity-
--------------[ Backup Statistics ]--------------
StartTime 1447707702.06 (Mon Nov 16 22:01:42 2015)
EndTime 1447707987.63 (Mon Nov 16 22:06:27 2015)
ElapsedTime 285.57 (4 minutes 45.57 seconds)
SourceFiles 92925
SourceFileSize 2516870978 (2.34 GB)
NewFiles 51428
NewFileSize 179314732 (171 MB)
DeletedFiles 22722
ChangedFiles 1196
ChangedFileSize 284383140 (271 MB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 75346
RawDeltaSize 210169249 (200 MB)
TotalDestinatio
Errors 0
-------
11G host1-backup
Unaware how to do Python-source-level performance monitoring.
duplicity --archive-dir /root/backup/
3.0G /root/backup/
68G /host1/
wc -l:
558303 /tmp/host2-
python-
Linux ext4
When I remove that --exclude-filelist /tmp/host2- run.rpmsafe with 558303 entries the performance is OK even with 0.7.05. That worked fine in 0.6.25.
The reason for it is to exclude unchanged system files which make no sense to backup: run.rpmsafe
rpm -Vva 2>/dev/null|perl -ne 's/^[.]{7}[.T][.] . // or next;chomp;use Cwd qw(realpath);print "- ".realpath($_)."\n" if -f;' >/tmp/host2-