file scanning is slow

[Ken VanDine]

I did a fresh reboot this morning, the daemon was idle when i restarted. Here is the log from startup to IDLE, which took 10 minutes.

I have 3051 files in My Files

[Facundo Batista (facundo)]

Sync should be receive also the file stat from HQ (taken before HQ starts hashing the file), and update it in the metadata, to prevent endless pointless re-hashing.

[Guillermo Gonzalez (verterok)]

A fix is available in the nigthly ppa.

[Guillermo Gonzalez (verterok)]

Just a followup about recent changes that affects perfomance:

There are 3 new options in syncdaemon:
 1) --log_level=LOG_LEVEL: allow setting the log level (DEBUG by default)
 2) --disable_fsync_md: This options disables the fsync call after each write on a metadata file. (fsync is used by default, *WARNING*: this might cause metadata corruption on power failure)
 3) --send_events_over_dbus: send internal event over DBus (mainly for debug, this was on by default iin previous versions)

I executed some tests with the different switches, the test data is the "drivers" subdir of the kernel source tree:
all on: 207,435 sec
no-send_events_over_dbus: 199,999351 sec.
no-logs: 182,999466 sec
no-logs + no-send_events_over_dbus: 165,482 sec
no-fsync: 110,999874 sec
no-fsync + no-send_events_over_dbus: 106,165 sec
no-logs + no-fsync: 83,999807 sec
no-logs + no-fsync + no-send_events_over_dbus: 76,179 sec

[Robert Collins (lifeless)]

On Tue, 2009-08-11 at 20:00 +0000, Guillermo Gonzalez wrote:
> Just a followup about recent changes that affects perfomance:
>
> There are 3 new options in syncdaemon:
> 1) --log_level=LOG_LEVEL: allow setting the log level (DEBUG by default)
> 2) --disable_fsync_md: This options disables the fsync call after each write on a metadata file. (fsync is used by default, *WARNING*: this might cause metadata corruption on power failure)

Disabling fsync is a very good idea for consumer environments -
triggering unnecessary disk writes sucks. There are a number of things
that can be done to mitigate against data loss in power failure
situations. We use these very heavily in bzr's repository format
design :). A bug is the wrong place to write these up, but I'd be happy
to talk about how to apply them to your metadata.

-Rob

[Guillermo Gonzalez (verterok)]

Hi Robert,

On Tue, Aug 11, 2009 at 6:11 PM, Robert
Collins<email address hidden> wrote:
> On Tue, 2009-08-11 at 20:00 +0000, Guillermo Gonzalez wrote:
>> Just a followup about recent changes that affects perfomance:
>>
>> There are 3 new options in syncdaemon:
>>  1) --log_level=LOG_LEVEL: allow setting the log level (DEBUG by default)
>>  2) --disable_fsync_md: This options disables the fsync call after each write on a metadata file. (fsync is used by default, *WARNING*: this might cause metadata corruption on power failure)
>
> Disabling fsync is a very good idea for consumer environments -
> triggering unnecessary disk writes sucks. There are a number of things
> that can be done to mitigate against data loss in power failure
> situations. We use these very heavily in bzr's repository format
> design :). A bug is the wrong place to write these up, but I'd be happy
> to talk about how to apply them to your metadata.
>
> -Rob

Would be great to talk about how bzr handle this.
If it's ok with you, I'll take a look to bzr source code and ping you on IRC.

Cheers,

--
Guillermo Gonzalez
<http://launchpad.net/~verterok>

[Robert Collins (lifeless)]

Please do. Our specific ways around it may not suite but I'm sure we can
come up with some ideas or concepts.

The bsaic forms of corruption are a failure to write something: a
write() call may not hit disk, and a series of writes may only partly
hit disk, and finally separate operations may hit disk independently.

See for instance the ext4 issues with kde settings: renames were hitting
disk but the data written to the temporary file were not (because the
machine was stopping cold).

-Rob