On 5/2/2011 4:27 AM, rew wrote:
> I'm pretty sure that 67000000 files is not a magic number. I'm pretty
> sure that 66999999 files will also be horribly slow.

Of course.

> Just the fact that /I/ happen to have 67 million files makes this bug
> invalid in your eyes?

No, it is the fact that this is a highly unusual configuration that Ted 
Tso seems to think is an abusive and unsupported use of the fs, and 
therefore, Ubuntu should not be trying to optimize its defaults for.

> My last "fight" with fsck resulted in an fsck time of something like
> three months. Provided your assumption of linear fsck time with file
> system parameters is valid, this would mean that 100x less files would
> result in an fsck time of 1 day. Unacceptable to many people. OR 1000
> times less files may take 2.4 hours. How do you explain your "not much"
> answer to your boss after he askes: "what did you do this morning?". "I
> turned on my workstation this morning and it decided to fsck the root
> partition, and the fsck took 2.4 hours. So this morning I mostly waited
> for my machine to boot... "

If it scaled linearly then I would expect 67 million inodes to take 
about 1.5 hours given that checking an fs with 300k takes half a minute. 
  Perhaps this bug should be reassigned to e2fsprogs and refactored to 
focus just on the pathological slow down of fsck with extreme numbers of 
inodes with many hard links.

You could also just disable the periodic fsck.

> Over the years people will have more and more files. This will continue
> to grow. I mentioned the fact that I was "beyond reasonable" back in
> 1987 with only 2200 files.

Maybe in another 10-20 years your average person might get that many, 
but by then I'm sure we'll be using a different fs entirely.  At present 
this isn't anywhere close to being an issue.

> I adapt the boot stuff of my fileservers so that they will boot first,
> (i.e. start all services that depend on just the root) and then they
> start fscking the data partitions. When done they will mount them and
> restart services that depend on them (nfs).
>
> This allows me to monitor their progress remotely. Stuff like that. But
> also I can already use the services that they provide that are not
> dependent on the large partition with most of the files.
>
> If we have just one partition this is no longer possible: ALL services
> depend on "fsck of /" .

That is the kind of thing that requires manual planning; it can not be 
set up like that by default.

> Another reason why splitting up the disk helps is that fsck is not
> linear with the disk space. Especially when fsck's working set is beyond
> the amount of RAM available.

That sounds like the problem.  If you are swapping heavily that would 
explain the pathological slow down.

> I work in data-recovery. Some of the recoveries are mostly images. As
> the number of pixels increases, the effects of jpeg compression
> increases as well. An image from a 12Mpixel camera is not twice as big
> as one from a 6 Mpixel camera. So, ten years from now when SLR cameras
> are 20-50 Mpixels, we'll have 4-8Mbytes of jpg image data. People will
> shoot and store many more images on their 20T drive. Now do the math of
> how many images will fit....

Any sane person will not keep millions of pictures because you could not 
possibly ever look at them all.  Just taking a million pictures at a 
rate of one every minute for 18 waking hours a day would take 2.5 years. 
  They also won't create 100 hard links to each one.