[Disk tests/storage_device_sdx] Test got fail.

[Gabriel Zhi Chen]

When I run test case [Disk tests/storage_device_sdx], system pops up 'Low Disk Space' warning and the test gets fail.

Please refer to screenshots and system logs in attachment.

[Daniel Manrique (roadmr)]

Wow, this machine's partition map is very strange!

Look at the mount information, I'm including only the relevant ones (all in-memory filesystems removed)

/dev/disk/by-uuid/c6e02c0a-1dd0-4570-868c-edaa6f729730 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
/dev/sda5 /home ext4 rw,relatime,data=ordered 0 0
/dev/sda1 /boot/efi vfat rw,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0

The uuid for the root partition maps to /dev/sda3, which is a 20-GB partition.

Gabriel, could you please run the "df -h" command and post the output here? this is to see how much free space /dev/sda3 actually has.

This would let us know what happens when we try to test /dev/sda.

For /dev/sdb, notice from the screenshot that it didn't find where it's mounted (because it's not, per the mount map I looked at). Thus it puts the "scratch disk" at "", meaning that bonnie++ will again use the root directory (look at the script, if $mount_point is empty, bonnie++ will write to /tmp/scratchdir).

So two things to do for this bug:

1- Improve the checking so if a disk is not mounted, we either mount it (may be dangerous!) or simply fail to run the test (rather than running on a different disk and producing bogus results).

2- Once Gabriel sends us free space information, we can determine why it's failing on sda (we need to look at sda3 which is fairly small, it may be legitimately filling up).

[Daniel Manrique (roadmr)]

FWIW, on a system with 2 GB RAM, Bonnie is writing a 4-GB file. So on Gabriel's system that has 8 GB, bonnie++ could easily be trying to write 16 GB of data, which on 20 GB root may, indeed, be filling it up.

[Jeff Lane  (bladernr)]

I am able to reliably recreate this on a 16GB SSD.

[Jeff Lane  (bladernr)]

Setting it to medium for now as it's a corner case that only affects small systems or weird partitioning layouts.

[Gabriel Zhi Chen (gabrielzchen)]

Hi Daniel,

'df -h' info pasted following :

***************************************

u@Lenovo-M4400-B10G0006J:~$ sudo df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 19G 3.5G 15G 19% /
udev 3.9G 4.0K 3.9G 1% /dev
tmpfs 1.6G 936K 1.6G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 3.9G 76K 3.9G 1% /run/shm
/dev/sda5 889G 242M 844G 1% /home
/dev/sda1 47M 350K 47M 1% /boot/efi

***************************************

[Gabriel Zhi Chen (gabrielzchen)]

 There is a screen shot of "df -h" command output, it looks more comfortable.

[Daniel Manrique (roadmr)]

OK, work to do for this bug:

1- Improve the checking so if a disk is not mounted, we either mount it (may be dangerous!) or simply fail to run the test (rather than running on a different disk and producing bogus results). See comment #4. Otherwise, it will use the default /tmp directory.

2- Check the available free space in the mount to be tested. If this is smaller than RAM x 2 (plus some reasonable safety margin), then set bonnie++ parameter to use a fixed file size that's smaller than the free space. If free space is bigger than RAM x 2, then let bonnie++ do its thing as usual.

I'll set to triaged for now.

Gabriel, I'm curious how the partitions were configured that way. Is this an OEM pre-install? if so, would it be possible to change the root (/dev/sda3) partition size to maybe 32 GB? This will allow you to run the test without problems. That's a workaround however, the real solution would be implementing the above.

[Jeff Lane  (bladernr)]

Ok, I think I have it sorted... doing final tests now. Bonnie uses 2X the RAM when deciding how much data to write... so I've added a calculation to discover the system memory, compare that to free disk space and if the free disk space is not more than 2x RAM, we cut the amount of RAM presented to Bonnie to 1/2 the amout of free space.

What that means is that a system with 12GB ram will require at least 24GB of free space for bonnie to run successfully.
If the systm only has 16GB of free space, we need to tell bonnie that the system only has 8GB of free space using the -r option.

That will force bonnie to only write 2x RAM, or 16GB total data in this example.

So far I've tested this on my 16GB SSH which only has about 7.8GB Free on a system with 6GB RAM. THe new method has worked on it, so I'll try once more on teh larger HDD to make sure it works.

[Jeff Lane  (bladernr)]

Here's the new output on the 16GB drive:
bladernr@klaatu:~/development/checkbox/checkbox-old/scripts$ sudo ./storage_test sdb
/dev/sdb is a block device
/dev/sdb reports a size of 15.8GB.
Running bonnie++ on /dev/sdb...
Putting scratch disk at /media/bladernr/0cb321f2-6c66-40c6-9606-a105dce17561
    Disk /dev/sdb has 7117MB available
    System has 5958MB RAM
    We need 11916MB of disk space for testing
    Insufficient disk space available for defaults.
    reducing memory footprint to be 1/2 of free disk space.
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
klaatu 7G 211 97 8988 5 6738 3 549 99 27483 7 121.9 7
Latency 252ms 4925ms 7479ms 32387us 80404us 11053ms
Version 1.97 ------Sequential Create------ --------Random Create--------
klaatu -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
                 16 3698 14 +++++ +++ 10362 27 8752 30 +++++ +++ 10632 28
Latency 10708us 1359us 1687us 773us 199us 381us
1.97,1.97,klaatu,1,1377823265,7G,,211,97,8988,5,6738,3,549,99,27483,7,121.9,7,16,,,,,3698,14,+++++,+++,10362,27,8752,30,+++++,+++,10632,28,252ms,4925ms,7479ms,32387us,80404us,11053ms,10708us,1359us,1687us,773us,199us,381us

[Jeff Lane  (bladernr)]

And here's hte new output on my laptop which has enough disk space:
bladernr@klaatu:~/development/checkbox/checkbox-old/scripts$ sudo ./storage_test sda
/dev/sda is a block device
/dev/sda reports a size of 500GB.
Running bonnie++ on /dev/sda...
Putting scratch disk at /
    Disk /dev/sda has 12884MB available
    System has 5958MB RAM
    We need 11916MB of disk space for testing
   Free disk space is sufficient to continue testing.
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
klaatu 12G 213 98 45805 18 20869 7 563 99 54051 10 142.0 7
Latency 56567us 1519ms 5996ms 49389us 322ms 858ms
Version 1.97 ------Sequential Create------ --------Random Create--------
klaatu -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
                 16 11319 38 +++++ +++ 28766 71 20495 66 +++++ +++ 29141 75
Latency 52381us 1558us 1485us 254us 123us 221us
1.97,1.97,klaatu,1,1377814965,12G,,213,98,45805,18,20869,7,563,99,54051,10,142.0,7,16,,,,,11319,38,+++++,+++,28766,71,20495,66,+++++,+++,29141,75,56567us,1519ms,5996ms,49389us,322ms,858ms,52381us,1558us,1485us,254us,123us,221us

[Gabriel Zhi Chen (gabrielzchen)]

Hi Daniel,

At present, our installation is an OEM pre-install, we could not change root partition size here because it is a fixed-size partition (20 GB). The size is based on OEM factory process and set.

I asked the software engineer Yuning <email address hidden>. He advice that we can run Bonnie ++ on a larger size partition, example : /home.

[Daniel Manrique (roadmr)]

Gabriel, thanks, Jeff just committed a fix that will enable this to run on smaller partitions. We also considered Yuning's idea, however if a system has no large partitions on the device we want to test, we will still need to adjust size accordingly, so Jeff's fix seemed to be more universal.

Note this fix should be in the next released version of checkbox , it should be ready for testing in a few days.

[Gabriel Zhi Chen (gabrielzchen)]

Daniel and Jeff, thank you all. When the new version is generated, I'll do more with verification.

[Jeffrey Chang (modern911)]

checkbox 0.16.8+bzr1990+201309061548~ubuntu12.04.1
On my system, I have a 32GB mSATA not been used at all.
And I will see error when running storate_test

modern911@jeffrey-Audi15:~$ sudo /usr/lib/checkbox/bin/storage_test sdb
/dev/sdb is a block device
/dev/sdb reports a size of 32.0GB.
Running bonnie++ on /dev/sdb...
Putting scratch disk at
    Disk /dev/sdb has MB available
    System has 3853MB RAM
    We need 7706MB of disk space for testing
    Insufficient disk space available for defaults.
    reducing memory footprint to be 1/2 of free disk space.
/usr/lib/checkbox/bin/storage_test: line 25: /2: syntax error: operand expected (error token is "/2")

[Daniel Manrique (roadmr)]

No space reported for /dev/sdb is a dead ringer for a non-mounted drive. See comment #4 for an idea on what's happening and what we could do (hint: if the drive is not mounted, we shouldn't try to test it with bonnie; we can try to mount it, but it will only lead to pain and tears).

[Daniel Manrique (roadmr)]

I'll set this to Triaged again pending tackling of item 1 from comment 4 (which caused the failure in comment 17).