Disk total stats are double

[Noel J. Bergman]

Consider the following:

~$ dstat --disk -Dtotal,sdb1,sdc1
 -dsk/total----dsk/sdb1----dsk/sdc1-
 read writ: read writ: read writ
 507k 461k: 46k 0 : 5.4B 43k
 43M 53M: 21M 0 : 0 26M
 38M 31M: 19M 0 : 0 15M
 42M 35M: 21M 0 : 0 17M

and also:

$ dstat --disk -Dtotal,sdb,sdc
-dsk/total----dsk/sdb-----dsk/sdc--
 read writ: read writ: read writ
 510k 464k: 47k 0 : 9.6B 45k
  53M 55M: 27M 0 : 0 27M
  35M 41M: 18M 0 : 0 20M
 103M 94M: 52M 0 : 0 46M

Notice that the disk/total for read and write is double that of the actual disk. Based upon looking at (trimmed):
~$ dstat -f
 --dsk/sda-----dsk/sda1---dsk/sda10----dsk/sda2----dsk/sda5----dsk/sda6----dsk/sda7----dsk/sda8----dsk/sda9----dsk/sdb-----dsk/sdb1----dsk/sdc-----dsk/sdc1-
 read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ
 209k 189k: 3.7B 0 : 157k 129k: 0 0 : 1.2B 0 : 1.1B 0 : 1B 0 : 51k 59k: 90B 766B: 270k 0 : 270k 0 : 9.5B 267k: 5.3B 267k
 884k 220k: 0 0 : 0 8192B: 0 0 : 0 0 : 0 0 : 0 0 : 884k 212k: 0 0 : 24M 0 : 24M 0 : 0 14M: 0 14M
   0 188k: 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 188k: 0 0 : 25M 0 : 25M 0 : 0 29M: 0 29M

 I suspect that dstat is processing both sd<x> and sd<x><y>, e.g., sda & sda1, resulting in the doubled values.

[dag (dag-wieers)]

On Sun, 14 Dec 2008, Noel J. Bergman wrote:

> Public bug reported:
>
> Consider the following:
>
> ~$ dstat --disk -Dtotal,sdb1,sdc1
> -dsk/total----dsk/sdb1----dsk/sdc1-
> read writ: read writ: read writ
> 507k 461k: 46k 0 : 5.4B 43k
> 43M 53M: 21M 0 : 0 26M
> 38M 31M: 19M 0 : 0 15M
> 42M 35M: 21M 0 : 0 17M
>
> and also:
>
> $ dstat --disk -Dtotal,sdb,sdc
> -dsk/total----dsk/sdb-----dsk/sdc--
> read writ: read writ: read writ
> 510k 464k: 47k 0 : 9.6B 45k
> 53M 55M: 27M 0 : 0 27M
> 35M 41M: 18M 0 : 0 20M
> 103M 94M: 52M 0 : 0 46M
>
> Notice that the disk/total for read and write is double that of the actual disk. Based upon looking at (trimmed):
> ~$ dstat -f
> --dsk/sda-----dsk/sda1---dsk/sda10----dsk/sda2----dsk/sda5----dsk/sda6----dsk/sda7----dsk/sda8----dsk/sda9----dsk/sdb-----dsk/sdb1----dsk/sdc-----dsk/sdc1-
> read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ
> 209k 189k: 3.7B 0 : 157k 129k: 0 0 : 1.2B 0 : 1.1B 0 : 1B 0 : 51k 59k: 90B 766B: 270k 0 : 270k 0 : 9.5B 267k: 5.3B 267k
> 884k 220k: 0 0 : 0 8192B: 0 0 : 0 0 : 0 0 : 0 0 : 884k 212k: 0 0 : 24M 0 : 24M 0 : 0 14M: 0 14M
> 0 188k: 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 188k: 0 0 : 25M 0 : 25M 0 : 0 29M: 0 29M
>
> I suspect that dstat is processing both sd<x> and sd<x><y>, e.g., sda &
> sda1, resulting in the doubled values.
>
> ** Affects: dstat (Ubuntu)
> Importance: Undecided
> Status: New

I cannot help if I don't know what dstat version and what kernel. On my
system I do not see that behaviour, but it is not impossible that this
behaviour exists with older dstat versions and/or older or newer kernel
versions.

This bugreport is useless if it does not specify this minimum information,
sorry.

--
-- dag wieers, <email address hidden>, http://dag.wieers.com/ --
[Any errors in spelling, tact or fact are transmission errors]

[Noel J. Bergman (noeljb)]

OS: Jaunty (2.6.28-2-generic)
dstat: 0.6.8-1

Very reproducible.

[Noel J. Bergman (noeljb)]

Dag,

I can also reproduce this with dstat 0.6.9, which I downloaded from your site:

noel@jaunty:~/dstat/dstat-0.6.9$ ./dstat --disk -Dtotal,sda
-dsk/total----dsk/sda--
 read writ: read writ
1114k 1067k: 557k 533k

I'll check it against Hardy and Intrepid when I have time. What do you see if you cat /proc/diskstats? Do you see both the sd<x> and sd<x><n> entries?

By the way, thank you for being here and supporting your software. :-)

[Noel J. Bergman (noeljb)]

Same on Intrepid:

noel@noel-intrepid:~/shared/dstat-0.6.9$ uname -r
2.6.27-9-generic

noel@noel-intrepid:~/shared/dstat-0.6.9$ ./dstat --disk -Dtotal,sda
-dsk/total----dsk/sda--
 read writ: read writ
3243k 114k:1622k 57k
   0 144k: 0 72k

[Noel J. Bergman (noeljb)]

Ah ... but not on Hardy:

noel@noel-hardy:~/shared/dstat-0.6.9$ uname -r
2.6.24-22-generic

noel@noel-hardy:~/shared/dstat-0.6.9$ ./dstat --disk -Dtotal,sda
-dsk/total----dsk/sda--
 read writ: read writ
2189k 107k:2189k 107k
   0 112k: 0 112k
   0 4096B: 0 4096B

So this defect occurs on Intrepid and Jaunty, but not on Hardy.

[Noel J. Bergman (noeljb)]

And, for completeness, Fedora 10 has the same problem as Intrepid and Jaunty:

[noel@noel-fedora10 dstat-0.6.9]$ uname -r
2.6.27.7-134.fc10.x86_64

[noel@noel-fedora10 dstat-0.6.9]$ ./dstat --disk -Dtotal,sda
-dsk/total----dsk/sda--
 read writ: read writ
1640k 2853k: 820k 1427k
   0 416k: 0 208k

[Noel J. Bergman (noeljb)]

According to http://www.mjmwired.net/kernel/Documentation/iostats.txt, the information changed after 2.6.24:

  "There are only *four* fields available for partitions on 2.6 machines."
  "In 2.6.25, the full statistic set is again available for partitions and disk and partition statistics are consistent again."

This might explain the difference between Hardy (2.6.24) and the rest (2.6.27 and later).

[dag (dag-wieers)]

On Mon, 15 Dec 2008, Noel J. Bergman wrote:

> OS: Jaunty (2.6.28-2-generic)
> dstat: 0.6.8-1
>
> Very reproducible.

Right, the problem likely is that /proc/diskstats in newer kernels also
provide proper stats for partitions, which it doesn't do on older kernels:

[dag@rhun ~]$ cat /proc/diskstats | grep sda
    8 0 sda 196482 22091 3600800 1524365 304150 614172 7355412 45110870 0 1321463 46646670
    8 1 sda1 1033 10862 2 4
    8 2 sda2 215888 3587367 919426 7355408
    8 3 sda3 1199 1411 0 0
    8 4 sda4 427 872 0 0

If you look at the plugin you will see what I mean.

--
-- dag wieers, <email address hidden>, http://dag.wieers.com/ --
[Any errors in spelling, tact or fact are transmission errors]

[Noel J. Bergman (noeljb)]

> If you look at the plugin you will see what I mean.

Yes, I had looked at it. And now I have a patch for you, which I'm attaching.

The output before the patch is:

$ ./dstat~ --disk -D total,sda,sda8,sda10
-dsk/total----dsk/sda-----dsk/sda8---dsk/sda10-
 read writ: read writ: read writ: read writ
 454k 467k: 227k 233k: 13k 20k: 214k 213k
8192B 0 :4096B 0 : 0 0 :4096B 0
8192B 0 :4096B 0 : 0 0 :4096B 0
8192B 0 :4096B 0 : 0 0 :4096B 0
   0 8192B: 0 4096B: 0 4096B: 0 0
8192B 64k:4096B 32k: 0 32k:4096B 0
   0 112k: 0 56k: 0 56k: 0 0

The result after the patch is:

$ ./dstat --disk -D total,sda,sda8,sda10
-dsk/total----dsk/sda-----dsk/sda8---dsk/sda10-
 read writ: read writ: read writ: read writ
 227k 234k: 227k 234k: 13k 20k: 214k 213k
   0 1060k: 0 1060k: 0 76k: 0 984k
   0 64k: 0 64k: 0 64k: 0 0

The patch, itself, just enhances the regex to filter out partitions when computing the total stat.

[dag (dag-wieers)]

On Mon, 15 Dec 2008, Noel J. Bergman wrote:

>> If you look at the plugin you will see what I mean.
>
> Yes, I had looked at it. And now I have a patch for you, which I'm
> attaching.
>
> The patch, itself, just enhances the regex to filter out partitions when
> computing the total stat.

Hi Noel,

Thanks for the patch, I modified it in 3 ways:

  - Fix it for the other dstat_disk* plugins (it is possible that for the
    2.4 kernels the same changes is being made at some point)

  - Fixed the discover functions as well (we do not want /dev/sda1 to
    appear in the full list)

  - Make sure all devices are being filtered (eg. sdaa1)

I committed the patch to subversion and it will be part of 0.6.10 when it
is released. Thanks a lot for the feedback and patch !

--
-- dag wieers, <email address hidden>, http://dag.wieers.com/ --
[Any errors in spelling, tact or fact are transmission errors]

[Noel J. Bergman (noeljb)]

Hi Dag. You're welcome. Glad that I was able to help and give you a start on the fix. With respect to the discover functions, are you seeing partitions under /sys/block/*? I see only devices, e.g., sda, and that is reflected in the output of dstat -f --disk.

[dag (dag-wieers)]

On Mon, 15 Dec 2008, Noel J. Bergman wrote:

> Hi Dag. You're welcome. Glad that I was able to help and give you a
> start on the fix. With respect to the discover functions, are you
> seeing partitions under /sys/block/*? I see only devices, e.g., sda,
> and that is reflected in the output of dstat -f --disk.

You're right for the normal dstat_disk function, the reason I did add the
same filter for all is to make the code more simple. (dstat_disk24 and
dstat_disk24old) Especially if you look at how it was simplified by the
subsequent patch. (same filter, same code)

Don't worry though, it has definitely improved thanks to your efforts.

--
-- dag wieers, <email address hidden>, http://dag.wieers.com/ --
[Any errors in spelling, tact or fact are transmission errors]

[Noel J. Bergman (noeljb)]

I wasn't worried. Just wanted to understand. And learned something in the process. :-)

I just looked at the code in svn (by the way, viewvc appears to be MIA). Now I get it -- you factored out the regex pattern into a shared property of the class. That makes total sense, and did simplify the rest of the code.

It has been great working with you on this fix. :-)

[Andres Mujica (andres.mujica)]

As the fix made it's way into upstream i'm marking as commited.

Thanks!!

[Noel J. Bergman (noeljb)]

The code was committed upstream, but never made it into even Karmic.

See http://svn.rpmforge.net/svn/trunk/tools/dstat/ChangeLog

We need to upgrade to version 0.7.0, which came out only a week ago.

[dag (dag-wieers)]

On Sat, 7 Nov 2009, Noel J. Bergman wrote:

> The code was committed upstream, but never made it into even Karmic.
>
> See http://svn.rpmforge.net/svn/trunk/tools/dstat/ChangeLog
>
> We need to upgrade to version 0.7.0, which came out only a week ago.

Beware, 0.7.0 has not been released yet. I was planning a release but it
has not yet happened.

--
-- dag wieers, <email address hidden>, http://dag.wieers.com/ --
[Any errors in spelling, tact or fact are transmission errors]

[Noel J. Bergman (noeljb)]

@dag, OK. The comments in the change log led me to believe that you had called it a release. If you'll keep me informed or work with me, I can prepare packages. I've already done it for myself based on SVN. How far are you from cutting it as a release?

[dag (dag-wieers)]

On Sat, 7 Nov 2009, Noel J. Bergman wrote:

> @dag, OK. The comments in the change log led me to believe that you had
> called it a release. If you'll keep me informed or work with me, I can
> prepare packages. I've already done it for myself based on SVN. How far
> are you from cutting it as a release?

I wanted to get some more testing before doing an official release. But
since that has been keeping me in the past from releases I don't know who
I can ask for testing.

In the past bug-reports only came in from distribution packages. I don't
think many people test subversion releases :-/ If you have hints on how to
get more people testing subversion, let me know.

--
-- dag wieers, <email address hidden>, http://dag.wieers.com/ --
[Any errors in spelling, tact or fact are transmission errors]

[JC Hulce (soaringsky)]

This bug has been marked as Fix Committed for over a year. If the fix has made it into Ubuntu, please mark this bug as Fix Released. If the fix has been released upstream, but not Ubuntu, create a new bug asking for the new version and tag it with upgrade-software-version. If this bug has not been fixed anywhere, change the status back to Confirmed.

[James Cowgill (jcowgill)]

This was fixed in 0.7.0 (lucid I believe?)