tc class statistics rates are all zero after upgrade to Trusty

[Visual Echo]

After upgrading an Ubuntu server from Lucid -> Precise -> Trusty the command:
tc -s class show dev eth0
no longer shows rates for my classification bands. This worked in Lucid, but not sure if something changed, or if I need to set some system configuration to make it work again. My current non-working version is:

$ dpkg -s iproute2
Package: iproute2
Status: install ok installed
Priority: important
Section: net
Installed-Size: 1147
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Multi-Arch: foreign
Version: 3.12.0-2
Replaces: iproute
Provides: arpd
Depends: libc6 (>= 2.14), libdb5.3
Recommends: libatm1 (>= 2.4.1-17~), libxtables10
Suggests: iproute2-doc
Conflicts: arpd, iproute (<< 20130000-1)
Conffiles:
 /etc/iproute2/group 3aea2c0e0dd75e13a5f8f48f2936915f
 /etc/iproute2/ematch_map b91e7f9b26918449bade9573f8871d61
 /etc/iproute2/rt_realms 7137bdf40e8d58c87ac7e3bba503767f
 /etc/iproute2/rt_tables a1313318d6778fe6b8c680248ef5a463
 /etc/iproute2/rt_dsfield 4264d5c7c8298300185aa04e1a736934
 /etc/iproute2/rt_protos 95ce0b4b5b79f5a8a45941fb418a904c
 /etc/iproute2/rt_scopes 6298b8df09e9bda23ea7da49021ca457
Description: networking and traffic control tools
 The iproute2 suite is a collection of utilities for networking and
 traffic control.
 .
 These tools communicate with the Linux kernel via the (rt)netlink
 interface, providing advanced features not available through the
 legacy net-tools commands 'ifconfig' and 'route'.
Original-Maintainer: Debian iproute2 Maintainers <email address hidden>
Homepage: http://www.linux-foundation.org/en/Net:Iproute2

Here is the current output of the command, note that 'rate 0bit 0pps' is common to all classes, in Lucid this would produce non-zero values for active traffic:

$ tc -s class show dev eth0
class htb 2:1 root rate 4000Kbit ceil 10000Kbit burst 1600b cburst 1600b
 Sent 140162776 bytes 377793 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
 lended: 23022 borrowed: 0 giants: 0
 tokens: 46812 ctokens: 18725

class htb 2:2 parent 2:1 leaf 3: prio 0 rate 1000Kbit ceil 4000Kbit burst 1600b cburst 1600b
 Sent 2021584 bytes 6949 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
 lended: 6505 borrowed: 189 giants: 0
 tokens: 187250 ctokens: 46812

class htb 2:3 parent 2:1 leaf 4: prio 1 rate 2000Kbit ceil 4000Kbit burst 1600b cburst 1600b
 Sent 16695548 bytes 216855 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
 lended: 216846 borrowed: 9 giants: 0
 tokens: 94375 ctokens: 47187

class htb 2:4 parent 2:1 leaf 5: prio 2 rate 1000Kbit ceil 4000Kbit burst 1600b cburst 1600b
 Sent 2963040 bytes 46619 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
 lended: 46619 borrowed: 0 giants: 0
 tokens: 193250 ctokens: 48312

class htb 2:5 parent 2:1 leaf 6: prio 3 rate 1000Kbit ceil 4000Kbit burst 1600b cburst 1600b
 Sent 99996 bytes 745 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
 lended: 745 borrowed: 0 giants: 0
 tokens: 183000 ctokens: 45750

class htb 2:6 parent 2:1 leaf 7: prio 4 rate 1000Kbit ceil 4000Kbit burst 1600b cburst 1600b
 Sent 80608262 bytes 79347 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
 lended: 50544 borrowed: 13902 giants: 0
 tokens: 190750 ctokens: 47687

class htb 2:7 parent 2:1 leaf 8: prio 5 rate 1000Kbit ceil 4000Kbit burst 1600b cburst 1600b
 Sent 37606539 bytes 26129 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
 lended: 5617 borrowed: 8922 giants: 0
 tokens: -178485 ctokens: 24056

class htb 2:8 parent 2:1 leaf 9: prio 6 rate 512000bit ceil 1000Kbit burst 1600b cburst 1600b
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
 lended: 0 borrowed: 0 giants: 0
 tokens: 390625 ctokens: 200000

class htb 2:9 parent 2:1 leaf a: prio 7 rate 1000Kbit ceil 1000Kbit burst 1600b cburst 1600b
 Sent 167807 bytes 1149 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
 lended: 1149 borrowed: 0 giants: 0
 tokens: 189875 ctokens: 189875

Did something change upstream to make the rates unavailable? Do I need to set some config to enable collection? Running the command as root does not make a difference. My kernel is:

Linux srvr1 3.13.0-46-generic #75-Ubuntu SMP Tue Feb 10 15:24:04 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Thank you!

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[JuanJo Ciarlante (jjo)]

By installing different kernel versions (trusty, manual download and dpkg -i),
I narrowed this down to:
- linux-image-3.8.0-44-generic: OK
- linux-image-3.11.0-26-generic: BAD (zero rate counters).

FYI I used this script:
# cat htb.sh
/sbin/tc qdisc del dev eth0 root
/sbin/tc qdisc add dev eth0 root handle 1: htb default 30
/sbin/tc class add dev eth0 parent 1: classid 1:1 htb rate 10000kbit burst 15k
/sbin/tc class add dev eth0 parent 1:1 classid 1:10 htb rate 2500kbit burst 15k
/sbin/tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip src 0/0 flowid 1:10

[JuanJo Ciarlante (jjo)]

FYI this has been reported to debian also (kernel 3.16):
https://lists.debian.org/debian-kernel/2014/11/msg00288.html

[JuanJo Ciarlante (jjo)]

FYI linux-image-4.1.0-040100rc1-generic_4.1.0-040100rc1.201504270235_i386.deb
(from ~kernel-ppa) failed the same way.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[JuanJo Ciarlante (jjo)]

FYI there are several changes at
https://www.kernel.org/pub/linux/kernel/v3.0/ChangeLog-3.10.12
that refer to htb rate handling.

[JuanJo Ciarlante (jjo)]

FYI tried iproute2-3.19.0, same zero rate output.

[Cristian (cristian-calabro)]

Same problem here. I'm using Ubuntu 14.04 trusty, kernel 3.13.0-24-generic.

[nico (laranicolas)]

Zero rate output :(

I leave my version.

Ubuntu 14.04.2 LTS (kernel 3.13.0-49-generic)

[penalvch (penalvch)]

Visual Echo, could you please test the latest upstream kernel available from the very top line at the top of the page (the release names are irrelevant for testing, and please do not test the daily folder) following https://wiki.ubuntu.com/KernelMainlineBuilds ? It will allow additional upstream developers to examine the issue.

If the test did not allow you to test to the issue (ex. you couldn't boot into the OS) please make a comment in your report about this, and continue to test the next most recent kernel version until you can test to the issue. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this bug is fixed in the mainline kernel, please add the following tags by clicking on the yellow circle with a black pencil icon, next to the word Tags, located at the bottom of the report description:
kernel-fixed-upstream
kernel-fixed-upstream-X.Y-rcZ

Where XY and Z are numbers corresponding to the kernel version.

If the mainline kernel does not fix this bug, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-X.Y-rcZ

Once testing of the upstream kernel is complete, please mark this bug's Status as Confirmed. Please let us know your results.

Thank you for your understanding.

[JuanJo Ciarlante (jjo)]

@peanlvch: FYI as per comment #4 I already tested v4.1-rc1-vivid, same bad results.

[Robie Basak (racb)]

@jjo

Please update the tags and change the status following peanlvch's instructions in comment #10. I'm not sure exactly which version you tested. Otherwise the bug may not reappear in the appropriate triage queue and may get lost.

[JuanJo Ciarlante (jjo)]

With kernels from http://kernel.ubuntu.com/~kernel-ppa/mainline/,
I've narrowed down to:
* OK: tc-class-stats.3.10.76-031076-generic.txt: rate 1600bit 2pps backlog 0b 0p requeues 0
* BAD: tc-class-stats.3.11.0-031100rc1-generic.txt: rate 0bit 0pps backlog 0b 0p requeues 0
* BAD: tc-class-stats.3.11.0-031100-generic.txt: rate 0bit 0pps backlog 0b 0p requeues 0

will update the tags as per above comment, tnx.

[JuanJo Ciarlante (jjo)]

As per comment #13, I've added the following tags:
* kernel-fixed-upstream-3.10
* kernel-bug-exists-upstream kernel-bug-exists-upstream-3.11rc1 kernel-bug-exists-upstream-4.1-rc1

Please correct them if I misunderstood the naming convention,

FYI my narrowed bisect corresponds to:
*** OK ***:
linux (3.10.76-031076.201504291035) saucy; urgency=low

  * Mainline build at commit: v3.10.76

*** BAD ***:
linux (3.11.0-031100rc1.201307141935) saucy; urgency=low

  * Mainline build at commit: v3.11-rc1

[JuanJo Ciarlante (jjo)]

FYI peeking at patch-3.11-rc1, shows

[...]
- struct gnet_stats_rate_est tcfc_rate_est;
+ struct gnet_stats_rate_est64 tcfc_rate_est;

with its correspondent addition:

+ * struct gnet_stats_rate_est64 - rate estimator
+ * @bps: current byte rate
+ * @pps: current packet rate
+ */
+struct gnet_stats_rate_est64 {
+ __u64 bps;
+ __u64 pps;
+};

FYI modding iproute2 3.19.0 to show EST64 vs EST(32) shows
tc using EST32:
http://paste.ubuntu.com/10963208/

[Robie Basak (racb)]

@Christopher

Why did you mark this Incomplete? It does not look Incomplete. As far as I can tell, all information requested has been provided.

[penalvch (penalvch)]

This report is missing the original reporter (Visual Echo) providing previously requested information.

Despite this, if you are having an issue, it will help immensely if you filed a new report via a terminal:
ubuntu-bug linux

Please feel free to subscribe me to it.

[Visual Echo (mr-crankypants)]

I'm terribly sorry but I think you have confused with a Linux Kernel
developer.

I run a site which uses Ubuntu, I found a bug, and I reported it. Try
http://pipes.mdve.net if you'd like to buy a wooden smoking pipe.

I'm a carpenter. Besides pushing work on other people, what is it that
*you* do?

Roger Venable
Ann Arbor, Michigan

On 2015-05-21 05:40, Christopher M. Penalver wrote:
> This report is missing the original reporter (Visual Echo) providing
> previously requested information.
>
> Despite this, if you are having an issue, it will help immensely if
> you filed a new report via a terminal:
> ubuntu-bug linux
>
> Please feel free to subscribe me to it.
>
> ** Changed in: linux (Ubuntu)
> Status: Confirmed => Incomplete
>
> ** Tags removed: kernel-bug-exists-upstream
> kernel-bug-exists-upstream-3.11rc1 kernel-bug-exists-upstream-4.1-rc1
> kernel-fixed-upstream-3.10
> ** Tags added: needs-upstream-testing
>
> --
> You received this bug notification because you are subscribed to the
> bug
> report.
> https://bugs.launchpad.net/bugs/1426589
>
> Title:
> tc class statistics rates are all zero after upgrade to Trusty
>
> Status in iproute2 package in Ubuntu:
> Confirmed
> Status in linux package in Ubuntu:
> Incomplete
>
> Bug description:
> After upgrading an Ubuntu server from Lucid -> Precise -> Trusty the
> command:
> tc -s class show dev eth0
> no longer shows rates for my classification bands. This worked in
> Lucid, but not sure if something changed, or if I need to set some
> system configuration to make it work again. My current non-working
> version is:
>
> $ dpkg -s iproute2
> Package: iproute2
> Status: install ok installed
> Priority: important
> Section: net
> Installed-Size: 1147
> Maintainer: Ubuntu Developers <email address hidden>
> Architecture: amd64
> Multi-Arch: foreign
> Version: 3.12.0-2
> Replaces: iproute
> Provides: arpd
> Depends: libc6 (>= 2.14), libdb5.3
> Recommends: libatm1 (>= 2.4.1-17~), libxtables10
> Suggests: iproute2-doc
> Conflicts: arpd, iproute (<< 20130000-1)
> Conffiles:
> /etc/iproute2/group 3aea2c0e0dd75e13a5f8f48f2936915f
> /etc/iproute2/ematch_map b91e7f9b26918449bade9573f8871d61
> /etc/iproute2/rt_realms 7137bdf40e8d58c87ac7e3bba503767f
> /etc/iproute2/rt_tables a1313318d6778fe6b8c680248ef5a463
> /etc/iproute2/rt_dsfield 4264d5c7c8298300185aa04e1a736934
> /etc/iproute2/rt_protos 95ce0b4b5b79f5a8a45941fb418a904c
> /etc/iproute2/rt_scopes 6298b8df09e9bda23ea7da49021ca457
> Description: networking and traffic control tools
> The iproute2 suite is a collection of utilities for networking and
> traffic control.
> .
> These tools communicate with the Linux kernel via the (rt)netlink
> interface, providing advanced features not available through the
> legacy net-tools commands 'ifconfig' and 'route'.
> Original-Maintainer: Debian iproute2 Maintainers
> <email address hidden>
> Homepage: http://www.linux-foundation.org/en/Net:Iproute2
>
> Here is the current output of the command, note that 'rate 0bit 0pps'
> is common to all classes, in Lucid this would produce n...

[Matti-Oskari Leppänen (mongrelx)]

Please check that

cat /sys/module/sch_htb/parameters/htp_rate_est

is set to 1

[JuanJo Ciarlante (jjo)]

Indeed that had been the case, thx for replying.

[Visual Echo (mr-crankypants)]

>> Do I need to set some config to enable collection?

Absotively Fantabulous! That's solved it for me too. I (original bug poster) waited for two years to see this bug get fixed (or any answer to my question) and I have a reward for you, either in USD cash via PayPal or Bitcoin. Check your inbox, thank you again.