bandwidth metering - Creating meter label rule doesn't match the metering concept.

[leegayeon]

In the following bug report, "remote_ip_prefix" is considered to be "source address/cidr" for ingress traffic, but this is not suitable for metering concepts.
https://bugs.launchpad.net/neutron/+bug/1528137

┌────┐ ┌────┐ ┌────┐
│external│──────────│router02│───────│ VMs │
└────┘ 100.100.20.0/24 └────┘ 10.0.1.0/24 └────┘
        │
        │ ┌────┐
        └─────────│ VMs │
         20.0.1.0/24 └────┘

In case of ingress traffic(inbound), source should be 0.0.0.0/0 and destination should be address/cidr of VMs .
That way, it is possible to meter bandwidth per address/cidr of VMs.

This is my test case.

1. Create Label
# neutron meter-label-create --tenant-id $TEANAT_ID --description "leegy" meter_ingress
Created a new metering_label:
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| description | leegy |
| id | b1c41f6f-3504-441d-aaa6-d655ca76bc08 |
| name | meter_ingress |
| project_id | e8c282b3d5e94776a655314e7ab86985 |
| shared | False |
| tenant_id | e8c282b3d5e94776a655314e7ab86985 |
+-------------+--------------------------------------+

2. Create rule
ingress rule(traffic from qg- interface to gr- interface), remote_ip_prefix is network cidr of VMs.

# neutron meter-label-rule-create --tenant-id $TENANT_ID --direction ingress $LABEL_ID 10.0.1.0/24
Created a new metering_label_rule:
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| direction | ingress |
| excluded | False |
| id | f9829983-fe3b-4848-8983-e3667dfe64df |
| metering_label_id | b1c41f6f-3504-441d-aaa6-d655ca76bc08 |
| remote_ip_prefix | 10.0.1.0/24 |
+-------------------+--------------------------------------+

3. Check iptables rules
I want to meter bandwidth from external to VMs.

[expected rules]
Chain neutron-meter-r-b1c41f6f-350 (1 references)
 pkts bytes target prot opt in out source destination
    0 0 neutron-meter-l-b1c41f6f-350 all -- qg-3f62cc89-83 * 0.0.0.0/0 10.0.1.0/24

[but result is...]
Chain neutron-meter-r-b1c41f6f-350 (1 references)
 pkts bytes target prot opt in out source destination
    0 0 neutron-meter-l-b1c41f6f-350 all -- qg-3f62cc89-83 * 10.0.1.0/24 0.0.0.0/0

4. Modify neutron source
neutron/services/metering/drivers/iptables/iptables_driver.py

    def _prepare_rule(self, ext_dev, rule, label_chain):
        remote_ip = rule['remote_ip_prefix']
        if rule['direction'] == 'egress':
            #dir_opt = '-d %s -o %s' % (remote_ip, ext_dev)
            dir_opt = '-s %s -o %s' % (remote_ip, ext_dev)
        else:
            #dir_opt = '-s %s -i %s' % (remote_ip, ext_dev)
            dir_opt = '-d %s -i %s' % (remote_ip, ext_dev)

        if rule['excluded']:
            ipt_rule = '%s -j RETURN' % dir_opt
        else:
            ipt_rule = '%s -j %s' % (dir_opt, label_chain)
        return ipt_rule

5. Check iptables rules
possble to meter the bandwidth from external to VMs.

Chain neutron-meter-r-b1c41f6f-350 (1 references)
 pkts bytes target prot opt in out source destination
    0 0 neutron-meter-l-b1c41f6f-350 all -- qg-3f62cc89-83 * 0.0.0.0/0 10.0.1.0/24

6. ping test
ping from qdhcp-namespace of VM network to another router gateway ip

# neutron net-list
+--------------------------------------+-----------+----------------------------------------+
| id | name | subnets |
+--------------------------------------+-----------+----------------------------------------+
| 19bd6565-07a1-4df3-9999-420cb5d01e0a | network02 | c00c950e-e4ac-4d79-915c-535114a4e401 |
| | | 10.0.1.0/24 |
| dca679c6-e294-49ef-addd-30fd6d6d0c53 | public2 | 47458829-cc7b-498d-8dd6-2a97c797cc61 |
| | | 100.100.20.0/24 |
+--------------------------------------+-----------+----------------------------------------+

# neutron router-list
+----------------------------+------------+----------------------------+-------------+-------+
| id | name | external_gateway_info | distributed | ha |
+----------------------------+------------+----------------------------+-------------+-------+
| 03c2fe17-175b-46b1-92fd- | meter_test | {"network_id": | False | False |
| 02260c703b64 | | "dca679c6-e294-49ef-addd- | | |
| | | 30fd6d6d0c53", | | |
| | | "enable_snat": false, | | |
| | | "external_fixed_ips": | | |
| | | [{"subnet_id": "47458829 | | |
| | | -cc7b-498d- | | |
| | | 8dd6-2a97c797cc61", | | |
| | | "ip_address": | | |
| | | "100.100.20.14"}]} | | |
| 1a30bf59-2281-457e-842c- | router02 | {"network_id": | False | False |
| 248cf026978d | | "dca679c6-e294-49ef-addd- | | |
| | | 30fd6d6d0c53", | | |
| | | "enable_snat": true, | | |
| | | "external_fixed_ips": | | |
| | | [{"subnet_id": "47458829 | | |
| | | -cc7b-498d- | | |
| | | 8dd6-2a97c797cc61", | | |
| | | "ip_address": | | |
| | | "100.100.20.11"}]} | | |
+----------------------------+------------+----------------------------+-------------+-------+

# ip netns exec qdhcp-19bd6565-07a1-4df3-9999-420cb5d01e0a ping 100.100.20.14
PING 100.100.20.14 (100.100.20.14) 56(84) bytes of data.
64 bytes from 100.100.20.14: icmp_seq=1 ttl=63 time=1.22 ms
64 bytes from 100.100.20.14: icmp_seq=2 ttl=63 time=0.086 ms
64 bytes from 100.100.20.14: icmp_seq=3 ttl=63 time=0.097 ms
64 bytes from 100.100.20.14: icmp_seq=4 ttl=63 time=0.101 ms
64 bytes from 100.100.20.14: icmp_seq=5 ttl=63 time=0.097 ms
^C
--- 100.100.20.14 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.086/0.321/1.224/0.451 ms

7. Check iptables rules
ping response is captured (5 pkts)

# ip netns exec qrouter-1a30bf59-2281-457e-842c-248cf026978d iptables -L -n -v

Chain neutron-meter-l-b1c41f6f-350 (1 references)
 pkts bytes target prot opt in out source destination
    5 420 all -- * * 0.0.0.0/0 0.0.0.0/0

Chain neutron-meter-r-b1c41f6f-350 (1 references)
 pkts bytes target prot opt in out source destination
    5 420 neutron-meter-l-b1c41f6f-350 all -- qg-3f62cc89-83 * 0.0.0.0/0 10.0.1.0/24

+
OpenStack version : newton
platform : ubuntu + openstack, it is occured on redhat osp and packstack.

[Brian Haley (brian-haley)]

Can you create a patch for this? Thanks.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/504891

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/516055

[OpenStack Infra (hudson-openstack)]

Change abandoned by Miguel Lavalle (<email address hidden>) on branch: master
Review: https://review.openstack.org/504891
Reason: This patchset duplicates https://review.openstack.org/#/c/516055/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/516055
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=92db1d4a2c49b1f675b6a9552a8cc5a417973b64
Submitter: Zuul
Branch: master

commit 92db1d4a2c49b1f675b6a9552a8cc5a417973b64
Author: leegy <email address hidden>
Date: Sun Oct 29 05:35:55 2017 +0000

    Fix meter-label-rule creation

    The existing method is to meter outgoing traffic from the entire VM
    to a specific "remote_ip_addr"(in case of outbound),
    but the method I suggested can meter outgoing traffic
    from a particular subnet to external network.

    From a metering point of view,
    it does not matter which address/CIDR is the destination for outbound.
    But, it is important to know where(subnet or VM) the packet leaves from.
    That way you can see how much VMs are using the external network.
    For outbound metering, dst is 0.0.0.0/0 and src is network of VMs
    and for inbound metering, src is 0.0.0.0/0 and dst is network of VMs.

    When creating a meter-label-rule, it is correct
    to specify src address for "remote_ip_prefix" option for outbound
    and specify dst address for inbound.
    If you are confused by the term "remote_ip_prefix",
    you need to clarify this option.

    Change-Id: Ia3f3e26410a73d7b44abae637703fda41c4bc09a
    Closes-Bug: #1716913

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.0.0b2 development milestone.

[Christian Sarrasin (sxc731)]

Thanks for reporting this and fixing it @leegy!

For whoever else lands here: the upshot of this issue is the the Neutron metering agent is inoperative. We have successfully tested the patch on Pike (it's very easy to deploy).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/561667

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/561668

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/561668
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a01780203b259dc4218e31261333123b60833a03
Submitter: Zuul
Branch: stable/ocata

commit a01780203b259dc4218e31261333123b60833a03
Author: leegy <email address hidden>
Date: Sun Oct 29 05:35:55 2017 +0000

    Fix meter-label-rule creation

    The existing method is to meter outgoing traffic from the entire VM
    to a specific "remote_ip_addr"(in case of outbound),
    but the method I suggested can meter outgoing traffic
    from a particular subnet to external network.

    From a metering point of view,
    it does not matter which address/CIDR is the destination for outbound.
    But, it is important to know where(subnet or VM) the packet leaves from.
    That way you can see how much VMs are using the external network.
    For outbound metering, dst is 0.0.0.0/0 and src is network of VMs
    and for inbound metering, src is 0.0.0.0/0 and dst is network of VMs.

    When creating a meter-label-rule, it is correct
    to specify src address for "remote_ip_prefix" option for outbound
    and specify dst address for inbound.
    If you are confused by the term "remote_ip_prefix",
    you need to clarify this option.

    Change-Id: Ia3f3e26410a73d7b44abae637703fda41c4bc09a
    Closes-Bug: #1716913
    (cherry picked from commit 92db1d4a2c49b1f675b6a9552a8cc5a417973b64)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/561667
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a06e6b20f7ef6c9a94950e214aab77d1c164865a
Submitter: Zuul
Branch: stable/pike

commit a06e6b20f7ef6c9a94950e214aab77d1c164865a
Author: leegy <email address hidden>
Date: Sun Oct 29 05:35:55 2017 +0000

    Fix meter-label-rule creation

    The existing method is to meter outgoing traffic from the entire VM
    to a specific "remote_ip_addr"(in case of outbound),
    but the method I suggested can meter outgoing traffic
    from a particular subnet to external network.

    From a metering point of view,
    it does not matter which address/CIDR is the destination for outbound.
    But, it is important to know where(subnet or VM) the packet leaves from.
    That way you can see how much VMs are using the external network.
    For outbound metering, dst is 0.0.0.0/0 and src is network of VMs
    and for inbound metering, src is 0.0.0.0/0 and dst is network of VMs.

    When creating a meter-label-rule, it is correct
    to specify src address for "remote_ip_prefix" option for outbound
    and specify dst address for inbound.
    If you are confused by the term "remote_ip_prefix",
    you need to clarify this option.

    Change-Id: Ia3f3e26410a73d7b44abae637703fda41c4bc09a
    Closes-Bug: #1716913
    (cherry picked from commit 92db1d4a2c49b1f675b6a9552a8cc5a417973b64)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ocata-eol release.