[RFE] Flow logs support for OpenStack Networking

[LIU Yulong]

What is Flow log:
https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-basics.html
https://www.alibabacloud.com/help/en/cen/user-guide/configure-a-flow-log

User traffic flow (connection 5-tuple) for one port or ports from one network (VPC) can be monitor and collect to the LOG service. Then it can be used for traffic analysis, attack detection and security.

It is not a port or NIC dimension, one port can have many flows (connections), the flow log will be something like this:
<neutron_port_id> <ip_src> <ip_dst> <l4_port_src> <l4_port_dst> <protocol> <accept/deny> <ingress/egress> <packets> <bytes> <collect_start_time> <collect_end_time>

More details about the Flow log record examples:
https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-records-examples.html

So for OpenStack Networking Service Neutron, how to:
1. write the security group rules accept/deny connection statistics (packets/bytes) data (does OVS have such ability, such as sFlow? conntrack is useful for such production?)
2. collect the data (neutron agents can do such work?)
3. report the data (metering-agent can report the data?)

[Brian Haley (brian-haley)]

Hi Liu,

We briefly talked about this in the Drivers meeting today, had some comments. Also, if you want to talk with the team more about it you would need to attend, see https://wiki.openstack.org/wiki/Meetings/NeutronDrivers for more info.

https://meetings.opendev.org/meetings/neutron_drivers/2024/neutron_drivers.2024-06-28-14.01.log.html#l-159

14:54:45 <slaweq> ovs can send sflow data to some monitoring tool IIRC
14:54:57 <slaweq> wouldn't that be enough?
14:55:11 <mlavalle> yes, ovs can do that
14:55:19 <mlavalle> I've tested it
14:55:33 <slaweq> for the SG rules accept/deny statistics we have SG logging - maybe that is enough
14:55:45 <slaweq> thx mlavalle for confirmation
14:56:09 <slaweq> I am not sure what data should neutron agents collets according to this rfe
14:56:28 <slaweq> I think this would require more detailed description IMO
14:56:36 <ralonsoh> I think we is thinking about OVS agent, but I'm just guessing
14:56:51 <slaweq> yes, probably
14:57:13 <haleyb> slaweq: right, there are some pieces in place, and i'm not sure either, but agree it is probably OVS related based on their deployments
14:57:15 <slaweq> but this agent can already be busy
14:58:53 <ralonsoh> can we request more info or to participate in this meeting?
14:59:15 <haleyb> I will put a comment in there asking, and yes, it would be better if he was in the meeting

[LIU Yulong (dragon889)]

Great to see the quick feedback.

For now, logging plugin and extenstion for ovs-agent seems not enough for the "Flow log":
1. it does not have each connection statistics data
2. packet is sent to neutron-openvswitch-agent, this is not so much efficiency. OVS-agent is a bit basy with other extensions now. There may be too many packets for ovs-agent to handle normal port_processing.

I'm not very familar with OVS sFlow, but seems it may have it's deficiencies as well:
1. lack of connections match rule for required packets
2. or it mirrors all packets to monitor? Performance side effect for ovs datapath?
3. still needs a new extension to configure the sFlow for ovs-agent