support fwaas v2 logging >= rocky

[James Page]

In order to support tracing of network traffic across an OpenStack deployment, logging of traffic traversing virtual routers on neutron-gateway/neutron-openvswitch units is required to have a complete picture of source -> firewall/router -> target network flows.

The FWaaS v2 driver supports a _log extension that is configured in the same way as the Neutron Security Group Log driver (which the charms already support).

Please add support for fwaas_v2_log for OpenStack Rocky or later.

This is somewhat complicated by the fact that fwaas_v2 is only available in the charms from stein onward; a new configuration option needs to be added to the neutron-api charm to support configuration of the version of the fwaas driver is to be used, along with a new configuration option to enable the log extension.

  fwaas-version: 1|2
  enable-firewall-group-logging: true|false

No migration path exists before stein from v1/v2 so if fwaas is already in use in Queens and Rocky deployments, the log feature will not be supportable.

[James Page (james-page)]

Sample log messages from gateway units:

2019-06-10 09:16:30 action=ACCEPT, project_id=f43842c4647d4912af7817a24c5044b5, log_resource_ids=['2c2353e9-b30b-495a-aa5f-4d720c4e3209'], port=0bf81ded-bf94-437d-ad49-063bba9be9bb, pkt=ethernet(dst='fa:16:3e:1e:ea:0a',ethertype=2048,src='fa:16:3e:41:6f:cc')ipv4(csum=11567,dst='192.168.21.182',flags=2,header_length=5,identification=11808,offset=0,option=None,proto=6,src='10.5.0.10',tos=0,total_length=60,ttl=63,version=4)tcp(ack=0,bits=2,csum=2889,dst_port=22,offset=10,option=[TCPOptionMaximumSegmentSize(kind=2,length=4,max_seg_size=8918), TCPOptionSACKPermitted(kind=4,length=2), TCPOptionTimestamps(kind=8,length=10,ts_ecr=0,ts_val=1575217414), TCPOptionNoOperation(kind=1,length=1), TCPOptionWindowScale(kind=3,length=3,shift_cnt=7)],seq=1144678318,src_port=58300,urgent=0,window_size=26754)

2019-06-10 09:16:34 action=DROP, project_id=f43842c4647d4912af7817a24c5044b5, log_resource_ids=['2c2353e9-b30b-495a-aa5f-4d720c4e3209'], port=0bf81ded-bf94-437d-ad49-063bba9be9bb, pkt=ethernet(dst='fa:16:3e:c6:58:5e',ethertype=2048,src='fa:16:3e:e0:2c:be')ipv4(csum=58033,dst='10.5.0.10',flags=2,header_length=5,identification=30869,offset=0,option=None,proto=6,src='192.168.21.182',tos=16,total_length=52,ttl=63,version=4)tcp(ack=4249435409,bits=17,csum=54161,dst_port=57906,offset=8,option=[TCPOptionNoOperation(kind=1,length=1), TCPOptionNoOperation(kind=1,length=1), TCPOptionTimestamps(kind=8,length=10,ts_ecr=1574867119,ts_val=512608)],seq=3550217559,src_port=22,urgent=0,window_size=3120)

2019-06-10 09:17:26 action=ACCEPT, project_id=f43842c4647d4912af7817a24c5044b5, log_resource_ids=['2c2353e9-b30b-495a-aa5f-4d720c4e3209'], port=0bf81ded-bf94-437d-ad49-063bba9be9bb, pkt=ethernet(dst='fa:16:3e:1e:ea:0a',ethertype=2048,src='fa:16:3e:41:6f:cc')ipv4(csum=59542,dst='192.168.21.182',flags=2,header_length=5,identification=29349,offset=0,option=None,proto=1,src='10.5.0.10',tos=0,total_length=84,ttl=63,version=4)icmp(code=0,csum=30536,data=echo(data=b'% \xfe\\\x00\x00\x00\x00%\xa4\x04\x00\x00\x00\x00\x00\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./01234567',id=29890,seq=1),type=8)

[James Page (james-page)]

Note that there is a bug in neutron-fwaas with regards to decoding the prefix of the log message under Python 3.

[James Page (james-page)]

bug 1832210 for the neutron-fwaas decoding issue.

[James Page (james-page)]

Note that this feature was introduced at Rocky.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/663934
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=27b4fb1538588690c9a5b851939bd6654eee5a6f
Submitter: Zuul
Branch: master

commit 27b4fb1538588690c9a5b851939bd6654eee5a6f
Author: Liam Young <email address hidden>
Date: Fri Jun 7 13:01:23 2019 +0000

    Add support for FWaaS v2 logging

    Enable support for configuration of FWaaS v2 firewall group
    logging. The feature can be enabled or disabled via the
    enable-firewall-group-logging flag.

    This feature is currently only enabled for FWaaS v2 at Stein
    for the charms (but is supported back to Queens in Neutron).

    Change-Id: I4c440e233ee16d4e756c575d8db70918ff062f3e
    Partial-Bug: 1831972

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/664249
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-openvswitch/commit/?id=9b0de9bbff162316f39f955c12d211345860972a
Submitter: Zuul
Branch: master

commit 9b0de9bbff162316f39f955c12d211345860972a
Author: James Page <email address hidden>
Date: Mon Jun 10 12:15:40 2019 +0100

    Add support for FWaaS v2 logging

    Enable support for configuration of FWaaS v2 firewall group
    logging.

    Configuration options mirror those for neutron-openvswitch
    for security group logging.

    This feature is currently only enabled for FWaaS v2 at Stein
    for the charms (but is supported back to Queens in Neutron).

    Change-Id: Ic60ee47078089c59ccb09b8659422e7ad7081149
    Partial-Bug: 1831972

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/663923
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=0a809a1a192401bfabc4215311302dec93feaf4b
Submitter: Zuul
Branch: master

commit 0a809a1a192401bfabc4215311302dec93feaf4b
Author: James Page <email address hidden>
Date: Fri Jun 7 13:33:34 2019 +0100

    Add support for FWaaS v2 logging

    Enable support for configuration of FWaaS v2 firewall group
    logging.

    Configuration options mirror those for neutron-openvswitch
    for security group logging.

    This feature is currently only enabled for FWaaS v2 at Stein
    for the charms (but is supported back to Queens in Neutron).

    Change-Id: If1b332eb0f581e9acba111f79ba578a0b7081dd2
    Partial-Bug: 1831972

[James Page (james-page)]

Marking this as fix committed; we have support @ stein; rocky is awkward due to the lack of v1->v2 migration tools.

ipfix monitoring fills the gap in earlier release combinations - its not perfect but does provide some visibility.