[RFE] (Operator-only) Add support 'firewall_group' for loggable resource type

[Cao Xuan Hoang]

This RFE is following up of the original RFE[1] to extend the work to support for FWaaS v2.

The purpose is to add support firewall_group for loggable resource type. In order to achieve that, we propose some changes:

- We need to make request validator and rpc callback to be more generic in order to support firewall_group. We already sent two ps for this action [2][3]
- On FWaaS, we will need to use above generic methods to register to Neutron side

Specification for this RFE also pushed to gerrit [4]

[1] https://bugs.launchpad.net/neutron/+bug/1468366
[2] https://review.openstack.org/#/c/529814/
[3] https://review.openstack.org/#/c/534227/
[4] https://review.openstack.org/#/c/509725/

[Armando Migliaccio (armando-migliaccio)]

Seems reasonable to tackle once bug 1468366 is released and documented.

[Miguel Lavalle (minsel)]

As of today, only two patches are left to be merged for https://bugs.launchpad.net/neutron/+bug/1468366. Both of those patches have been +2ed. So it seems timely that we discuss the next step described in the RFE in the drivers meeting

[Miguel Lavalle (minsel)]

This RFE was discussed in the Neutron drivers meeting. Agreed next steps are:

1) Reach out to FWaaS team to assess whether FWaaS 2.0 is mature enough to support logging
2) Request from submitter of this RFE more details on how this is planned to be done. Please read the discussion here http://eavesdrop.openstack.org/meetings/neutron_drivers/2018/neutron_drivers.2018-01-19-14.00.log.html#l-67 and provide suitable answers to the questions raised

[Sridar Kandaswamy (skandasw)]

Thanks Miguel, regarding (1) the API as such has been around for a few cycles now and is mature, we have also completed the support for VM ports in the Queens cycle - so i think we are in a good position to take on logging. And being a reporting feature it can also help with usability.

[Cuong Nguyen (cuongnv)]

Hi Miguel,

Regard questions of driver team, I've updated description of this RFE.
Basically, to achieve this RFE we plan to work on two main items:

- On Neutron side, make request validator and rpc callback to be more generic in order to support firewall_group.
- On FWaaS side, we will use above generic methods to register resource type to Neutron.

We also already pushed some ps and a spec for this RFE (mentioned in description), please help to take a look and give us some comments.

[Cuong Nguyen (cuongnv)]

Basically our solution is similar with what Ihar said in the discussion of driver team http://eavesdrop.openstack.org/meetings/neutron_drivers/2018/neutron_drivers.2018-01-19-14.00.log.html#l-84

[Miguel Lavalle (minsel)]

This RFE is approved

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/553738

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/509725
Committed: https://git.openstack.org/cgit/openstack/neutron-specs/commit/?id=6a1372079a7605d2205ebadbc89241f543750849
Submitter: Zuul
Branch: master

commit 6a1372079a7605d2205ebadbc89241f543750849
Author: Cao Xuan Hoang <email address hidden>
Date: Mon Oct 2 16:26:21 2017 +0700

    (Operator-only) Extend logging framework for FWaaS v2

    This spec follows up the original spec[1] to extend the
    work to support for FWaaS v2.

    [1] https://specs.openstack.org/openstack/neutron-specs/specs/pike/logging-API-for-security-group-rules.html

    Partial-Bug: #1720727

    Co-Authored-By: Yushiro FURUKAWA <email address hidden>
    Co-Authored-By: Nguyen Phuong An <email address hidden>
    Co-Authored-By: Cuong Nguyen <email address hidden>
    Change-Id: I409265229853f3d5c7e7a698c5b0787586ac1558

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/573952

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kim Bao Long (<email address hidden>) on branch: master
Review: https://review.openstack.org/573952
Reason: I will move these chains to neutron-fwaas

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/576338

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/574683
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=44e8c91f3c011b91865f85547e1f9fc3d9b4636c
Submitter: Zuul
Branch: master

commit 44e8c91f3c011b91865f85547e1f9fc3d9b4636c
Author: Nguyen Phuong An <email address hidden>
Date: Tue Jun 12 16:39:47 2018 +0700

    Introduce l3_logging extension base

    This patch introduces base class for L3 logging extension base.
    It will be used for SnatLogExtension and FWaaSLogExtension.

    Change-Id: I5372fa4d93f51692cebc9dc4df5b4fe79b1ef554
    Related-Bug: #1720727

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/578718

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/580575

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/580976

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/582498
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e82a17527d0c1a59ee92f53ba5fc6752c5034dc4
Submitter: Zuul
Branch: master

commit e82a17527d0c1a59ee92f53ba5fc6752c5034dc4
Author: Kim Bao Long <email address hidden>
Date: Fri Jul 13 17:23:21 2018 +0700

    [doc] Follow-up logging service for firewall_group

    This patch updates the networking guide to include 'firewall_group'
    as a new resource-type in 'Neutron Packet Logging Framework'

    Change-Id: I94d779fcabc64417d4d561c659270501c0651a66
    Co-Authored-By: Nguyen Phuong An <email address hidden>
    Partial-Bug: #1720727

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/534227
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a4ffcab4cae23f1f7b3cc31f1ef922af9582ac2c
Submitter: Zuul
Branch: master

commit a4ffcab4cae23f1f7b3cc31f1ef922af9582ac2c
Author: Cao Xuan Hoang <email address hidden>
Date: Tue Jan 16 17:22:59 2018 +0700

    [log] Generic RPC stuffs for logging in server side

    This patch intends to make generic RPC stuffs for logging in server side.
    So logging can be extended to be used by other resources like firewall.

    Partial-Bug: #1720727
    Change-Id: I6f93a71c2ae725e3b97d8abc5f57093894792116

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/529814
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cdb973e77dc33dcccfb8c62eed69e4b2318acd8d
Submitter: Zuul
Branch: master

commit cdb973e77dc33dcccfb8c62eed69e4b2318acd8d
Author: Cuong Nguyen <email address hidden>
Date: Fri Dec 22 17:09:20 2017 +0700

    Generic validate_request method for logging

    This patch adds a generic validate_request method for logging, So
    logging can be extended to other resources like firewall group.

    Co-Authored-By: Nguyen Phuong An <email address hidden>
    Co-Authored-By: Kim Bao Long <email address hidden>

    Partial-Bug: #1720727
    Change-Id: I9d580645131a9f1b8233ea5f6e0378cd98f023f1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/580575
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=00b923ddf37de2176168f5a525b4190dd4e04c4d
Submitter: Zuul
Branch: master

commit 00b923ddf37de2176168f5a525b4190dd4e04c4d
Author: Nguyen Phuong An <email address hidden>
Date: Fri Jul 6 14:38:04 2018 +0700

    Adding resources callback handler

    This patch allows logging plugin to handle resource callback
    events AFTER_UPDATE/AFTER_CREATE/AFTER_DELETE of security
    group, firewall group.

    Co-Authored-By: Kim Bao Long <email address hidden>
    Partial-Bug: #1720727
    Change-Id: I1cb170de1dbb7ac5380d386d850241f3c4a2f225

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/576338
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=3b1590ff69602b33c7ff14f5fcb6099d9a26fef4
Submitter: Zuul
Branch: master

commit 3b1590ff69602b33c7ff14f5fcb6099d9a26fef4
Author: Nguyen Phuong An <email address hidden>
Date: Tue Jun 19 10:34:35 2018 +0700

    FWaaS v2: L3 logging agent extension

    This patch introduces L3 logging agent extension for firewall group.
    It also configures the extension for devstack when log plugin is
    enabled.

    Co-Authored-By: Kim Bao Long <email address hidden>
    Partial-Bug: #1720727
    Change-Id: I4d9af5325f157fbb35ea6fdb25723268856a0db4

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/574128
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=d38f76e68a814d99597bebb9bc60ce8a76ee4af7
Submitter: Zuul
Branch: master

commit d38f76e68a814d99597bebb9bc60ce8a76ee4af7
Author: Nguyen Phuong An <email address hidden>
Date: Mon Jun 11 10:45:10 2018 +0700

    Introduce accepted/dropped/rejected chains for future processing

    This patch introduces new neutron-l3-agent-accepted/dropped/rejected
    chains. These chains are used for future processing like Firewall logging.

    Co-Authored-By: Kim Bao Long <email address hidden>
    Partial-Bug: #1720727
    Change-Id: Iecc5f3a9e32c7b17848e2ed89d9bcbd85900ed5a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/578718
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=d865155757b4c46488083c1827623d635352c62a
Submitter: Zuul
Branch: master

commit d865155757b4c46488083c1827623d635352c62a
Author: Nguyen Phuong An <email address hidden>
Date: Thu Jun 28 14:32:46 2018 +0700

    Add notification callback events for FWaaS v2

    This patch adds notification callback events (AFTER_CREATE,
    AFTER_DELETE, AFTER_UPDATE) for firewall group, firewall policy and
    firewall rule resource. These events could be used by the other service
    such as logging.

    TODO: UTs should be added to cover all of code of driver_api in Stein.

    Co-Authored-By: Kim Bao Long <email address hidden>
    Change-Id: Ic500270718c128284e337301fb2af0e04d850b5d
    Partial-Bug: #1720727

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/589991

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/532792
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=dd9aec1745838c1b9a5ce252d21e0fa5e02b01bb
Submitter: Zuul
Branch: master

commit dd9aec1745838c1b9a5ce252d21e0fa5e02b01bb
Author: Cuong Nguyen <email address hidden>
Date: Thu Jan 11 17:08:48 2018 +0700

    Add log validator for FWaaS v2 logging

    This patch adds a method to validate logging request for FWaaS.
    It also introduces get_fwg_attached_to_port method in db_v2 as
    helper function for logging feature.

    Co-Authored-By: Nguyen Phuong An <email address hidden>
    Co-Authored-By: Kim Bao Long <email address hidden>
    Partial-Bug: #1720727
    Change-Id: I723b48a75ba3f19476785bec0c5d5295d04aac95

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/530715
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=fb1dab78575a3f67eb60e6675bbd1ffcb1b32228
Submitter: Zuul
Branch: master

commit fb1dab78575a3f67eb60e6675bbd1ffcb1b32228
Author: Cao Xuan Hoang <email address hidden>
Date: Tue Jan 2 15:52:47 2018 +0700

    [log]: Add rpc stuff for logging

    This patch adds rpc stuff for logging extension as below:
    - Reserves a rpc listener for listening callback from log drivers.
    - Also provides log_db_api:
            + get_logs_bound_port
            + get_logs_bound_fwg
            + get_fwg_log_info_for_port
            + get_fwg_log_info_for_log_resources
    This patch also introduces get_fwg_ports_in_tenant method in
    firewall_db_v2.py as helper function for further processing.

    Co-Authored-By: Nguyen Phuong An <email address hidden>
    Co-Authored-By: Kim Bao Long <email address hidden>
    Partial-Bug: #1720727
    Change-Id: I6ad3fcec9a233286fdbd039d6d0e3a842c027f3f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/580976
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=ea430ee82ec4b777b6156962f65f844ae8a23966
Submitter: Zuul
Branch: master

commit ea430ee82ec4b777b6156962f65f844ae8a23966
Author: Kim Bao Long <email address hidden>
Date: Mon Jul 9 16:22:52 2018 +0700

    Adding resources callback handler for FWaaS v2 logging

    This patch allows logging plugin to trigger logging driver in event of
    receiving callback events AFTER_UPDATE/AFTER_CREATE/AFTER_DELETE from
    firewall group resources.

    Co-Authored-By: Nguyen Phuong An <email address hidden>
    Partial-Bug: #1720727
    Change-Id: I4e529273fe978d52e8948b844880a48f342fbd8c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/530694
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=15cabc798d35a0fad854cba3886b05344263c4a9
Submitter: Zuul
Branch: master

commit 15cabc798d35a0fad854cba3886b05344263c4a9
Author: Nguyen Phuong An <email address hidden>
Date: Mon Jul 9 13:22:22 2018 +0700

    Add python binding for libnetfilter_log

    This patch adds a python binding for libnetfilter_log, it can be used
    to capture NFLOG packets in network namespace from logging service in
    FWaaS v2.

    NFLogWrapper should be covered by functional test. Functional tests
    should be added in the future.

    Co-Authored-By: Kim Bao Long <email address hidden>
    Partial-Bug: #1720727
    Change-Id: I2bf0beac5ba373d47c226927d1922f3eb59af501

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/553738
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=507392be7de4d0c3711eb9143fcc4e9455371470
Submitter: Zuul
Branch: master

commit 507392be7de4d0c3711eb9143fcc4e9455371470
Author: Cuong Nguyen <email address hidden>
Date: Thu May 17 17:08:16 2018 +0700

    [log] FWaaS L3 Logging driver based iptables

    This patch implements logging driver in L3 for firewall group
    base discussed on the patch [1]

    [1] https://review.openstack.org/#/c/509725/

    Co-Authored-By: Nguyen Phuong An <email address hidden>
    Co-Authored-By: Kim Bao Long <email address hidden>
    Partial-Bug: #1720727
    Change-Id: I1194a622c546068991f44559e3f9e343430fd6f9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/589991
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=5b3ac1ebda160e80ba3a38c1f113f85b797572d6
Submitter: Zuul
Branch: master

commit 5b3ac1ebda160e80ba3a38c1f113f85b797572d6
Author: Yushiro FURUKAWA <email address hidden>
Date: Tue Aug 7 22:21:55 2018 +0900

    Add releasenote for FWaaS v2 logging

    This commit adds releasenote for FWaaS v2 logging feature[1].

    [1] https://specs.openstack.org/openstack/neutron-specs/specs/rocky/extend-logging-framework-to-support-for-FWaaS-v2.html

    Partial-Bug: #1720727
    Change-Id: Ib7df1d574004f3a0eb45786c3995d715022dc46d

[Slawek Kaplonski (slaweq)]

Looking at patches merged and related to this I'm marking that as fixed now.