[OSSA-2021-004] Linuxbridge ARP filter bypass on Netfilter platforms (CVE-2021-38598)

[Jake Yip]

We are running an OpenStack cloud with linux bridge. We have found that, in certain conditions, ARP spoofing protection is not working as intended. This allows a user do bad things like spoof gratuitous ARP to DoS another user's virtual machine. More details below.

In an environment using linux bridge, neutron-linuxbridge-agent uses ebtables to prevent ARP spoofing. A list of typical ebtables rules for a VM looks like this:

 :neutronMAC-tapdb545a8c-8f DROP
 :neutronARP-tapdb545a8c-8f DROP
 -A PREROUTING -i tapdb545a8c-8f -j neutronMAC-tapdb545a8c-8f
 -A PREROUTING -p ARP -i tapdb545a8c-8f -j neutronARP-tapdb545a8c-8f
 -A neutronMAC-tapdb545a8c-8f -i tapdb545a8c-8f --among-src fa:16:3e:84:cd:b4 -j RETURN
 -A neutronARP-tapdb545a8c-8f -p ARP --arp-ip-src 192.0.2.5 -j ACCEPT

The neutronARP-xxx chain, however, has a problem during the creation of it. The source for that [1] looks like this:

 ebtables(['-N', vif_chain, '-P', 'DROP'])
 ebtables(['-F', vif_chain])

This creates a chain with default policy of DROP, and FLUSHes any existing rules.

However, we have found that in certain OS, the FLUSH reverts the default policy back to RETURN. E.g.

 root@jake-focal:~# eatables -t nat -N newchain -P DROP
 root@jake-focal:~# ebtables-save | grep newchain
 :newchain DROP
 root@jake-focal:~# ebtables -t nat -F newchain
 root@jake-focal:~# ebtables-save | grep newchain
 :newchain RETURN
 root@jake-focal:~# ebtables --version
 ebtables 1.8.4 (nf_tables)

The OSes that exhibit this issue seems to be OSes that uses ebtables-nft - Ubuntu Focal, CentOS Stream.

Ubuntu Bionic is fine. E.g.

 root@jake-bionic:~# ebtables -t nat -N newchain -P DROP
 root@jake-bionic:~# ebtables-save | grep newchain
 :newchain DROP
 root@jake-bionic:~# ebtables -t nat -F newchain
 root@jake-bionic:~# ebtables-save | grep newchain
 :newchain DROP
 root@jake-bionic:~# ebtables --version
 ebtables v2.0.10-4 (December 2011)

I have a patch for this, but as this is a security issue I am refraining from posting it up to OpenStack's Gerrit. Also, this might have been fixed in master, but it still affects Ussuri and Victoria. Please advise on what I should do next?

[1] https://opendev.org/openstack/neutron/src/branch/stable/ussuri/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py#L135-L139

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

Thank you for the report. That was solved in master and W in [1]. This patch flushes or creates the related "vif_chain" and then adds a default DROP rule [2][3].

I'll backport this patch up to Queens. Although the patch description is not related to this one, it is indeed solving what is described.

Regards.

[1]https://review.opendev.org/q/I9463b000f6f63e65aaf91d60b30f6c92c01e3baf
[2]https://review.opendev.org/c/openstack/neutron/+/785177/3/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py#122
[3]https://review.opendev.org/c/openstack/neutron/+/785177/3/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py#128

[Jeremy Stanley (fungi)]

Rodolfo: Thanks for working on this. Unfortunately when you pushed your backports today you linked to this private vulnerability report in the commit messages, breaking the embargo. As a result, I'm going ahead and switching the bug to Public Security status and proceeding under our public report workflow.

[Jeremy Stanley (fungi)]

We've been treating other unintended anti-spoofing filter bypasses as class A reports lately, most recently OSSA-2021-001 which had a similar impact and risks (though with differing details obviously), so I'll start putting together an impact description and we'll publish an advisory once the backports for stable/victoria and stable/ussuri merge.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Jeremy:

This is the first time I actually provide a patch for a bug with an embargo. Sorry for breaking it. Next time I'll be more careful.

Regards.

[Jeremy Stanley (fungi)]

Rodolfo: Don't sweat it, the other anti-spoofing bypass vulnerability I mentioned was public for a very long time before it was fixed, and I was going to start a conversation about switching this to public ahead of publication anyway. We just prefer to have the opportunity to discuss that decision first if the report was originally provided in private.

For future reference though, when a report of a suspected vulnerability is still in Private Security status we keep a paragraph at the top of the bug description which includes reminders not to mention or link the report in public code review, mailing lists, IRC and so on (I removed it from this bug when switching it to Public Security, but you can find it in the full activity log). The hope is that developers will read that notice before taking further action.

Anyway, thanks for the fixes and for moving this along, I'll push up a draft advisory we can all review, and then file a CVE assignment request with that text once we're satisfied with the details in it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ossa/+/804116

[Jeremy Stanley (fungi)]

Please review the proposed advisory text and affected version ranges in https://review.opendev.org/804116 and, if that looks correct, I'll apply to MITRE for a CVE using that description.

Jake: I've credited the discovery to "Jake Yip" and "Nectar" but please let me know if you'd like to go by a different name and/or list a different organizational affiliation (the university, maybe something else?). Thanks!

[Jake Yip (waipengyip)]

Hi Jeremy,

Please credit it to

"Jake Yip from ARDC and Justin Mammarella from the University of Melbourne"

Thanks!

[Jeremy Stanley (fungi)]

Jake: Thanks for the correction, I've updated it in the proposed advisory now as requested.

[Jeremy Stanley (fungi)]

I have requested a CVE through MITRE's form based on the summary in https://review.opendev.org/804116 and will update that proposed advisory, as well as this bug, with it once assigned.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/804056
Committed: https://opendev.org/openstack/neutron/commit/fafa5dacd5057120562184a734e7345e7c0e9639
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit fafa5dacd5057120562184a734e7345e7c0e9639
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Apr 7 13:16:21 2021 +0000

    Make ARP protection commands compatible with "ebtables-nft"

    "nftables" compatible binary, "ebtables-nft", is not 100% compatible
    with the legacy API, as reported in LP#1922892.

    This patch fixes the following issues when using "ebtables-nft" (while
    keeping compatibility with legacy binary):
    - When a new chain is created, a default DROP rule is added at the end
      of the chain (append). This will prevent the error code 4 when the
      chain is listed.
    - The chain rules are added at the begining of the chain (insert),
      before the default DROP rule. This will prioritize the port rules.
    - The MAC rules are cleaned before the new ones are added. That will
      prevent the deletion of any new needed rule, now added after the
      deletion.
    - The "ebtables" command will retry on error code 4. This is the
      error returned when the chains are listed and no rule is present
      in a new created chain (reporeted in LP#1922892).

    This code is backwards compatible, that means it works with the legacy
    "ebtables" binary; this is currently installed in the Neutron CI [1].
    In order to test with the new binary, "ebtables-nft", two new CI jobs
    are added to the periodic queue [2].

    [1]https://github.com/openstack/neutron/blob/1ad9ca56b07ffdc9f7e0bc6a62af61961b9128eb/roles/legacy_ebtables/tasks/main.yaml
    [2]https://review.opendev.org/c/openstack/neutron/+/785144

    Closes-Bug: #1922892
    Related-Bug: #1508155
    Closes-Bug: #1938670

    Conflicts:
        neutron/tests/unit/plugins/ml2/drivers/linuxbridge/agent/test_arp_protect.py

    Change-Id: I9463b000f6f63e65aaf91d60b30f6c92c01e3baf
    (cherry picked from commit 0a931391d8990f3e654b4bfda24ae4119c609bbf)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/804057
Committed: https://opendev.org/openstack/neutron/commit/f034de001ba224c3add364ca5d00cc15e26c2e96
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit f034de001ba224c3add364ca5d00cc15e26c2e96
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Apr 7 13:16:21 2021 +0000

    Make ARP protection commands compatible with "ebtables-nft"

    "nftables" compatible binary, "ebtables-nft", is not 100% compatible
    with the legacy API, as reported in LP#1922892.

    This patch fixes the following issues when using "ebtables-nft" (while
    keeping compatibility with legacy binary):
    - When a new chain is created, a default DROP rule is added at the end
      of the chain (append). This will prevent the error code 4 when the
      chain is listed.
    - The chain rules are added at the begining of the chain (insert),
      before the default DROP rule. This will prioritize the port rules.
    - The MAC rules are cleaned before the new ones are added. That will
      prevent the deletion of any new needed rule, now added after the
      deletion.
    - The "ebtables" command will retry on error code 4. This is the
      error returned when the chains are listed and no rule is present
      in a new created chain (reporeted in LP#1922892).

    This code is backwards compatible, that means it works with the legacy
    "ebtables" binary; this is currently installed in the Neutron CI [1].
    In order to test with the new binary, "ebtables-nft", two new CI jobs
    are added to the periodic queue [2].

    [1]https://github.com/openstack/neutron/blob/1ad9ca56b07ffdc9f7e0bc6a62af61961b9128eb/roles/legacy_ebtables/tasks/main.yaml
    [2]https://review.opendev.org/c/openstack/neutron/+/785144

    Closes-Bug: #1922892
    Related-Bug: #1508155
    Closes-Bug: #1938670

    Conflicts:
        neutron/tests/unit/plugins/ml2/drivers/linuxbridge/agent/test_arp_protect.py

    Change-Id: I9463b000f6f63e65aaf91d60b30f6c92c01e3baf
    (cherry picked from commit 0a931391d8990f3e654b4bfda24ae4119c609bbf)
    (cherry picked from commit fafa5dacd5057120562184a734e7345e7c0e9639)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/804058
Committed: https://opendev.org/openstack/neutron/commit/4245963c71280bebd94e9005230d610931a7c2df
Submitter: "Zuul (22348)"
Branch: stable/train

commit 4245963c71280bebd94e9005230d610931a7c2df
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Apr 7 13:16:21 2021 +0000

    Make ARP protection commands compatible with "ebtables-nft"

    "nftables" compatible binary, "ebtables-nft", is not 100% compatible
    with the legacy API, as reported in LP#1922892.

    This patch fixes the following issues when using "ebtables-nft" (while
    keeping compatibility with legacy binary):
    - When a new chain is created, a default DROP rule is added at the end
      of the chain (append). This will prevent the error code 4 when the
      chain is listed.
    - The chain rules are added at the begining of the chain (insert),
      before the default DROP rule. This will prioritize the port rules.
    - The MAC rules are cleaned before the new ones are added. That will
      prevent the deletion of any new needed rule, now added after the
      deletion.
    - The "ebtables" command will retry on error code 4. This is the
      error returned when the chains are listed and no rule is present
      in a new created chain (reporeted in LP#1922892).

    This code is backwards compatible, that means it works with the legacy
    "ebtables" binary; this is currently installed in the Neutron CI [1].
    In order to test with the new binary, "ebtables-nft", two new CI jobs
    are added to the periodic queue [2].

    [1]https://github.com/openstack/neutron/blob/1ad9ca56b07ffdc9f7e0bc6a62af61961b9128eb/roles/legacy_ebtables/tasks/main.yaml
    [2]https://review.opendev.org/c/openstack/neutron/+/785144

    Closes-Bug: #1922892
    Related-Bug: #1508155
    Closes-Bug: #1938670

    Conflicts:
        neutron/tests/unit/plugins/ml2/drivers/linuxbridge/agent/test_arp_protect.py

    Change-Id: I9463b000f6f63e65aaf91d60b30f6c92c01e3baf
    (cherry picked from commit 0a931391d8990f3e654b4bfda24ae4119c609bbf)
    (cherry picked from commit fafa5dacd5057120562184a734e7345e7c0e9639)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ossa/+/804116
Committed: https://opendev.org/openstack/ossa/commit/5bfba3e739b9988206a51fc564a05cc32b23a791
Submitter: "Zuul (22348)"
Branch: master

commit 5bfba3e739b9988206a51fc564a05cc32b23a791
Author: Jeremy Stanley <email address hidden>
Date: Tue Aug 10 16:41:27 2021 +0000

    Add OSSA-2021-004 (CVE-2021-38598)

    Change-Id: I91b44e7fab3209170efd8dc594cb1b442ee48c2d
    Closes-Bug: #1938670

[Jeremy Stanley (fungi)]

I've distributed the advisory for this to relevant mailing lists now.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.4.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.2.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron train-eol release.