Deallocation of fixed IP occurs before security group refresh – leading to potential security issue in error / race conditions

[Phil Day]

In network/manager/deallocate_fixed_ip the call to mark the IP as no longer allocated occurs before the call to update security group rules. This means that if an error occurs in the security group processing, or if for some reason it is very slow there is a risk that that the address is reused by another tenant before the rules relating to this address have been fully revoked.

Moving the db call to after the call to trigger the security group refresh would avoid the issue when an error occurs (the fixed_ip should not be released in this case).

Slowness in updating security group rules is much harder to deal with without changing the calls to refresh rules into an rpc.call() – which would have performance impact. In the case where force_dhcp_release in not set then there is a reasonable delay before the address is reused. Maybe a mechanism is required for a generally less aggressive way of recycling fixed_IPs – for example selecting the address with the oldest updated_at timestamp rather than just picking the first free address in the table.

[Russell Bryant (russellb)]

This does appear to have some security implication. I would consider it a "Low" severity security vulnerability, since I'm not sure there is a way to do a targeted exploit with this. Even if it was exploited, it's a very small window of time if anything.

[Thierry Carrez (ttx)]

Agreed it's a welcome strengthening, not an exploitable security issue IMHO.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/10404

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/10404
Committed: http://github.com/openstack/nova/commit/44132acbe91092de1188a3015a2c7155b5ec774c
Submitter: Jenkins
Branch: master

commit 44132acbe91092de1188a3015a2c7155b5ec774c
Author: David McNally <email address hidden>
Date: Fri Jul 27 13:32:14 2012 +0100

    Moving where the fixed ip deallocation happens.

    Fixes bug 1021352.

    In network/manager/deallocate_fixed_ip the call
    to mark the IP as no longer allocated occurs before
    the call to update security group rules. This means
    that if an error occurs in the security group
    processing, or if for some reason it is very slow
    there is a risk that that the address is reused by
    another tenant before the rules relating to this address
    have been fully revoked.

    Moving the db call to after the call to trigger the
    security group refresh would avoid the issue when an
    error occurs (the fixed_ip should not be released in
    this case).

    Change-Id: Iaba1af5c9a17fbbb82e42522b1060773de61550a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/10499

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/10499
Committed: http://github.com/openstack/nova/commit/413c6413df10b854f8adfd2cbfcfe89fd12288ac
Submitter: Jenkins
Branch: stable/essex

commit 413c6413df10b854f8adfd2cbfcfe89fd12288ac
Author: David McNally <email address hidden>
Date: Fri Jul 27 13:32:14 2012 +0100

    Moving where the fixed ip deallocation happens.

    Fixes bug 1021352.

    In network/manager/deallocate_fixed_ip the call
    to mark the IP as no longer allocated occurs before
    the call to update security group rules. This means
    that if an error occurs in the security group
    processing, or if for some reason it is very slow
    there is a risk that that the address is reused by
    another tenant before the rules relating to this address
    have been fully revoked.

    Moving the db call to after the call to trigger the
    security group refresh would avoid the issue when an
    error occurs (the fixed_ip should not be released in
    this case).

    Change-Id: Iaba1af5c9a17fbbb82e42522b1060773de61550a
    (cherry picked from commit 44132acbe91092de1188a3015a2c7155b5ec774c)

[Adam Gandelman (gandelman-a)]

Please find the attached test log from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, Nova has been deployed and configured across multiple nodes using precise-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

Please Note the list of installed packages at the top and bottom of the report.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Trunk review: https://review.openstack.org/10404
Stable review: https://review.openstack.org/10499

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

[Adam Gandelman (gandelman-a)]

Test coverage log.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 2012.1.3+stable-20120827-4d2a4afe-0ubuntu1

---------------
nova (2012.1.3+stable-20120827-4d2a4afe-0ubuntu1) precise-proposed; urgency=low

  * New upstream snapshot, fixes FTBFS in -proposed. (LP: #1041120)
  * Resynchronize with stable/essex (4d2a4afe):
    - [5d63601] Inappropriate exception handling on kvm live/block migration
      (LP: #917615)
    - [ae280ca] Deleted floating ips can cause instance delete to fail
      (LP: #1038266)

nova (2012.1.3+stable-20120824-86fb7362-0ubuntu1) precise-proposed; urgency=low

  * New upstream snapshot. (LP: #1041120)
  * Dropped, superseded by new snapshot:
    - debian/patches/CVE-2012-3447.patch: [d9577ce]
    - debian/patches/CVE-2012-3371.patch: [25f5bd3]
    - debian/patches/CVE-2012-3360+3361.patch: [b0feaff]
  * Resynchronize with stable/essex (86fb7362):
    - [86fb736] Libvirt driver reports incorrect error when volume-detach fails
      (LP: #1029463)
    - [272b98d] nova delete lxc-instance umounts the wrong rootfs (LP: #971621)
    - [09217ab] Block storage connections are NOT restored on system reboot
      (LP: #1036902)
    - [d9577ce] CVE-2012-3361 not fully addressed (LP: #1031311)
    - [e8ef050] pycrypto is unused and the existing code is potentially insecure
      to use (LP: #1033178)
    - [3b4ac31] cannot umount guestfs (LP: #1013689)
    - [f8255f3] qpid_heartbeat setting in ineffective (LP: #1030430)
    - [413c641] Deallocation of fixed IP occurs before security group refresh
      leading to potential security issue in error / race conditions
      (LP: #1021352)
    - [219c5ca] Race condition in network/deallocate_for_instance() leads to
      security issue (LP: #1021340)
    - [f2bc403] cleanup_file_locks does not remove stale sentinel files
      (LP: #1018586)
    - [4c7d671] Deleting Flavor currently in use by instance creates error
      (LP: #994935)
    - [7e88e39] nova testsuite errors on newer versions of python-boto (e.g.
      2.5.2) (LP: #1027984)
    - [80d3026] NoMoreFloatingIps: Zero floating ips available after repeatedly
      creating and destroying instances over time (LP: #1017418)
    - [4d74631] Launching with source groups under load produces lazy load error
      (LP: #1018721)
    - [08e5128] API 'v1.1/{tenant_id}/os-hosts' does not return a list of hosts
      (LP: #1014925)
    - [801b94a] Restarting nova-compute removes ip packet filters (LP: #1027105)
    - [f6d1f55] instance live migration should create virtual_size disk image
      (LP: #977007)
    - [4b89b4f] [nova][volumes] Exceeding volumes, gigabytes and floating_ips
      quotas returns general uninformative HTTP 500 error (LP: #1021373)
    - [6e873bc] [nova][volumes] Exceeding volumes, gigabytes and floating_ips
      quotas returns general uninformative HTTP 500 error (LP: #1021373)
    - [7b215ed] Use default qemu-img cluster size in libvirt connection driver
    - [d3a87a2] Listing flavors with marker set returns 400 (LP: #956096)
    - [cf6a85a] nova-rootwrap hardcodes paths instead of using
      /sbin:/usr/sbin:/usr/bin:/bin (LP: #1013147)
    - [2efc87c] affinity filters don't work if scheduler_hints is None
      (LP: #1007573)
  ...

[Clint Byrum (clint-fewbar)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.