[SRU] With QuantumManager, nova-network does not start dnsmasq during initialization

Bug #977759 reported by Mandar Vaze on 2012-04-10
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Mandar Vaze
Essex
Undecided
Unassigned
nova (Ubuntu)
Undecided
Unassigned
Precise
Undecided
Chuck Short

Bug Description

Scenario :

1. nova is configured to use QuantumManager
2. VMs are running
3. nova-network process is stopped/killed
4. All dnsmasq processes are killed
5. nova-network is restarted

Expected Result :
dnsmasq should be restarted when nova-network process comes up

Actual Result :
dnsmasq process is NOT started by nova-network during initialization

Tested On :
Branch : master
git commit nova : 384b758166ed72fc16da7c5ad18eedc044bbc633

Related branches

CVE References

Mandar Vaze (mandarvaze) on 2012-04-10
Changed in nova:
assignee: nobody → Mandar Vaze (mandarvaze)
Mandar Vaze (mandarvaze) wrote :

 nova/network/quantum/manager.py : update_dhcp() calls restart_dhcp() which checks and starts dnsmasq if needed.

NetworkManager calls self._setup_network_on_host from init_host
For FlatDHCPManager and VlanManager update_dhcp() is called from _setup_network_on_host

But QuantumManager does not call update_dhcp() during init_host() - hence the problem.

Mandar Vaze (mandarvaze) wrote :

Please see above comment for the root-cause.

Here is the proposed solution:

I've tested the following code and confirmed that killed dnsmasq are started with following code changes.

90a91
> super(QuantumManager, self).init_host()
108a110,124
>
> #Similar to FlatDHCPMananger
> def _setup_network_on_host(self, context, network):
> """Sets up network on this host."""
> network['dhcp_server'] = self._get_dhcp_ip(context, network)
> self.l3driver.initialize_gateway(network)
>
> if not FLAGS.fake_network:
> dev = self.driver.get_dev(network)
> self.driver.update_dhcp(context, dev, network)
> if(FLAGS.use_ipv6):
> self.driver.update_ra(context, dev, network)
> gateway = utils.get_my_linklocal(dev)
> self.db.network_update(context, network['id'],
> {'gateway_v6': gateway})

If I hear no comments on the above solution, I'm planning to go ahead with Gerrit review with above changes.

Fix proposed to branch: master
Review: https://review.openstack.org/6452

Changed in nova:
status: New → In Progress

Reviewed: https://review.openstack.org/6452
Committed: http://github.com/openstack/nova/commit/30c8293ea7d4c9464e0139fb488e202fd0947f52
Submitter: Jenkins
Branch: master

commit 30c8293ea7d4c9464e0139fb488e202fd0947f52
Author: Mandar Vaze <email address hidden>
Date: Wed Apr 11 01:43:22 2012 -0700

    QuantumManager will start dnsmasq during startup. Fixes bug 977759

    Added _setup_network_on_host method, which calls update_dhcp
    if quantum_use_dhcp is set.

    Change-Id: I193212037873001a03da7b7a484f61a5c13b5de8

Changed in nova:
status: In Progress → Fix Committed
Mandar Vaze (mandarvaze) on 2012-05-02
tags: added: essex-backport

Reviewed: https://review.openstack.org/7267
Committed: http://github.com/openstack/nova/commit/26dc6b75c73f10c2da7628ce59e225d1006d9d1c
Submitter: Jenkins
Branch: stable/essex

commit 26dc6b75c73f10c2da7628ce59e225d1006d9d1c
Author: Mandar Vaze <email address hidden>
Date: Wed Apr 11 01:43:22 2012 -0700

    QuantumManager will start dnsmasq during startup. Fixes bug 977759

    Added _setup_network_on_host method, which calls update_dhcp
    if quantum_use_dhcp is set.

    Change-Id: I193212037873001a03da7b7a484f61a5c13b5de8

tags: added: in-stable-essex
Devin Carlen (devcamcar) on 2012-05-22
Changed in nova:
milestone: none → folsom-1
importance: Undecided → Medium
Thierry Carrez (ttx) on 2012-05-23
Changed in nova:
status: Fix Committed → Fix Released
Chuck Short (zulcss) on 2012-05-30
Changed in nova (Ubuntu Precise):
status: New → In Progress
Changed in nova (Ubuntu):
status: New → In Progress

** Impact **

When using nova with quantum network manager, dnsmasq is started after nova-network has been restarted therefore no new IP addresses for isntances.

** Development Fix **

This has been fixed in the development release in https://review.openstack.org/6452 and fixed in quantal.

** Stable fix **

This has been fixed in the stable/essex tree in https://review.openstack.org/7267

** Test Case **

Scenario :

1. nova is configured to use QuantumManager
2. VMs are running
3. nova-network process is stopped/killed
4. All dnsmasq processes are killed
5. nova-network is restarted

Expected Result :
dnsmasq should be restarted when nova-network process comes up

Actual Result :
dnsmasq process is NOT started by nova-network during initialization

** Regression Impact **

Minimal this is a non-standard case on Ubuntu.

summary: - With QuantumManager, nova-network does not start dnsmasq during
+ [SRU] With QuantumManager, nova-network does not start dnsmasq during
initialization
Chuck Short (zulcss) on 2012-06-08
Changed in nova (Ubuntu Precise):
milestone: none → ubuntu-12.04.1
assignee: nobody → Chuck Short (zulcss)

Hello Mandar, or anyone else affected,

Accepted nova into precise-proposed. The package will build now and be available in a few hours. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in nova (Ubuntu Precise):
status: In Progress → Fix Committed
tags: added: verification-needed
Adam Gandelman (gandelman-a) wrote :

Please find the attached Jenkins job results from the Ubuntu Server Team's CI
infrastructure. As part of the verification process for this bug, Nova has
been deployed and configured across multiple nodes using precise-proposed as
an installation source. After successful bring-up and configuration of the
cluster, a number of exercises and smoke tests have be invoked to ensure the
updated package did not introduce any regressions. A number of test iterations
were carried out to catch any possible transient errors.

Note the list of installed packages at the top and bottom of the report.

For records of upstream test coverage of this update, please see the
Jenkins links in the comments of the relevant upstream code-review:

https://review.openstack.org/7267

As per the provisional Micro Release Exception granted to this package by
the Technical Board, we hope this contributes toward verification of this
update.

Dave Walker (davewalker) on 2012-07-03
tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 2012.1+stable~20120612-3ee026e-0ubuntu1

---------------
nova (2012.1+stable~20120612-3ee026e-0ubuntu1) precise-proposed; urgency=low

  * New upstream snapshot. (LP: #1010473)
  * Dropped, superseeded by new snapshot:
    - debian/patches/upstream/0001-fix-bug-where-nova-ignores-glance-host-in-imageref.patch
    - debian/patches/upstream/0002-Stop-libvirt-test-from-deleting-instances-dir.patch
    - debian/patches/upstream/0003-Allow-unprivileged-RADOS-users-to-access-rbd-volumes.patch
    - debian/patches/upstream/0004-Fixed-bug-962840-added-a-test-case.patch
    - debian/patches/upstream/0005-Populate-image-properties-with-project_id-again.patch
    - debian/patches/upstream/0006-Use-project_id-in-ec2.cloud._format_image.patc
    - debian/patches/CVE-2012-2101.patch
    - debian/patches/CVE-2012-2654.patch
  * Resynchronize with stable/essex:
    - 3ee026e Only invoke .lower() on non-None protocols. (LP: #1010514)
    - f0a9f47 Create a utf8 version of the dns_domains table. (LP: #993663)
    - 84a43e1 Report memory correctly on Xen. (LP: #997014)
    - 8c72924 Add libvirt get_console_output tests: pty and file. (LP: #990237)
    - 4e423cd Fix Multi_Scheduler to process host capabilities. (LP: #1000403)
    - 4aea7f1 Nail pep8 dependencies to 1.0.1
    - 2b3bbc4 handle updated qemu-img info output. (LP: #1000261)
    - 2d7d51c Fix type of snapshot_id column to match db. (LP: #962615)
    - ec70c69 Generate a Changelog for Nova
    - e5e890f Fix nova.tests.test_nova_rootwrap on Fedora 17. (LP: #992916)
    - 9e9a554 Ec2 handle strings with "0x" (LP: #983206)
    - 26dc6b7 QuantumManager will start dnsmasq during startup. Fixes (LP: #977759)
    - 7028d66 Introduced flag base_dir_name. (LP: #973194)
    - 76b525a Get unit tests functional in OS X.
    - facb936 Update KillFilter to handle 'deleted' exe's. (LP: #967931)
    - 1209af4 Checks if value is string or not before decode. (LP: #952176)
    - 1209af4 Fix timeout in EC2 CloudController.create_image(). (LP: #989764)
    - 108e74b Re-add console_log from console_console_output(). (LP: #987335)
    - 48a0768 Don't leak RPC connections on timeouts or other exceptions. (LP: #968843)
    - 7c64de9 Cloudpipe tap vpn not always working. (LP: #975043)
    - 5ab5051 add libvirt_inject_key flag fix (LP: #971640)
    - 6c68ef5 Xen: Pass session to destroy_vdi. (LP: #988615)
    - 015744e Delete fixed_ips when network is deleted. (LP: #754900)
  * Add debian/scripts/changelog.sh to help generate the changelog.
  * Add debian/nova-common.docs:
    - Include changelog and README.rst
  * debian/rules: Generate a tarball from git snapshot.
  * debian/patches/fix-pep8-errors.patch: Fix pep8 errors due to pep8 upstream
    migration.
 -- Chuck Short <email address hidden> Tue, 05 Jun 2012 09:50:59 -0400

Changed in nova (Ubuntu Precise):
status: Fix Committed → Fix Released
Chuck Short (zulcss) on 2012-07-31
Changed in nova (Ubuntu):
status: In Progress → Fix Released
Thierry Carrez (ttx) on 2012-09-27
Changed in nova:
milestone: folsom-1 → 2012.2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers