CI: Nova compute fails with KeyError: 'host' when installing undercloud

[Sagi (Sergey) Shnaidman]

Undercloud installation fails: nova-compute service fails to start with error "KeyError: 'host' ":

http://logs.openstack.org/75/452275/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-nv/8c798b8/logs/oooq/undercloud/var/log/nova/nova-compute.log.txt.gz#_2017-04-07_22_41_52_525

ERROR nova Traceback (most recent call last):
ERROR nova File "/usr/bin/nova-compute", line 10, in <module>
ERROR nova sys.exit(main())
ERROR nova File "/usr/lib/python2.7/site-packages/nova/cmd/compute.py", line 56, in main
ERROR nova topic=CONF.compute_topic)
ERROR nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 241, in create
ERROR nova periodic_interval_max=periodic_interval_max)
ERROR nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 117, in __init__
ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)
ERROR nova File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 538, in __init__
ERROR nova self.driver = driver.load_compute_driver(self.virtapi, compute_driver)
ERROR nova File "/usr/lib/python2.7/site-packages/nova/virt/driver.py", line 1624, in load_compute_driver
ERROR nova virtapi)
ERROR nova File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
ERROR nova return import_class(import_str)(*args, **kwargs)
ERROR nova File "/usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 151, in __init__
ERROR nova default='nova.virt.firewall.NoopFirewallDriver')
ERROR nova File "/usr/lib/python2.7/site-packages/nova/virt/firewall.py", line 37, in load_driver
ERROR nova return fw_class(*args, **kwargs)
ERROR nova File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/firewall.py", line 339, in __init__
ERROR nova self.nwfilter = NWFilterFirewall(kwargs['host'])
ERROR nova KeyError: 'host'
ERROR nova

http://logs.openstack.org/75/452275/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-nv/8c798b8/logs/oooq/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-04-07_22_41_52

INFO: Error: Systemd start for openstack-nova-compute failed!
INFO: journalctl log for openstack-nova-compute:
INFO: -- Logs begin at Fri 2017-04-07 21:44:33 UTC, end at Fri 2017-04-07 22:41:52 UTC. --
INFO: Apr 07 22:41:46 undercloud systemd[1]: Starting OpenStack Nova Compute Server...
INFO: Apr 07 22:41:52 undercloud systemd[1]: openstack-nova-compute.service: main process exited, code=exited, status=1/FAILURE
INFO: Apr 07 22:41:52 undercloud systemd[1]: Failed to start OpenStack Nova Compute Server.
INFO: Apr 07 22:41:52 undercloud systemd[1]: Unit openstack-nova-compute.service entered failed state.
INFO: Apr 07 22:41:52 undercloud systemd[1]: openstack-nova-compute.service failed.
INFO: 
INFO: Error: /Stage[main]/Nova::Compute/Nova::Generic_service[compute]/Service[nova-compute]/ensure: change from stopped to running failed: Systemd start for openstack-nova-compute failed!

http://logs.openstack.org/34/454134/6/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-nv/6d8f1f7/logs/oooq/undercloud/var/log/nova/nova-compute.log.txt.gz#_2017-04-07_23_18_24_352
http://logs.openstack.org/97/422297/9/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-nv/a24f76e/logs/oooq/undercloud/var/log/nova/nova-compute.log.txt.gz#_2017-04-07_22_23_11_236

[Michele Baldessari (michele)]

Likely this change in puppet-nova broke us:
commit b3ea24394c6010a1475c6d19fe798dfe698fcccf
Author: ZhongShengping <[e-mail]>
Date: Fri Apr 7 15:29:16 2017 +0800

    Deprecate firewall_driver option

    Option "firewall_driver" from group "DEFAULT" is deprecated for removal.
    Its value may be silently ignored in the future.

    Change-Id: I72805cb69150e38d48fe8f4f22aa6865536550fd
    Closes-Bug: #1680724

[Michele Baldessari (michele)]

So the problem is that we are using a puppet-nova version which has the change at comment#1 but we're using a nova package that has an older version where the following change has not been applied yet:
commit 064da0853848c44a495f20912d835fbafa800515
Author: Matt Riedemann <email address hidden>
Date: Tue Feb 28 17:10:54 2017 -0500

    Default firewall_driver to nova.virt.firewall.NoopFirewallDriver

    In Ocata we changed use_neutron=True as the default. When configuring
    Nova to use Neutron, one is supposed to set the firewall_driver
    config option to nova.virt.firewall.NoopFirewallDriver, but that was
    not the default. This change makes it the default to match the default
    for use_neutron.

    Change-Id: If012acdd0ef2a66948ae45288cccb766262efb68

Maybe we should just pin puppet-nova to a commit before b3ea24394c6010a1475c6d19fe798dfe698fcccf, until we get a newer openstack-nova package?

[Michele Baldessari (michele)]

Am trying a temprevert of puppet-nova here: https://review.openstack.org/454984
It'd be grand if folks can clue me in if that is the right place

[Emilien Macchi (emilienm)]

I'm reverting the puppet-nova patch: https://review.openstack.org/454991

[Emilien Macchi (emilienm)]

It sounds like https://review.openstack.org/#/c/439187/3/nova/conf/network.py was not suggesting to remove the option but to override it, like we already did in puppet-nova. The revert should fix it...

[Emilien Macchi (emilienm)]

Revert merged in puppet-nova, removing tripleo project from here. We'll need to figure out if something actually needs to be done in puppet-nova regarding the change in nova.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/454991
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=d19c93c55174246cad28a0aa64e1aeca9f960edb
Submitter: Jenkins
Branch: master

commit d19c93c55174246cad28a0aa64e1aeca9f960edb
Author: Emilien Macchi <email address hidden>
Date: Sat Apr 8 13:58:01 2017 +0000

    Revert "Deprecate firewall_driver option"

    It broke OVB jobs, let's revert it until we figure why.

    This reverts commit b3ea24394c6010a1475c6d19fe798dfe698fcccf.

    Change-Id: I8361ae0277db744cd38518217f89c6c4466e1fd4
    Related-Bug: #1681029