Deployment of env with 2 nodegroups on baremetal lab fails due to dnsmasq issue

[Alexander Gromov]

MOS 9.x, snapshot info: http://paste.openstack.org/show/561661/

fuel-devops 3.0.1

fuel-qa stable/mitaka

Steps to reproduce:
1. Download template.yaml from attachment and copy it to fuel-qa/system_test/tests_templates/tests_configs/

2. Export the following variables:

export ISO_PATH=<ISO_PATH>
export V_ENV_DIR=<VENV_PATH>
export MOSQA_IPMI_USER=<username>
export MOSQA_IPMI_PASSWORD=<password>

export EXTRA_RPM_REPOS='mos-os,http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/os-2016-06-23-135731/x86_64|mos-proposed,http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-2016-08-17-164322/x86_64|mos-updates,http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/updates-2016-06-23-135916/x86_64|mos-security,http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/security-2016-06-23-140002/x86_64|mos-holdback,http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/holdback-2016-06-23-140047/x86_64'

export UPDATE_FUEL_MIRROR='http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/os-2016-06-23-135731/x86_64 http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-2016-08-17-164322/x86_64 http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/updates-2016-06-23-135916/x86_64 http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/security-2016-06-23-140002/x86_64 http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/holdback-2016-06-23-140047/x86_64'

export EXTRA_DEB_REPOS='mos-proposed,deb http://mirror.seed-cz1.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-08-17-164322 mos9.0-proposed main restricted|mos-updates,deb http://mirror.seed-cz1.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-08-17-164322 mos9.0-updates main restricted|mos-security,deb http://mirror.seed-cz1.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-08-17-164322 mos9.0-security main restricted|mos-holdback,deb http://mirror.seed-cz1.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-08-17-164322 mos9.0-holdback main restricted'

export CONFIG_NAME=template
export UPDATE_MASTER=true
export ENV_NAME=mos_9.x_2_nodegroups_baremetal
export DISABLE_SSL=true

3. Deploy environment:
./run_system_test.py run 'system_test.create_deploy_ostf' --with-config $CONFIG_NAME

Expected results:
Deployment is successfully completed.

Actual results:
Deployment fails with the following error: http://paste.openstack.org/show/561647/
DNS doesn't work on the master node.
    [root@nailgun ~]# ping www.yandex.ru
    ping: unknown host www.yandex.ru
    [root@nailgun ~]#

Details:
http://pastebin.com/CJQy3V5z
DNS doesn't work after cluster creation and updating nodegroups on the master node.
dnsmasq service is not active because due to configuration (listen-address field is empty).

Deployment is successful for MOS 9.0 release ISO, but fails for MOS 9.x.

[Alexander Gromov (agromov)]

Also there are some problems with network configuration for some networks from nodegroup with id 2.

[Timur Nurlygayanov (tnurlygayanov)]

The issue affects all NFV / SRIOV / DPDK bare-metal automated test cases.

[Aleksandr Didenko (adidenko)]

Listen address was added in this patch:
https://review.openstack.org/#/c/349721/3/deployment/puppet/fuel/lib/puppet/parser/functions/create_dnsmasq_dhcp_ranges.rb

It looks like there's a problem with this new algorithm because on a broken env Fuel node has two admin networks from two nodegroups:

10.109.9.0/24 - default
10.109.2.0/24 - baremetal-rack01

Fuel node has only one admin interface:

2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet 10.109.9.2/24 brd 10.109.9.255 scope global enp0s3

Which is a valid setup since non-default admin networks may be forwarded via dhcrelay and they don't have to be directly connected to the Fuel node interfaces.

So dhcp range for default admin network is correct:

listen-address=10.109.9.2
dhcp-range=range_34782bcb,10.109.9.3,10.109.9.254,255.255.255.0,120m
dhcp-option=net:range_34782bcb,option:router,10.109.9.1
dhcp-boot=net:range_34782bcb,pxelinux.0,boothost,10.109.9.2
dhcp-match=set:ipxe,175
dhcp-option-force=tag:ipxe,210,http://10.109.9.2/cobbler/boot/

But for baremetal-rack01 nodegroup it's broken:

listen-address=
dhcp-range=range_892a8620,10.109.2.2,10.109.2.254,255.255.255.0,120m
dhcp-option=net:range_892a8620,option:router,10.109.2.1
dhcp-boot=net:range_892a8620,pxelinux.0,boothost,10.109.9.2
dhcp-match=set:ipxe,175
dhcp-option-force=tag:ipxe,210,http://10.109.9.2/cobbler/boot/

It should not have 'listen-address=' directive for baremetal-rack01 nodegroup.

[Maksim Malchuk (mmalchuk)]

need another new algorithm, the related issue (not duplicate) : https://bugs.launchpad.net/fuel/+bug/1616087

[Alex Schultz (alex-schultz)]

The algorithm should still be OK, the bug is in the listen-address= which is the template improperly adding the line when the address is not blank.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/359970

[Alex Schultz (alex-schultz)]

Also it's invalid for 10.0 because https://review.openstack.org/#/c/349735/ has yet to be merged

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/359970
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9a391046addf25c4031d58e901a3d864bc8a14e2
Submitter: Jenkins
Branch: stable/mitaka

commit 9a391046addf25c4031d58e901a3d864bc8a14e2
Author: Alex Schultz <email address hidden>
Date: Wed Aug 24 09:36:15 2016 -0600

    Fix dnsmasq listen-address

    We should only include listen-address in the dnsmasq configuration if it
    is not empty. Previously we might end up with a blank listen-address if
    it was set to ""

    Change-Id: Ia555b49e98018bb3f1982877fd19868089c77d20
    Closes-Bug: #1615250

[Timur Nurlygayanov (tnurlygayanov)]

Fix was verified on NFV bare-metal QA lab.