Cluster NTP and DNS settings are invalid if Fuel deployed by system tests

[Maksym Strukov]

{"build_id": "2015-04-26_22-54-38", "build_number": "352", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-04-26_22-54-38", "build_number": "352", "api": "1.0", "fuel-library_sha": "6bdf783e2bffdce80ecffcca2915e6d32a8ccdd7", "nailgun_sha": "f9f17fe395f3e38c27d3d484302728a37ad8012d", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "2b311b3b82a1e2df1dc3484a0f37e282273cd988", "astute_sha": "c1793f982fda7e3fc7b937ccaa613c649be6a144", "fuel-ostf_sha": "b38602c841deaa03ddffc95c02f319360462cbe3", "release": "6.1", "fuelmain_sha": "01288380950bc89d572cf5902141c9a393ada950"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "6bdf783e2bffdce80ecffcca2915e6d32a8ccdd7", "nailgun_sha": "f9f17fe395f3e38c27d3d484302728a37ad8012d", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "2b311b3b82a1e2df1dc3484a0f37e282273cd988", "astute_sha": "c1793f982fda7e3fc7b937ccaa613c649be6a144", "fuel-ostf_sha": "b38602c841deaa03ddffc95c02f319360462cbe3", "release": "6.1", "fuelmain_sha": "01288380950bc89d572cf5902141c9a393ada950"}

Steps:
1. Deploy Fuel by system tests (./utils/jenkins/system_tests.sh -t test -w $(pwd) -j fuelweb_test -i $ISO_PATH -o --group=setup)
2. Login to masternode (web ui)
3. Create new cluster
4. Open Settings tab

Expected:
all settings are valid

Actual:
NTP and DNS settings are invalid (see attached screenshot)

[Timur Nurlygayanov (tnurlygayanov)]

status changed to Confirmed because we have no any links to the commits with fixes.

[Igor Zinovik (izinovik)]

Hitted this bug today.

Nailgun generates settings for new environment based on values
that appear in /etc/nailgun/settings.yaml (nailgun docker container).

In my setup UPSTREAM_DNS and UPSTREAM_NTP was incorrectly configured:
[root@nailgun nailgun]# grep UPSTREAM !$
grep UPSTREAM settings.yaml
DNS_UPSTREAM: 10.109.0.1 8.8.8.8
NTP_UPSTREAM: '10.109.0.1, , '

IP addresses in DNS_UPSTREAM stanza must be separated by comma, not by space.
NTP_UPSTREAM is also incorrectly formed.

When you try to create new environment with broken settings you will also
get a error message when you press 'Finish' button in cluster creation wizard (see screenshot).

In nailgun json response I see that 'dns_list' field is incorrect (see screenshot).

[root@nailgun ~]# fuel --fuel-version
api: '1.0'
astute_sha: 83b23783edbfacd25aded7a79e0aca17fc883e79
auth_required: true
build_id: 2015-05-07_03-59-48
build_number: '388'
feature_groups:
- mirantis
fuel-library_sha: a7a794e68086014f8163f79a3dad1c9a074b1f3a
fuel-ostf_sha: 1a09c09ee7f6b7622dfc7e37dce70c4537c0b532
fuelmain_sha: 68e6f08e23845516e1995df59e038ebecb26e6b8
nailgun_sha: 7437e500c3e2673b124af4c7634ac04121b1c949
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: af6c9c3799b9ec107bcdc6dbf035cafc034526ce
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 83b23783edbfacd25aded7a79e0aca17fc883e79
      build_id: 2015-05-07_03-59-48
      build_number: '388'
      feature_groups:
      - mirantis
      fuel-library_sha: a7a794e68086014f8163f79a3dad1c9a074b1f3a
      fuel-ostf_sha: 1a09c09ee7f6b7622dfc7e37dce70c4537c0b532
      fuelmain_sha: 68e6f08e23845516e1995df59e038ebecb26e6b8
      nailgun_sha: 7437e500c3e2673b124af4c7634ac04121b1c949
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: af6c9c3799b9ec107bcdc6dbf035cafc034526ce
      release: '6.1'

[Andrew Woodward (xarses)]

partial fix for ntp from https://bugs.launchpad.net/fuel/+bug/1452665

[Andrew Woodward (xarses)]

fix in 1452665 still results in issues. We now get

"10.110.0.1, undef, undef" for NTP servers

[Igor Zinovik (izinovik)]

This bug affect vCenter support.
VMware tab does not appear when enabled vCenter support, so
there is no way to configure vCenter for your cluster, because of this bug.

[Igor Zinovik (izinovik)]

Here is a workaround for this problem:
1. SSH into fuel master node
2. Jump into nailgun docker container
3. Edit DNS_UPSTREAM and NTP_UPSTREAM in /etc/nailgun/settings.yaml
4. Restart nailgun

# ssh root@10.20.0.2
fuel# dockerctl shell nailgun
nailgun-lxc# vim /etc/nailgun/settings.yaml
...
# Use comma separated list of IP addressess
DNS_UPSTREAM: 10.20.0.1, 8.8.8.8
NTP_UPSTREAM: 10.20.0.1
...
nailgun-lxc# supervisorctl restart nailgun

[Matthew Mosesohn (raytrac3r)]

Please attach /etc/fuel/astute.yaml and /etc/fuel/version.yaml. I'm not sure how these undefs got written. You should have fields NTP2 and NTP3 with value of '' in astute.yaml. Maybe something was committed to fuel-qa that directly modifies the YAML because fuelmenu writes its config with empty strings if the user removes the value. I can't reproduce this, so it must be based on some changes made by system tests.

[Matthew Mosesohn (raytrac3r)]

This patch looks related. It should not delete the other NTP servers, but instead set them to blank: https://review.openstack.org/#/c/170141/ It would have broken before or after my patch, regardless

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/181461

[Andrew Woodward (xarses)]

Requested log output

http://paste.openstack.org/show/217322/

[Andrew Woodward (xarses)]

I've reproduced this on ISO's 357, 385, 390, and 394. Iso's are deployed using

'nosetests fuelweb_test.tests.base_test_case:SetupEnvironment.setup_master' let me know if you need access to debug

[Maksym Strukov (unbelll)]

Results on iso http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_6.1_iso/1181/ with proposed patch https://review.openstack.org/#/c/181461/ with original repro steps:

Actual:
DNS List = 10.109.5.1, 8.8.8.8
NTP server list = 10.109.5.1

Seems all fine.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/181461
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f2a6e5f4f323b82349c5f6bb43d27445af1bd307
Submitter: Jenkins
Branch: master

commit f2a6e5f4f323b82349c5f6bb43d27445af1bd307
Author: Matthew Mosesohn <email address hidden>
Date: Fri May 8 19:55:26 2015 +0300

    Correct multiple DNS servers and tolerate missing NTP

    Multiple DNS servers in astute.yaml were not valid input
    in the UI because they were missing a comma.

    NTP config from system tests removes NTP2 and NTP3 servers.
    Modified nailgun settings to permit empty NTP server
    configuration.

    Change-Id: Ie316e37817e889553840f4923d1426cb54e1589c
    Closes-Bug: #1449026

[Dennis Dmitriev (ddmitriev)]

Released:
{"build_id": "2015-05-12_08-34-41", "build_number": "406", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-05-12_08-34-41", "build_number": "406", "api": "1.0", "fuel-library_sha": "156fb11bbf3e12e7c73a9a3ac785c9d33d4ac343", "nailgun_sha": "0d077692e813720410c52bc720a8462725446e0d", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "af6c9c3799b9ec107bcdc6dbf035cafc034526ce", "astute_sha": "e319b19158fc416d911edf0c06667e810c457b02", "fuel-ostf_sha": "51b41cba7572aefa4a98e40fdecdbc05efb2e1ea", "release": "6.1", "fuelmain_sha": "51b86bb24b27742a22b23e2ae3dfc850c47e5fbf"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "156fb11bbf3e12e7c73a9a3ac785c9d33d4ac343", "nailgun_sha": "0d077692e813720410c52bc720a8462725446e0d", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "af6c9c3799b9ec107bcdc6dbf035cafc034526ce", "astute_sha": "e319b19158fc416d911edf0c06667e810c457b02", "fuel-ostf_sha": "51b41cba7572aefa4a98e40fdecdbc05efb2e1ea", "release": "6.1", "fuelmain_sha": "51b86bb24b27742a22b23e2ae3dfc850c47e5fbf"}