OS hosts not resolving after env update

[Igor Yozhikov]

Strange behavior was investigated during bug-fix process of the https://bugs.launchpad.net/mos/+bug/1364026 .
Short follow-up:
release 2014.1 (5.0) -> Upload and unpack tarball with fuel-5.1 release (build #499) -> update cluster to release 2014.1.1-5.0.2

Result:
...
[root@nailgun ~]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online
---|----------|---------------------|---------|---------------|-------------------|------------|---------------|-------
2 | discover | slave-01_controller | 1 | 10.108.20.134 | 64:df:89:aa:ad:7a | controller | | True
1 | ready | slave-04_compute | 1 | 10.108.20.3 | 72:fe:42:d2:cc:4c | compute | | True
4 | ready | slave-05_compute | 1 | 10.108.20.6 | 42:e4:79:46:bd:47 | compute | | True
3 | ready | slave-03_controller | 1 | 10.108.20.5 | 6a:58:24:b4:35:41 | controller | | True
5 | ready | slave-02_controller | 1 | 10.108.20.7 | 16:4a:1f:28:a2:41 | controller | | True
[root@nailgun ~]# ping node-1
ping: unknown host node-1
...
Also, I have rebooted server instance with id=2 and it boots into bootstrap instead of normal boot from HDD. You can see the result above in the fuel node output table.

[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "499"
  build_id: "2014-09-05_00-01-17"
  astute_sha: "b622d9b36dbdd1e03b282b9ee5b7435ba649e711"
  fuellib_sha: "6ebadd157c5153e9bb3ed53fa73507a5c73572c3"
  ostf_sha: "f7b5d9d0d1cfaba5f1fe1e2c634493e92bce11db"
  nailgun_sha: "b36dccbc28584532ca1bdb679b096cc3f5f6f80a"
  fuelmain_sha: "b3afef75675ae3ab251a4f990f3432968eb746eb"

[Artem Panchenko (apanchenko-8)]

Reproduced the same issue using upgrade archive # 503. Here is cobbler's systems configuration after upgrade (looks good):

http://paste.openstack.org/show/108763/

and dnsmasq config (incomplete):

http://paste.openstack.org/show/108764/

Also I discovered that /var/lib/cobbler/cobbler_hosts file is empty. After I restarted cobbler container configuration file was generated correctly:

http://paste.openstack.org/show/108765/

resolving worked fine and deployed nodes weren't bootstrapped after restart.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/120047

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/120066

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/120047
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=e3f947cc4142210499a282bc5f183c333552fa23
Submitter: Jenkins
Branch: master

commit e3f947cc4142210499a282bc5f183c333552fa23
Author: Matthew Mosesohn <email address hidden>
Date: Tue Sep 9 12:55:03 2014 +0400

    Trigger cobbler_sync after profile creation

    Set cobbler_sync to always run every puppet
    run just in case. Nailgun cobbler sync should
    trigger after profile creation.

    Closes-Bug: #1366875

    Change-Id: I9107e42374e0013e63a40856af7224e3811b6063

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/120066
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=82091e0d61f252619a0842d0f8debb6b602a61fe
Submitter: Jenkins
Branch: master

commit 82091e0d61f252619a0842d0f8debb6b602a61fe
Author: Evgeniy L <email address hidden>
Date: Tue Sep 9 15:31:41 2014 +0400

    Upgrades, copy cobbler files right after container is created

    Supervisor start is too slow, puppet run (and cobbler sync)
    passes before we copy the data for cobbler.

    The solution is to copy all required files right
    after container is created.

    Change-Id: I6b4b0ab879d31e43a70e5c9e72c83b1de47ddfb2
    Related-bug: #1366875