[puppet] Deployment failed with puppet error at /Stage[main]/Main/Exec on openstack-controller/openstack-controller.pp

[Dmitry Tyzhnenko]

Deployment failed with puppet error at /Stage[main]/Main/Exec on openstack-controller/openstack-controller.pp

 (/Stage[main]/Main/Exec[create-m1.micro-flavor]/returns) change from notrun to 0 failed: bash -c "nova flavor-create --is-public true m1.micro auto 64 0 1" returned 1 instead of one of [0]

Fuel ISO 7.0-164

Scenario:
 1. Create new environment
 2. Choose Neutron, TUN
 3. Choose Ceph for volumes and Ceph for images
 4. Add 5 controller
 5. Add 2 compute
 6. Add 2 ceph
 7. Verify networks
 8. Deploy the environment
 9. Verify networks
 10. Run OSTF tests

Expected result:
  All step pass

Actual result:
  Step 8 has failed on deploying primary controller

Astute error - http://paste.openstack.org/show/414283/
Puppet error - http://paste.openstack.org/show/414285/

Fuel version -

{
    'build_id': '2015-08-12_15-16-47',
    'build_number': '164',
    'release_versions': {'2015.1.0-7.0': {'VERSION': {
        'build_id': '2015-08-12_15-16-47',
        'build_number': '164',
        'api': '1.0',
        'fuel-library_sha': '22f848670e49d89fc04aaed4d8efd1b07360cbe7',
        'nailgun_sha': 'fff6bda090fac15c48b27cca7832a70f8e381101',
        'feature_groups': ['mirantis'],
        'fuel-nailgun-agent_sha': 'e01693992d7a0304d926b922b43f3b747c35964c',
        'openstack_version': '2015.1.0-7.0',
        'fuel-agent_sha': '57145b1d8804389304cd04322ba0fb3dc9d30327',
        'production': 'docker',
        'python-fuelclient_sha': '26fc025e0fc5791b62e5ed8561a6016bf8a406bc',
        'astute_sha': 'e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc',
        'fuel-ostf_sha': '58220583f10fa47f12291488ef77854809c68310',
        'release': '7.0',
        'fuelmain_sha': '67e5214c0dc5d4ba6da4ae651cef9934800459a9',
        }}},
    'auth_required': true,
    'api': '1.0',
    'fuel-library_sha': '22f848670e49d89fc04aaed4d8efd1b07360cbe7',
    'nailgun_sha': 'fff6bda090fac15c48b27cca7832a70f8e381101',
    'feature_groups': ['mirantis'],
    'fuel-nailgun-agent_sha': 'e01693992d7a0304d926b922b43f3b747c35964c',
    'openstack_version': '2015.1.0-7.0',
    'fuel-agent_sha': '57145b1d8804389304cd04322ba0fb3dc9d30327',
    'production': 'docker',
    'python-fuelclient_sha': '26fc025e0fc5791b62e5ed8561a6016bf8a406bc',
    'astute_sha': 'e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc',
    'fuel-ostf_sha': '58220583f10fa47f12291488ef77854809c68310',
    'release': '7.0',
    'fuelmain_sha': '67e5214c0dc5d4ba6da4ae651cef9934800459a9',
    }

CRM status on node-1:

root@node-1:~# crm status
Last updated: Fri Aug 14 14:36:26 2015
Last change: Fri Aug 14 12:42:50 2015
Stack: corosync
Current DC: node-1.domain.local (1) - partition WITHOUT quorum
Version: 1.1.12-561c4cf
1 Nodes configured
10 Resources configured

Online: [ node-1.domain.local ]

 Clone Set: clone_p_vrouter [p_vrouter]
     Started: [ node-1.domain.local ]
 vip__management (ocf::fuel:ns_IPaddr2): Started node-1.domain.local
 vip__vrouter_pub (ocf::fuel:ns_IPaddr2): Started node-1.domain.local
 vip__vrouter (ocf::fuel:ns_IPaddr2): Started node-1.domain.local
 vip__public (ocf::fuel:ns_IPaddr2): Started node-1.domain.local
 Master/Slave Set: master_p_conntrackd [p_conntrackd]
     Masters: [ node-1.domain.local ]
 Clone Set: clone_p_haproxy [p_haproxy]
     Started: [ node-1.domain.local ]
 Clone Set: clone_p_mysql [p_mysql]
     Started: [ node-1.domain.local ]
 Clone Set: clone_p_dns [p_dns]
     Started: [ node-1.domain.local ]
 Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
     Masters: [ node-1.domain.local ]

[Dmitry Tyzhnenko (dtyzhnenko)]

It seems to be happen because nova has problem with connection

root@node-1:~# . openrc
root@node-1:~# nova flavor-create --is-public true m1.micro auto 64 0 1
ERROR (ConnectionRefused): Unable to establish connection to https://public.fuel.local:5000/v2.0/tokens
root@node-1:~# ping public.fuel.local
ping: unknown host public.fuel.local

[Bartłomiej Piotrowski (bpiotrowski)]

I can't reproduce the error on said iso. Also OSTF finished correctly.

[Stanislaw Bogatkin (sbogatkin)]

It happens due ssl_add_trust_chain task has priority 6200 and openstack-controller task has priority 5300. So dns hostnames tried to be used before they was added to hosts. Why it happened - is a question, actually. I will try to ask astute team about this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/213743

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/213743
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=2c9d1523c537232bbf6f0f3c49552810adb6e627
Submitter: Jenkins
Branch: master

commit 2c9d1523c537232bbf6f0f3c49552810adb6e627
Author: Michael Polenchuk <email address hidden>
Date: Mon Aug 17 17:33:17 2015 +0300

    Explicitly set public ssl host dependence

    Setup dependence for public ssl record addition in hosts file
    to avoid using of uninitialized hostname.

    Change-Id: Ib41efb84de39b4ac26995c6a461015123e6380e9
    Closes-bug: #1484963

[Dmitry Tyzhnenko (dtyzhnenko)]

verified on 7.0-219

{
    'build_id': '2015-08-23_15-01-12',
    'build_number': '219',
    'release_versions': {'2015.1.0-7.0': {'VERSION': {
        'build_id': '2015-08-23_15-01-12',
        'build_number': '219',
        'api': '1.0',
        'fuel-library_sha': '3a3ea6d9849bc1ba35c1bd882f0a0678b20d2e51',
        'nailgun_sha': '7790ce872512ecdf21689e6a5f970dd7119febdb',
        'feature_groups': ['mirantis'],
        'fuel-nailgun-agent_sha': 'e01693992d7a0304d926b922b43f3b747c35964c',
        'openstack_version': '2015.1.0-7.0',
        'fuel-agent_sha': '4c2ab9d6c623d345086c6e2874d1df81fd96a942',
        'production': 'docker',
        'python-fuelclient_sha': 'fc7b63aa6900fe3b2c183108ba6a13e868bc0472',
        'astute_sha': '53c86cba593ddbac776ce5a3360240274c20738c',
        'fuel-ostf_sha': '16839cbf471b7142b04c0d2c2d94786bc486fefe',
        'release': '7.0',
        'fuelmain_sha': 'a494e6628319abfef57e1754f6453cf8f1a4bc65',
        }}},
    'auth_required': true,
    'api': '1.0',
    'fuel-library_sha': '3a3ea6d9849bc1ba35c1bd882f0a0678b20d2e51',
    'nailgun_sha': '7790ce872512ecdf21689e6a5f970dd7119febdb',
    'feature_groups': ['mirantis'],
    'fuel-nailgun-agent_sha': 'e01693992d7a0304d926b922b43f3b747c35964c',
    'openstack_version': '2015.1.0-7.0',
    'fuel-agent_sha': '4c2ab9d6c623d345086c6e2874d1df81fd96a942',
    'production': 'docker',
    'python-fuelclient_sha': 'fc7b63aa6900fe3b2c183108ba6a13e868bc0472',
    'astute_sha': '53c86cba593ddbac776ce5a3360240274c20738c',
    'fuel-ostf_sha': '16839cbf471b7142b04c0d2c2d94786bc486fefe',
    'release': '7.0',
    'fuelmain_sha': 'a494e6628319abfef57e1754f6453cf8f1a4bc65',
    }