Redeployment of env with 'error' status has failed

[Anastasia Palkina]

1. Create new environment (Ubuntu)
2. Add 3 controllers and 2 computes
3. Start deployment. It was successful
4. Run OSTF test. It was successful
5. Add 1 cinder
6. Redeploy environment. It was successful
7. Remove the 'cinder' node and remove a 'controller' node
8. Add 'controller'+'cinder' node
9. Start deployment. It has failed with error: Timeout of deployment is exceeded.
10. Try to deploy env again. Redeployment has failed with error:

[596] Error running RPC method granular_deploy: Failed to execute hook 'puppet' Puppet run failed.

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaSm5ITFJieDhDeUU/view?usp=sharing

"build_id": "260", "build_number": "260", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "260", "build_number": "260", "api": "1.0", "fuel-library_sha": "e055af9dee6fbaf84cb8e86a9ebff9b25b07792e", "nailgun_sha": "3de0f32fe9e09fbeee8a695d738f31fe2347e55a", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4", "release": "7.0", "fuelmain_sha": "994bb9a8a2a3c4ab1f16e57b479d052abe45f921"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "e055af9dee6fbaf84cb8e86a9ebff9b25b07792e", "nailgun_sha": "3de0f32fe9e09fbeee8a695d738f31fe2347e55a", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4", "release": "7.0", "fuelmain_sha": "994bb9a8a2a3c4ab1f16e57b479d052abe45f921"

[Dima Shulyak (dshulyak)]

Actually last log contains command - 2015-09-01 15:20:46 +0000 Puppet (debug): Executing '/usr/bin/openstack service list --quiet --format csv --long'

And this is what i have if will try to execute it by hand on node-6

root@node-6:~# /usr/bin/openstack service list --quiet --format csv --long
Traceback (most recent call last):
  File "/usr/bin/openstack", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/dist-packages/openstackclient/shell.py", line 342, in main
    return OpenStackShell().run(argv)
  File "/usr/lib/python2.7/dist-packages/openstackclient/shell.py", line 79, in __init__
    command_manager=commandmanager.CommandManager('openstack.cli'))
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 74, in __init__
    self._set_streams(stdin, stdout, stderr)
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 82, in _set_streams
    locale.setlocale(locale.LC_ALL, '')
  File "/usr/lib/python2.7/locale.py", line 579, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

As for this bug we have 2 problems:

1. Hanged keystone.pp which leads to different failures in future
2. Absence of proper interruption of hung puppet processes during next runs

2nd is not critical and i will create ticket for 8.0, for the 1st i would like to ask someone from puppet team to take a look

[Dima Shulyak (dshulyak)]

Currently this problem is reproducible on Anastasia environment, for some reason we have hanged puppet from previous deployment..

28895 ? Ssl 0:26 /usr/bin/ruby /usr/bin/puppet apply /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp --modulepath=/etc/puppet/modules --logdest syslog --trace --no-report --debug --evaltrace --logdest /var/log/puppet.log
root@node-6:~# ps -o lstart= -p 28895
Tue Sep 1 15:19:58 2015

And now each time when you want to start new puppet - astute concludes that puppet is hung

[Michael Polenchuk (mpolenchuk)]

Got the same s##
I guess the keystone process(e.g. wsgi) makes puppet wait forever.
"strace" show wait(4, ) ... and that's all.
As a workaround: send TERM signal to puppet process.

[Dima Shulyak (dshulyak)]

One more detail - it actually tries to receive stuff from keystone

952 ? Ss 0:01 /usr/bin/python /usr/bin/openstack service list --quiet --format csv --long

Process 952 attached
recvfrom(3,

tcp ESTAB 0 0 192.168.0.4:55044 192.168.0.4:35357 users:(("openstack",952,3))

[Michael Polenchuk (mpolenchuk)]

https://ask.openstack.org/en/question/66460/keystone-client-hangs/

[Michael Polenchuk (mpolenchuk)]

tcp 0 0 node-2.test.domai:58612 node-2.test.domai:35357 ESTABLISHED 6512/python off (0.00/0/0)
tcp6 180 0 node-2.test.domai:35357 node-2.test.domai:58612 ESTABLISHED - keepalive (1.79/0/0)

# netstat -tpln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::35357 :::* LISTEN 16950/apache2
tcp6 0 0 :::5000 :::* LISTEN 16950/apache2

Is it Ok?