Deploy cluster failed with error Could not evaluate: Could not find init script or upstart conf file for 'nova-compute'"

[Artem Hrechanychenko]

Steps:
1. Create new environment
2. Choose Neutron, Vlan
3. Choose cinder for volumes and ceph for images/ephemeral/objects
4. Add 3 controllers
5. Add 3 cinder+ceph
6. Add 1 Compute
7. Verify networks
8. Start deployment
9. Stop on provisioning
10. Start deployment again <<< failed here
11. Verify networks
12. Run OSTF tests

Actual result:
Error
Deployment has failed. Method granular_deploy. Failed to execute hook 'enable_nova_compute_service' Puppet run failed. Check puppet logs for details

from Astute logs we see that enable_nova_compute.pp could finished:

[819] Error running RPC method granular_deploy: Failed to execute hook 'enable_nova_compute_service' Puppet run failed. Check puppet logs for details --- uids: - '7' parameters: puppet_modules: /etc/puppet/modules puppet_manifest: /etc/puppet/modules/osnailyfacter/modular/astute/enable_compute.pp timeout: 3600 cwd: / priority: 900 fail_on_error: true type: puppet id: enable_nova_compute_service , trace: ["/usr/share/gems/gems/astute-8.0.0/lib/astute/nailgun_hooks.rb:64:in `block in process'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/nailgun_hooks.rb:26:in `each'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/nailgun_hooks.rb:26:in `process'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/deployment_engine/granular_deployment.rb:233:in `post_deployment_actions'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/deployment_engine.rb:75:in `deploy'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/orchestrator.rb:216:in `deploy_cluster'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/orchestrator.rb:52:in `granular_deploy'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/dispatcher.rb:92:in `granular_deploy'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:189:in `dispatch_message'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:146:in `block in dispatch'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `call'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `block in each'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:144:in `each_with_index'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:144:in `dispatch'", "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:123:in `block in perform_main_job'"]

if close look to puppet logs on failed node - Could not evaluate: Could not find init script or upstart conf file for 'nova-compute'

full puppet error logs:
http://paste.openstack.org/show/487267/

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

[Matthew Mosesohn (raytrac3r)]

There are connectivity issues in your environment, causing this dependency issue:
 python-nova : Depends: websockify (>= 0.6.1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
apt-get update failed several times on this environment.

There is nothing wrong with package ordering.

[Vladimir Kuklin (vkuklin)]

It is seems to be not connectivity issues. It means that some dependencies are broken and apt-get cannot install nova packages.

[Matthew Mosesohn (raytrac3r)]

Sorry, connectivity isn't a problem. The python-websockify issue is the problem. It can't be installed because of some dependency.

[Matthew Mosesohn (raytrac3r)]

I reproduced with ISO 570 as well, and my compute node deployed completely fine. There was probably an issue with a dependency for python-websockify, but its issue is not captured in logs and the environment is gone.

[Roman Podoliaka (rpodolyaka)]

Possibly a duplicate https://bugs.launchpad.net/fuel/+bug/1546567

[Alexander Petrov (apetrov-n)]

Reproduced on RC2, iso 570 on virtualbox.

ENV: 3 controller, 2 compute+storage

Error
Deployment has failed. Method granular_deploy. Failed to execute hook 'enable_nova_compute_service' Puppet run failed. Check puppet logs for details

http://paste.openstack.org/show/487400/

[Alexander Petrov (apetrov-n)]

In addition here is the diagnostic snapshot
https://drive.google.com/file/d/0B_TfvzG2XTbqdFBuU1NhT0RYTkE/view?usp=sharing

[Matthew Mosesohn (raytrac3r)]

Can you get me into the environment? I still can't see _WHY_ python-nova failed to install. It didn't fail on my env when trying to reproduce.

[Matthew Mosesohn (raytrac3r)]

According to apt-cache policy, it looks like ubuntu-updates repository was configured but wasn't functioning on the failed node. I managed to destroy apt on the repeoduced node, but I'll redeploy it and see if I can reproduce.

[Dmitry Pyzhov (dpyzhov)]

This issue is about temporary upstream repo outage. We still investigating the issue, but it definitely is not critical. High priority is the right priority so we can postpone backport.

[Matthew Mosesohn (raytrac3r)]

Back to incomplete on master because it might happen on 9.0, but it hasn't been reproduced yet.

[Matthew Mosesohn (raytrac3r)]

We had a reproduced env, but it's gone now. Still incomplete until we have another reproduced env

[Dmitry Pyzhov (dpyzhov)]

Workaround is to simply restart the deployment

[Olga Gusarenko (ogusarenko)]

Removing the release-notes tag as confirmed with SMEs.