Deployed environment:
Mos 8.0
3 controllers
2 compute+Cinder with LVM
1 ironic
This environment is ready for weeks.
What I do is to deploy a new compute node(compute+cinder), that caused the whole cluster marked as Error on "Fuel" web, I tryed the deploy again. And the whole cluster began redeploying!!
Logs:
2016-03-11 02:22:20 +0000 Puppet (debug): Executing '/usr/bin/dpkg-query -W --showformat '${Status} ${Package} ${Version}\n' python-nova'
2016-03-11 02:22:20 +0000 Puppet (debug): Executing '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install python-nova'
2016-03-11 02:22:20 +0000 Puppet (err): Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install python-nova' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
python-nova : Depends: websockify (>= 0.6.1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
/usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:219:in `execute'
/usr/lib/ruby/vendor_ruby/puppet/provider/command.rb:23:in `execute'
/usr/lib/ruby/vendor_ruby/puppet/provider.rb:237:in `block in has_command'
/usr/lib/ruby/vendor_ruby/puppet/provider.rb:463:in `block in create_class_and_instance_method'
/usr/lib/ruby/vendor_ruby/puppet/provider/package/apt.rb:73:in `install'
/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/apt_fuel.rb:49:in `install'
root@node-9:~# apt-get -q -y -o DPkg::Options::=--force-confold install python-nova
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
python-nova : Depends: websockify (>= 0.6.1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
root@node-9:~# apt-cache policy websockify
websockify:
Installed: (none)
Candidate: 0.6.1+dfsg1-1~u14.04+mos1
Version table:
0.6.1+dfsg1-1~u14.04+mos1 0
1000 http://10.0.20.2:8080/mirrors/mos-repos/ubuntu/8.0/ mos8.0/main amd64 Packages
0.5.1+dfsg1-3ubuntu0.14.04.1 0
500 http://10.0.64.242/ubuntu/ trusty-updates/universe amd64 Packages
0.5.1+dfsg1-3 0
500 http://10.0.64.242/ubuntu/ trusty/universe amd64 Packages
root@node-9:~# dpkg -l | grep websockify
root@node-9:~#
I use a local ubuntu & mos repo, no connective issue.
Updates:
2016-05-17
Today I emulated this issue by disable dependence package python-numpy for websockify from local ubuntu repo:
root@node-3:~# apt-cache policy python-numpy
python-numpy:
Installed: 1:1.8.2-0ubuntu0.1
Candidate: 1:1.8.2-0ubuntu0.1
Version table:
*** 1:1.8.2-0ubuntu0.1 0
500 http://10.14.64.242/ubuntu/ trusty-updates/main amd64 Packages
100 /var/lib/dpkg/status
1:1.8.1-1ubuntu1 0
500 http://10.14.64.242/ubuntu/ trusty/main amd64 Packages
mv python-numpy_1.8.1-1ubuntu1_amd64.deb python-numpy_1.8.1-1ubuntu1_amd64.deb.bak
mv python-numpy_1.8.2-0ubuntu0.1_amd64.deb python-numpy_1.8.2-0ubuntu0.1_amd64.deb.bak
then deploy a new compute node;
Install python-nova failed as expect:
Get:90 http://10.14.20.2:8080/liberty-8.0/ubuntu/x86_64/ mos8.0/main python-sqlalchemy-ext amd64 1.0.8~u14.04+mos1 [18.4 kB]
Get:91 http://10.14.20.2:8080/liberty-8.0/ubuntu/x86_64/ mos8.0/main websockify amd64 0.6.1+dfsg1-1~u14.04+mos1 [37.6 kB]
Fetched 12.3 MB in 0s (26.7 MB/s)
E: Failed to fetch http://10.14.64.242/ubuntu/pool/main/p/python-numpy/python-numpy_1.8.2-0ubuntu0.1_amd64.deb 404 Not Found
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
/usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:219:in `execute'
/usr/lib/ruby/vendor_ruby/puppet/provider/command.rb:23:in `execute'
/usr/lib/ruby/vendor_ruby/puppet/provider.rb:237:in `block in has_command'
/usr/lib/ruby/vendor_ruby/puppet/provider.rb:463:in `block in create_class_and_instance_method'
new compute node was marked as "Err", then all controller & other nodes go through the puppet jobs and done smoothly, then all nodes marked as "Err" a bit later which exactly the same as this bug reported.
then restore the package python-numpy, and deploy again, the controller go through the puppet scripts again&again..., stuck at 11%(openstack was not redeployed) almost 1 hour and a half later, all nodes restored to ready status,
So the openstack environment was not actually redeployed and vm creation was still working. I may have misunderstanding for controller became "Deploy" status. But the robustness still may be a problem, as deploy new node may impact the whole environment, potential risk still exist.
Please attach diagnostic snapshot.