After downgrade of fuel from 5.1 to 5.0 on vCenter's machine one of slaves can't bootstrapping after cobbler error

[Tatyana Dubyk]

After downgrade of fuel from 5.1 to 5.0 on vCenter's machine one of slaves can't bootstrapping after cobbler error

----------------------Steps to reproduce---------------------------------------------

on vcenter machine 172.18.170.88:

1. Set up lab with fuel version 5.0 (I used iso 5.0 build 26)

2. Create new environment (Ubuntu, simple one node)
     Network: Nova Network Flat DHCP

     setting for vcenter creation:
                    VCENTER_IP='172.16.0.254'
                    <email address hidden>'
                    VCENTER_PASSWORD='Qwer!1234'
                    VCENTER_CLUSTERS='Cluster1'

3. Add 2 nodes with roles: 1 controller and 1 cinder (as storage type by default cinder lvm over iSCSI for volumes)
4. Make deploy of environment
5. Verify network connectivity
6. Run ostf
7. download upgrade patch 496 and perform update from Fuel 5.0 -> 5.1
8. Upgrade openstack env from 5.0 to 5.0.2
9. Run ostf, Verify network connectivity
10. Downgrade env from 5.0.2-> 5.0
11. Run ostf, Verify network connectivity
12. downgrade fuel from 5.1 to 5.0
13. check that master node and slave-1 available and ready, but slave-2 can't perform bootstraping.

Expected result:
Node booted with bootstrap image successfully

Actual result:
Node bootstrapping fails
on slave-2 error: boot failed, unable to locate config file

I've tried to reboot master node and all slaves , but the result is the same as on step 13.

--------------------------------------------------------------
after upgrade from 5.0 -> 5.1

api: '1.0'
astute_sha: 8cc2c5b29d75ec2f60787e225564426a0508b3e4
auth_required: true
build_id: 2014-09-04_00-01-17
build_number: '496'
feature_groups:
- mirantis
fuellib_sha: bc23f1359b35ec90d5c7bc9f33778b4f2922d409
fuelmain_sha: 435c43fe558898566ef8254ec09cc12bcb809c4b
nailgun_sha: 3d965a7acd9361a361da92970d5bc5a1be77c969
ostf_sha: 4dcd99cc4bfa19f52d4b87ed321eb84ff03844da
production: docker
release: '5.1'

after downgrade from 5.1 -> 5.0
api: '1.0'
astute_sha: a7eac46348dc77fc2723c6fcc3dbc66cc1a83152
build_id: 2014-05-27_05-51-41
build_number: '26'
fuellib_sha: 2f79c0415159651fc1978d99bd791079d1ae4a06
fuelmain_sha: 505741e4f431f85a8d0252fc42754d10c0326c1a
mirantis: 'yes'
nailgun_sha: bd09f89ef56176f64ad5decd4128933c96cb20f4
ostf_sha: a8b7660082a6f152794c610d6abe30d360fd577d
production: docker
release: '5.0'

[Evgeniy L (rustyrobot)]

Tatyana, I have a question, how did you perform 'downgrade fuel from 5.1 to 5.0' ?
The thing is we don't have downgrading of fuel master node, we have rollback which is run automatically if there was any error during the upgrade procedure.

[Tatyana Dubyk (tdubyk)]

To perform fuel's downgrade from 5.1 to 5.0 I've used this commands:
rm /etc/supervisord.d/current

ln -s /etc/supervisord.d/5.0 /etc/supervisord.d/current

rm /etc/fuel/version.yaml

ln -s /etc/fuel/5.0/version.yaml /etc/fuel/version.yaml

rm /var/www/nailgun/bootstrap

ln -s /var/www/nailgun/5.0_bootstrap /var/www/nailgun/bootstrap

/etc/init.d/supervisord stop

docker stop $(docker ps -q)

/etc/init.d/supervisord start

[Tatyana Dubyk (tdubyk)]

Maybe you to muddle together downgrading of Fuel platform and downgrading of openstack using fuel ui. I mean in bug report downgrading of Fuel platform.

[Evgeniy L (rustyrobot)]

Try to replace this step
ln -s /var/www/nailgun/5.0_bootstrap /var/www/nailgun/bootstrap

With this
rm -f /var/www/nailgun/bootstrap
cp -r /var/www/nailgun/5.0_bootstrap /var/www/nailgun/bootstrap

[Tatyana Dubyk (tdubyk)]

Ok, Evgeniy , a little bit later I'll try to reproduce it again with your recommendations. thanks.

[Tatyana Dubyk (tdubyk)]

I've reproduce this bug one more time again. But I think its appears due to my method that I use for rollback of Fuel. I need some expert evaluation may this bug has sense or not. because user can perform rollback with this commands, but chance of it not so high.

[Evgeniy L (rustyrobot)]

I'll try to reproduce the problem.

[Evgeniy L (rustyrobot)]

Checked, created cluster, ran upgrade, deployed cluster, made rollback, rebooted the nodes, nodes were bootstrapped.

It looks like the problem was related to cobbler container which was not ready at the moment of nodes bootstraping.