OSTF has crashed because instances don't switch into active state

[Iryna Vovk]

Fuel ISO# MirantisOpenStack-6.1

Enviroment:
   OS: Ubuntu
   Compute: QEMU
   Networking Setup: Neutron with vlan segmentation
   Ceph storage: No
   Additional services: all by default

Steps to reproduce:
Log into the Fuel web UI
Press “New OpenStack Environment”
Specify Environment name as test
Set QEMU as compute
Select 'Neutron with VLAN segmentation' as a network configuration
Set “default” glance and cinder
Do not use any additional services
Press “Create”
Open the Settings tab of the Fuel web UI
Select the Contrail plugin checkbox and configure plugin settings
Select "Assign public network to all nodes" checkbox
Configure network
Add 3 nodes with “Operating system” role
Rename these nodes in the template “contrail-[1-3]”
Add 1 node with “Controller” and 1 node with “Compute” role
Start deploy
Check Controller and Contrail nodes status
Add 1 node with “Controller” and 1 node with “Compute” role
Start deploy
After the end of deploy run OSTF tests

Actual result:
OSTF has crashed because instances don't switch into active state

VERSION:
api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
      build_id: 2015-06-19_13-02-31
      build_number: '525'
      feature_groups:
      - mirantis
      fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
      fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
      fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
      nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
      release: '6.1'

[Iryna Vovk (ivovk)]

Failed OSTF tests you can see in screenshots

[Oleksandr Martsyniuk (omartsyniuk)]

The reason of OSTF tests failed is that an instance cannot be spawned with TestVM image. Must be a Glance problem.
We have uploaded another image to glance, and it works fine. Also the affected image can not be deleted, glance logs on controllers are full of 'permission denied' errors.

We need to reproduce this issue following the exact steps with the same configuration, but without Contrail plugin installed.

[Iryna Vovk (ivovk)]

I tried to reproduce this issue without redeploy procedure and tryid to create instance. Instance was created successfully without any errors.

Steps to reproduce:

1. Log into the Fuel web UI
2. Press “New OpenStack Environment”
3. Specify Environment name as test
4. Set QEMU as compute
5. Select 'Neutron with VLAN segmentation' as a network configuration
6. Set “default” glance and cinder
Do not use any additional services
7. Press “Create”
8. Open the Settings tab of the Fuel web UI
9. Select the Contrail plugin checkbox and configure plugin settings
10. Select "Assign public network to all nodes" checkbox
11. Configure network
12. Add 3 nodes with “Operating system” role
13. Rename these nodes in the template “contrail-[1-3]”
14. Add 1 node with “Controller” and 1 node with “Compute” role
15. Start deploy
16. Check Controller and Contrail nodes status
17. Try to create instance

And now I'm trying to reproduce this scenario without Contrail Plugin. I'll post comment after verifying.

[Iryna Vovk (ivovk)]

Redeploy with 'assign public network to all nodes' function has failed (without plugin)

Steps to reproduce:

1. Log into the Fuel web UI
2. Press “New OpenStack Environment”
3. Specify Environment name as test
4. Set QEMU as compute
5. Select 'Neutron with VLAN segmentation' as a network configuration
6. Set “default” glance and cinder
Do not use any additional services
7. Press “Create”
8. Open the Settings tab of the Fuel web UI
Select "Assign public network to all nodes" checkbox
9. Configure network
10. Add 1 node with “Controller” and 1 node with “Compute” role
11. Start deploy
12. Check Controller and Contrail nodes status
13. Add 1 node with “Controller” and 1 node with “Compute” role
14. Start deploy

Actual Result:
Redeploy has failed.

[Iryna Vovk (ivovk)]

error from /var/log/puppet.log:

root@node-1:~# grep "(err)" /var/log/puppet.log
2015-07-09 10:46:47 +0000 Puppet (err): Could not start Service[swift-container-replicator]: Execution of '/sbin/start swift-container-replicator' returned 1:
2015-07-09 10:46:47 +0000 /Stage[main]/Swift::Storage::Container/Swift::Storage::Generic[container]/Service[swift-container-replicator]/ensure (err): change from stopped to running failed: Could not start Service[swift-container-replicator]: Execution of '/sbin/start swift-container-replicator' returned 1:
2015-07-09 10:46:51 +0000 Puppet (err): Could not start Service[swift-container-sync]: Execution of '/sbin/start swift-container-sync' returned 1:
2015-07-09 10:46:51 +0000 /Stage[main]/Swift::Storage::Container/Service[swift-container-sync]/ensure (err): change from stopped to running failed: Could not start Service[swift-container-sync]: Execution of '/sbin/start swift-container-sync' returned 1:
2015-07-09 10:46:59 +0000 Puppet (err): Could not start Service[swift-proxy]: Execution of '/sbin/start swift-proxy' returned 1:
2015-07-09 10:46:59 +0000 /Stage[main]/Swift::Proxy/Service[swift-proxy]/ensure (err): change from stopped to running failed: Could not start Service[swift-proxy]: Execution of '/sbin/start swift-proxy' returned 1:
2015-07-09 10:47:01 +0000 Puppet (err): Could not start Service[swift-account-replicator]: Execution of '/sbin/start swift-account-replicator' returned 1:
2015-07-09 10:47:01 +0000 /Stage[main]/Swift::Storage::Account/Swift::Storage::Generic[account]/Service[swift-account-replicator]/ensure (err): change from stopped to running failed: Could not start Service[swift-account-replicator]: Execution of '/sbin/start swift-account-replicator' returned 1:
2015-07-09 10:48:12 +0000 Puppet (err): Could not start Service[swift-container-replicator]: Execution of '/sbin/start swift-container-replicator' returned 1:
2015-07-09 10:48:12 +0000 /Stage[main]/Swift::Storage::Container/Swift::Storage::Generic[container]/Service[swift-container-replicator]/ensure (err): change from stopped to running failed: Could not start Service[swift-container-replicator]: Execution of '/sbin/start swift-container-replicator' returned 1:
2015-07-09 10:48:13 +0000 Puppet (err): Could not start Service[swift-container-sync]: Execution of '/sbin/start swift-container-sync' returned 1:
2015-07-09 10:48:13 +0000 /Stage[main]/Swift::Storage::Container/Service[swift-container-sync]/ensure (err): change from stopped to running failed: Could not start Service[swift-container-sync]: Execution of '/sbin/start swift-container-sync' returned 1:
2015-07-09 10:48:16 +0000 Puppet (err): Could not start Service[swift-proxy]: Execution of '/sbin/start swift-proxy' returned 1:
2015-07-09 10:48:16 +0000 /Stage[main]/Swift::Proxy/Service[swift-proxy]/ensure (err): change from stopped to running failed: Could not start Service[swift-proxy]: Execution of '/sbin/start swift-proxy' returned 1:
2015-07-09 10:48:18 +0000 Puppet (err): Could not start Service[swift-account-replicator]: Execution of '/sbin/start swift-account-replicator' returned 1:
2015-07-09 10:48:18 +0000 /Stage[main]/Swift::Storage::Account/Swift:...

[Oleksiy Molchanov (omolchanov)]

I have tried to reproduce case "Redeploy with 'assign public network to all nodes' function has failed (without plugin)" and it was successful. But in your logs I have found swift related problem:

[Errno 2] No such file or directory: '/etc/swift/object.ring.gz'

For the original issue, it seems to be only plugin related.

[Iryna Vovk (ivovk)]

I have verified this issue with the latest version of Plugin and deploy was successful. This issue is not reproducible now.