[regression] Deployment fails on /bin/bash "/etc/puppet/shell_manifests/upload_cirros_command.sh" returned 1 instead of one of [0]

[Volodymyr Shypyguzov]

Steps to reproduce:
            1. Create cluster
            2. Add 1 node with controller role and mongo
            3. Add 1 nodes with compute role
            4. Set install Ceilometer option
            5. Deploy the cluster <<< Fail
            6. Verify Heat, Ceilometer services
            7. Run OSTF platform tests

Expected result: cluster deployed
Actual result: deployment failed with the following error: Deployment has failed. All nodes are finished. Failed tasks: Task[upload_cirros/1]

In glance-api.log:
2016-05-17 02:47:29.449 31034 ERROR swiftclient ClientException: Endpoint for object-store not found - have you specified a region?

[Anastasia Kuznetsova (akuznetsova)]

Issue was reproduced on 9.0-mos-364

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

version

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Igor Yozhikov (iyozhikov)]

Found in glance logs:
Endpoint or object-store not found - have you specified a region?
2016-05-17 02:47:29.449 31034 ERROR swiftclient Traceback (most recent call last):
2016-05-17 02:47:29.449 31034 ERROR swiftclient File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1553, in _retry
2016-05-17 02:47:29.449 31034 ERROR swiftclient self.url, self.token = self.get_auth()
2016-05-17 02:47:29.449 31034 ERROR swiftclient File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1507, in get_auth
2016-05-17 02:47:29.449 31034 ERROR swiftclient timeout=self.timeout)
2016-05-17 02:47:29.449 31034 ERROR swiftclient File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 617, in get_auth
2016-05-17 02:47:29.449 31034 ERROR swiftclient auth_version=auth_version)
2016-05-17 02:47:29.449 31034 ERROR swiftclient File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 557, in get_auth_keystone
2016-05-17 02:47:29.449 31034 ERROR swiftclient 'have you specified a region?' % service_type)
2016-05-17 02:47:29.449 31034 ERROR swiftclient ClientException: Endpoint for object-store not found - have you specified a region?
2016-05-17 02:47:29.449 31034 ERROR swiftclient

[Igor Yozhikov (iyozhikov)]

2016-05-17T02:47:29.659781+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) SubjectAltNameWarning
2016-05-17T02:47:29.662354+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) Error finding address for https://public.fuel.local:9292/v2/images/a74a0e97-f313-4cbb-9bf2-ee0d767e0790/file: Unable to establish connection to https://public.fuel.local:9292/v2/images/a74a0e97-f313-4cbb-9bf2-ee0d767e0790/file
2016-05-17T02:47:29.662354+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) Image 'TestVM' upload from '/usr/share/cirros-testvm/cirros-x86_64-disk.img' FAILED!
2016-05-17T02:47:29.662354+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) +------------------+--------------------------------------+
2016-05-17T02:47:29.662354+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) | Property | Value |
2016-05-17T02:47:29.662354+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) +------------------+--------------------------------------+
2016-05-17T02:47:29.662354+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) | checksum | None |
2016-05-17T02:47:29.663216+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) | container_format | bare |
2016-05-17T02:47:29.663692+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) | created_at | 2016-05-17T02:47:29Z |
2016-05-17T02:47:29.664197+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) | disk_format | qcow2 |
2016-05-17T02:47:29.664757+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) | id | a74a0e97-f313-4cbb-9bf2-ee0d767e0790 |
2016-05-17T02:47:29.665239+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) | min_disk | 0 |
2016-05-17T02:47:29.665667+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) | min_ram | 64 |
2016-05-17T02:47:29.666195+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) | name | TestVM |
2016-05-17T02:47:29.666777+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) | owner | 4efcbf5d7a2f4f23a5c678a0d51f6606 |
2016-05-17T02:47:29.667295+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) | protected | False |
2016-05-17T02:47:29.667851+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) | size | None |
2016-05-17T02:47:29.668291+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) | status | queued |
2016-05-17T02:47:29.668705+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) | tags | [] |
2016-05-17T02:47:29.669596+00:00 notice: (/Stage[main]/Main/Exec[upload_cirros_...

[Timur Nurlygayanov (tnurlygayanov)]

The issue reproduced for any clusters, here is the result for deployment with one controller and without any additional components:

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.services_ha_one_controller/109/testReport/(root)/deploy_heat_ha_one_controller_neutron/deploy_heat_ha_one_controller_neutron/

[Dmitry Ilyin (idv1985)]

It may be related with the Swift bug when rings were not synced properly in some cases and swift was working only on the primary controller.

Let's try to reproduce this after this patch is merged
https://review.openstack.org/#/c/320374/

[Ivan Berezovskiy (iberezovskiy)]

https://review.openstack.org/#/c/320374/ - this patch contains related fix. Please recheck the issue on the latest ISO with that patch.

[Volodymyr Shypyguzov (vshypyguzov)]

Verified on 9.0 iso #416 deployment finished successfully