no object storage deployed if ceph selected for images but not for objects

Bug #1604879 reported by Alexey Deryugin on 2016-07-20
38
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Sergii Rizvan
6.1.x
High
Alexey Stupnikov
7.0.x
High
Sergii Rizvan
8.0.x
High
Sergii Rizvan
Mitaka
High
Vladimir Khlyunev
Newton
High
Sergii Rizvan

Bug Description

Detailed bug description:
 Swift not deployed if ceph selected for images but not for objects

Steps to reproduce:
 Deploy Fuel with Ceph selected for images and disabled for objects
Expected results:
 OSTF tests passes
Actual result:
 OSTF fails with error:
  Check state of haproxy backends on controllers (failure)
  Dead backends ['object-storage node-1 Status: DOWN/L4CON Sessions: 0 Rate: 0']
Reproducibility:
 100%
Workaround:
 Execute swift_task manually
Additional information:
 Looks like this [0] patch is a root cause.

[0] https://review.openstack.org/#/c/342886/

tags: added: area-library
Dmitry Pyzhov (dpyzhov) on 2016-08-04
tags: added: 9.1-proposed

Reviewed: https://review.openstack.org/348938
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f1dd09eef838de7093bba9902d7d0bb90fd98d9d
Submitter: Jenkins
Branch: stable/mitaka

commit f1dd09eef838de7093bba9902d7d0bb90fd98d9d
Author: Oleksiy Molchanov <email address hidden>
Date: Fri Jul 29 18:41:26 2016 +0300

    Fix for all possible storage cases

    * Enable swift only if images and objects ceph are disabled
    * Enable Radosgw through YAQL
    * Set S3 endpoint using radosgw task

    Change-Id: Iead5167210c4132badb866afc25d4ef14e27f6b2
    Closes-Bug: 1604879

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/355459
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4f456400cfcb0355a9d87158727aeb74b365e473
Submitter: Jenkins
Branch: master

commit 4f456400cfcb0355a9d87158727aeb74b365e473
Author: Oleksiy Molchanov <email address hidden>
Date: Fri Jul 29 18:41:26 2016 +0300

    Fix for all possible storage cases

    * Enable swift only if images and objects ceph are disabled
    * Enable Radosgw through YAQL
    * Set S3 endpoint using radosgw task

    Change-Id: Iead5167210c4132badb866afc25d4ef14e27f6b2
    Closes-Bug: 1604879

tags: added: on-verification
Dmitry Belyaninov (dbelyaninov) wrote :

Verified on snapshot #183.
SWIFT was installed and test passed.

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/351257
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Reviewed: https://review.openstack.org/379284
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=8e64d190e29b15ed0507b4662046c879bd0c3681
Submitter: Jenkins
Branch: stable/7.0

commit 8e64d190e29b15ed0507b4662046c879bd0c3681
Author: Rodion Tikunov <email address hidden>
Date: Wed Sep 28 18:39:05 2016 +0300

    Fix for all possible storage cases

     * Enable swift only if images and objects ceph are disabled
     * Set S3 endpoint using radosgw task

    Change-Id: Iead5167210c4132badb866afc25d4ef14e27f6b2
    Closes-Bug: #1604879
    Closes-Bug: #1590700
    (cherry picked from commit f1dd09eef838de7093bba9902d7d0bb90fd98d9d)

Reviewed: https://review.openstack.org/379550
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6c872469529d50ee188c3f35e734a576984f93bf
Submitter: Jenkins
Branch: stable/8.0

commit 6c872469529d50ee188c3f35e734a576984f93bf
Author: Rodion Tikunov <email address hidden>
Date: Wed Sep 28 13:21:21 2016 +0300

    Fix for all possible storage cases

     * Enable swift only if images and objects ceph are disabled
     * Set S3 endpoint using radosgw task

    Change-Id: I8611669217827ccc8aad64b89557783470b3d0b3
    Closes-Bug: #1604879
    Closes-Bug: #1590700
    (cherry picked from commit f1dd09eef838de7093bba9902d7d0bb90fd98d9d)

tags: removed: on-verification
tags: added: on-verification

Bug was checked on MOS 7.0 + MU6 updates, but another issue appeared:
s3 endpoint have broken adminurl and internalurl links.

Steps to reproduce:
1) Deploy 3 controller with Ceph OSD + 1 compute node
Storage Backends:
Ceph RBD for volumes (Cinder)
Ceph RadosGW for objects (Swift API)
Ceph RBD for ephemeral volumes (Nova)
Ceph RBD for images (Glance)
2) ssh to controller and perform:
source openrc
openstack endpoint show s3

Actual results:
root@node-1:~# openstack endpoint show s3
+--------------+----------------------------------+
| Field | Value |
+--------------+----------------------------------+
| adminurl | ://10.109.1.2:8080 |
| enabled | True |
| id | c212aed9a1d840d1b6a65b3de8f45252 |
| internalurl | ://:8080 |
| publicurl | https://public.fuel.local:8080 |
| region | RegionOne |
| service_id | 32665d855ed340429200ac5504b8ca01 |
| service_name | swift_s3 |
| service_type | s3 |
+--------------+----------------------------------+

From puppet.log:
2016-10-04 08:20:40 +0000 Puppet::Type::Keystone_endpoint::ProviderOpenstack (debug): OpenStack request: 'endpoint create ["swift_s3", "--region", "RegionOne", "--publicurl", "https://public.fuel.local:8080", "--internalurl", "://:8080", "--adminurl", "://10.109.1.2:8080"]' returned: '{:adminurl=>"://10.109.1.2:8080", :id=>"c212aed9a1d840d1b6a65b3de8f45252", :internalurl=>"://:8080", :publicurl=>"https://public.fuel.local:8080", :region=>"RegionOne", :service_id=>"32665d855ed340429200ac5504b8ca01", :service_name=>"swift_s3", :service_type=>"s3"}'

Reviewed: https://review.openstack.org/382170
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=78983699bbe6ec8af258a914fc980ffe20847476
Submitter: Jenkins
Branch: stable/7.0

commit 78983699bbe6ec8af258a914fc980ffe20847476
Author: Rodion Tikunov <email address hidden>
Date: Tue Oct 4 17:32:16 2016 +0300

    Fix urls for radosgw s3 endpoint

    Some urls for radosgw s3 endpoints misses their values. Change
    them according to swift/keystone.pp.

    Change-Id: Iabe4c8111f85e4cbf4e085976ab6f572bc153112
    Closes-Bug: #1604879

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

Verified on MOS 7.0 + MU6 updates.

Actual results:
root@node-1:~# openstack endpoint show object-store
+--------------+-----------------------------------------+
| Field | Value |
+--------------+-----------------------------------------+
| adminurl | http://10.109.17.2:8080/swift/v1 |
| enabled | True |
| id | b67d6c7d725e4748940276d92367c5d7 |
| internalurl | http://10.109.17.2:8080/swift/v1 |
| publicurl | https://public.fuel.local:8080/swift/v1 |
| region | RegionOne |
| service_id | c6c57bed5f084c789f518618298fddec |
| service_name | swift |
| service_type | object-store |
+--------------+-----------------------------------------+
root@node-1:~# openstack endpoint show s3
+--------------+----------------------------------+
| Field | Value |
+--------------+----------------------------------+
| adminurl | http://10.109.17.2:8080 |
| enabled | True |
| id | 7e6f389555554e5aa2bb840eef01c50c |
| internalurl | http://10.109.17.2:8080 |
| publicurl | https://public.fuel.local:8080 |
| region | RegionOne |
| service_id | bed778907426403db13e3486f4e4375a |
| service_name | swift_s3 |
| service_type | s3 |
+--------------+----------------------------------+

tags: removed: on-verification

This issue was fixed in the openstack/fuel-library 10.0.0 release.

tags: added: on-verification
Alexey Stupnikov (astupnikov) wrote :

Nominated for 6.1 as I have marked bug #1585574 as a duplicate of this one. Closing as won't fix for 6.1-updates as we only fix security issues there.

Change abandoned by Sergii Rizvan (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/454084

Reviewed: https://review.openstack.org/456197
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=fdbc795c876c0add0a73948ace259505947b6786
Submitter: Jenkins
Branch: stable/8.0

commit fdbc795c876c0add0a73948ace259505947b6786
Author: Sergii Rizvan <email address hidden>
Date: Tue Apr 11 20:12:37 2017 +0300

    Correct logical expressions for storage backends

    After introducing the change Iead5167210c4132badb866afc25d4ef14e27f6b2
    swift isn't installed if ceph used as a backend for images
    and not for object-storage service. In this sutiation we end up
    with absence of object-storage at all which is wrong.

    The commit introduces changes which makes possible to swift
    been installed if ceph used only for image service.

    Change-Id: If319845aa9f2d9b3ca07936350ef374f504679cb
    Closes-Bug: #1604879

Sergii Rizvan (srizvan) wrote :

Actually patch https://review.openstack.org/#/q/Iead5167210c4132badb866afc25d4ef14e27f6b2,n,z doesn't fix the main issue: object storage still isn't deployed if ceph selected for images but not for objects. That's why I've introduced another patch https://review.openstack.org/#/q/If319845aa9f2d9b3ca07936350ef374f504679cb,n,z which makes possible to swift been installed for object-store if ceph used only for image service.

Sergii Rizvan (srizvan) wrote :

Note for QA:
The patch should been tested in several storage backend configurations. The required minimum is:
1. No Ceph backend at all https://pageshot.net/images/f20bf975-95be-467c-9dc4-87fd200f6060.png
2. Ceph backend for Glance only https://pageshot.net/images/3cf13747-2abe-4576-a250-1ceccb6ffe71.png
3. Ceph backend for Glance and Ceph RadosGW for objects https://pageshot.net/images/65252a5d-a5e2-46c7-bae2-1ab73c51d7a4.png
4. Swift for Glance and Ceph RadosGW for objects https://pageshot.net/images/65cd5324-b0cb-4989-8680-daf3041152cb.png

In all cases object storage should be deployed correctly and it's should be possible to create containers, upload objects to them, etc.

Reviewed: https://review.openstack.org/458816
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=285c29cf12b2cdc5ac978b36bc5777beefb6af46
Submitter: Jenkins
Branch: stable/8.0

commit 285c29cf12b2cdc5ac978b36bc5777beefb6af46
Author: Anton Chevychalov <email address hidden>
Date: Fri Apr 21 15:32:18 2017 +0300

    Enable swift when there is no RadosGW

    There is a regression after
    If319845aa9f2d9b3ca07936350ef374f504679cb.
    According to current logic RadosGW and Swift can be installed
    in a same time.

    That commit changes logic. If we have RadosGW, then we have no
    Swift. If we have no RadosGW, then we have Swift.

    Change-Id: I2e41551dc7df9f3be81df26000b1ae2bb7d76b39
    Closes-Bug: #1604879

Ekaterina Shutova (eshutova) wrote :

Verified on 8.0 mu4 updates.

tags: removed: on-verification

Reviewed: https://review.openstack.org/455787
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=987b3a6af7eee9bf9779ee08e65ac399c2b1eca1
Submitter: Jenkins
Branch: stable/7.0

commit 987b3a6af7eee9bf9779ee08e65ac399c2b1eca1
Author: Sergii Rizvan <email address hidden>
Date: Tue Apr 11 20:12:37 2017 +0300

    Correct logical expressions for storage backends

    After introducing the change Iead5167210c4132badb866afc25d4ef14e27f6b2
    swift isn't installed if ceph used as a backend for images
    and not for object-storage service. In this sutiation we end up
    with absence of object-storage at all which is wrong.

    The commit introduces changes which makes possible
    to swift been installed if ceph used only for image service.

    Change-Id: If319845aa9f2d9b3ca07936350ef374f504679cb
    Closes-Bug: #1604879

Sergii Rizvan (srizvan) on 2017-05-23
Changed in fuel:
status: Fix Committed → In Progress
assignee: Oleksiy Molchanov (omolchanov) → Sergii Rizvan (srizvan)
tags: added: on-verification

Verified on 7.0 mu8 updates.

The first three cases from https://bugs.launchpad.net/fuel/+bug/1604879/comments/29 were checked, the forth case is unavailable from Fuel UI.
Object storage is deployed correctly for all three cases and it's possible to create containers, upload objects to them, etc.

tags: removed: on-verification

Reviewed: https://review.openstack.org/465595
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=57bb7299b75ab5f9457878cc0bfb88af74ed767a
Submitter: Jenkins
Branch: master

commit 57bb7299b75ab5f9457878cc0bfb88af74ed767a
Author: Sergii Rizvan <email address hidden>
Date: Tue Apr 11 20:12:37 2017 +0300

    Correct logical expressions for storage backends

    After introducing the change Iead5167210c4132badb866afc25d4ef14e27f6b2
    swift isn't installed if ceph used as a backend for images
    and not for object-storage service. In this sutiation we end up
    with absence of object-storage at all which is wrong.

    The commit introduces changes which makes possible
    to swift been installed if ceph used only for image service.

    Change-Id: If319845aa9f2d9b3ca07936350ef374f504679cb
    Closes-Bug: #1604879

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/465590
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9e3e2da3053bb5e697bd2cacae4a682512598992
Submitter: Jenkins
Branch: stable/newton

commit 9e3e2da3053bb5e697bd2cacae4a682512598992
Author: Sergii Rizvan <email address hidden>
Date: Tue Apr 11 20:12:37 2017 +0300

    Correct logical expressions for storage backends

    After introducing the change Iead5167210c4132badb866afc25d4ef14e27f6b2
    swift isn't installed if ceph used as a backend for images
    and not for object-storage service. In this sutiation we end up
    with absence of object-storage at all which is wrong.

    The commit introduces changes which makes possible
    to swift been installed if ceph used only for image service.

    Change-Id: If319845aa9f2d9b3ca07936350ef374f504679cb
    Closes-Bug: #1604879

tags: added: in-stable-ocata

Reviewed: https://review.openstack.org/465594
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=fd8268b89be861b1e3edd710dcf48e5a2c7fd98f
Submitter: Jenkins
Branch: stable/ocata

commit fd8268b89be861b1e3edd710dcf48e5a2c7fd98f
Author: Sergii Rizvan <email address hidden>
Date: Tue Apr 11 20:12:37 2017 +0300

    Correct logical expressions for storage backends

    After introducing the change Iead5167210c4132badb866afc25d4ef14e27f6b2
    swift isn't installed if ceph used as a backend for images
    and not for object-storage service. In this sutiation we end up
    with absence of object-storage at all which is wrong.

    The commit introduces changes which makes possible
    to swift been installed if ceph used only for image service.

    Change-Id: If319845aa9f2d9b3ca07936350ef374f504679cb
    Closes-Bug: #1604879

Fix proposed to branch: stable/mitaka
Change author: Sergii Rizvan <email address hidden>
Review: https://review.fuel-infra.org/36845

Change abandoned by Max Yatsenko <email address hidden> on branch: stable/mitaka
Review: https://review.fuel-infra.org/36845

Fix proposed to branch: 9.0/mitaka
Change author: Sergii Rizvan <email address hidden>
Review: https://review.fuel-infra.org/36847

Reviewed: https://review.fuel-infra.org/36847
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 62f42f387e678257e4ab16eabaf31e361b6c2164
Author: Sergii Rizvan <email address hidden>
Date: Thu Oct 12 07:59:51 2017

Correct logical expressions for storage backends

After introducing the change Iead5167210c4132badb866afc25d4ef14e27f6b2
swift isn't installed if ceph used as a backend for images
and not for object-storage service. In this sutiation we end up
with absence of object-storage at all which is wrong.

The commit introduces changes which makes possible
to swift been installed if ceph used only for image service.

Change-Id: If319845aa9f2d9b3ca07936350ef374f504679cb
Closes-Bug: #1604879

Vladimir Jigulin (vjigulin) wrote :

Verified on mos9+mu3 proposed

Deployed with:
Ceph RBD for volumes (Cinder)Configures Cinder to store volumes in Ceph RBD images.
Ceph RBD for images (Glance)Configures Glance to use the Ceph RBD backend to store images.
Ceph RBD for ephemeral volumes (Nova)

Without proposed repo:
root@node-2:~# openstack endpoint show s3
No service with a type, name or ID of 's3' exists.
root@node-2:~# openstack endpoint show object-store
No service with a type, name or ID of 'object-store' exists.

With proposed:
root@node-2:~# openstack endpoint show s3
+--------------+----------------------------------+
| Field | Value |
+--------------+----------------------------------+
| adminurl | http://10.109.7.9:8080 |
| enabled | True |
| id | 8eb8b8ad5c9e4e179005c3a077ca0c8a |
| internalurl | http://10.109.7.9:8080 |
| publicurl | http://10.109.10.6:8080 |
| region | RegionOne |
| service_id | 1ee7fbb29f534b3f9e8accbb152323de |
| service_name | swift_s3 |
| service_type | s3 |
+--------------+----------------------------------+
root@node-2:~# openstack endpoint show object-store
+--------------+-----------------------------------------------+
| Field | Value |
+--------------+-----------------------------------------------+
| adminurl | http://10.109.7.9:8080/v1/AUTH_%(tenant_id)s |
| enabled | True |
| id | 86119e71e30b486388145e1f9da0e0ef |
| internalurl | http://10.109.7.9:8080/v1/AUTH_%(tenant_id)s |
| publicurl | http://10.109.10.6:8080/v1/AUTH_%(tenant_id)s |
| region | RegionOne |
| service_id | c60c7967302744a1b87aa9cc366f8cc6 |
| service_name | swift |
| service_type | object-store |
+--------------+-----------------------------------------------+

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers