Ceph RADOS Gateway Charm

hook failed: "identity-service-relation-changed" for keystone:identity-service

Bug #1708464 reported by Narinder Gupta
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceph RADOS Gateway Charm
Fix Released
Critical
David Ames
Ceph RADOS Gateway Charm 17.08

Bug Description

I deployed the Ocata release with Latest git version of charm and founds out that with Keystone v3 it refuses to relate with identity service.

i clone latest ceph-radosgw charm and having this issue http://paste.ubuntu.com/25233428/ where relation change with keystone failed during deployment

juju status at http://paste.ubuntu.com/25233488/

fetching charms locally using http://paste.ubuntu.com/25233493/

bundle file to deploy http://paste.ubuntu.com/25233492/

Revision history for this message
James Page (james-page) wrote :

bug 1691875 should have resolved v3 compatibility

Revision history for this message
Narinder Gupta (narindergupta) wrote :
Download full text (5.9 KiB)

somehow still seeing error with the fixed charm code.

2017-08-03 13:51:06 DEBUG identity-service-relation-changed /usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py:136: UserWarning: Using keystoneclient sessions has been deprecated. Please update your software to use keystoneauth1.
2017-08-03 13:51:06 DEBUG identity-service-relation-changed warnings.warn('Using keystoneclient sessions has been deprecated. '
2017-08-03 13:51:07 DEBUG identity-service-relation-changed Traceback (most recent call last):
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/identity-service-relation-changed", line 353, in <module>
2017-08-03 13:51:07 DEBUG identity-service-relation-changed hooks.execute(sys.argv)
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/core/hookenv.py", line 731, in execute
2017-08-03 13:51:07 DEBUG identity-service-relation-changed self._hooks[hook_name]()
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1931, in wrapped_f
2017-08-03 13:51:07 DEBUG identity-service-relation-changed restart_functions)
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/core/host.py", line 659, in restart_on_change_helper
2017-08-03 13:51:07 DEBUG identity-service-relation-changed r = lambda_f()
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1930, in <lambda>
2017-08-03 13:51:07 DEBUG identity-service-relation-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/identity-service-relation-changed", line 229, in identity_changed
2017-08-03 13:51:07 DEBUG identity-service-relation-changed identity_joined(relid)
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/identity-service-relation-changed", line 223, in identity_joined
2017-08-03 13:51:07 DEBUG identity-service-relation-changed setup_keystone_certs()
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/utils.py", line 325, in _inner2_defer_if_unavailable
2017-08-03 13:51:07 DEBUG identity-service-relation-changed return f(*args, **kwargs)
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/utils.py", line 459, in setup_keystone_certs
2017-08-03 13:51:07 DEBUG identity-service-relation-changed get_ks_ca_cert(settings['admin_token'], auth_endpoint, certs_path)
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/utils.py", line 325, in _inner2_defer_if_unavailable
2017-08-03 13:51:07 DEBUG identi...

Read more...

Revision history for this message
Narinder Gupta (narindergupta) wrote :

Another data point all openstack charm from openstack master branch works except ceph-radogw. If i uses ceph-radosgw from 17.02 branch it works fine. So looks like issue exist which may or may not related to keystone v3.

Revision history for this message
Narinder Gupta (narindergupta) wrote :

Still getting the same error today when get the charm from latest branch.

juju logs from the unit-ceph-radosgw

http://paste.ubuntu.com/25367267/

git clone https://github.com/openstack/charm-hacluster.git $distro/hacluster
git clone https://github.com/openstack/charm-ceilometer.git $distro/ceilometer
git clone https://github.com/openstack/charm-ceilometer-agent.git $distro/ceilometer-agent
git clone https://github.com/openstack/charm-ceph-mon.git $distro/ceph-mon
git clone https://github.com/openstack/charm-ceph-osd.git $distro/ceph-osd
#git clone -b stable/17.02 https://github.com/openstack/charm-ceph-radosgw.git $distro/ceph-radosgw
git clone https://github.com/openstack/charm-ceph-radosgw.git $distro/ceph-radosgw
git clone https://github.com/openstack/charm-cinder.git $distro/cinder
git clone https://github.com/openstack/charm-cinder-ceph.git $distro/cinder-ceph
git clone https://github.com/openstack/charm-glance.git $distro/glance
git clone https://github.com/openstack/charm-keystone.git $distro/keystone
git clone https://github.com/openstack/charm-percona-cluster.git $distro/percona-cluster
git clone https://github.com/openstack/charm-neutron-api.git $distro/neutron-api
git clone https://github.com/openstack/charm-neutron-gateway.git $distro/neutron-gateway
git clone https://github.com/openstack/charm-neutron-openvswitch.git $distro/neutron-openvswitch
git clone https://github.com/openstack/charm-nova-cloud-controller.git $distro/nova-cloud-controller
git clone https://github.com/openstack/charm-nova-compute.git $distro/nova-compute
git clone https://github.com/openstack/charm-openstack-dashboard.git $distro/openstack-dashboard
git clone https://github.com/openstack/charm-rabbitmq-server.git $distro/rabbitmq-server
git clone https://github.com/openstack/charm-heat.git $distro/heat
git clone https://github.com/openstack/charm-lxd.git $distro/lxd
git clone https://github.com/openbaton/juju-charm.git $distro/openbaton

Revision history for this message
Narinder Gupta (narindergupta) wrote :
Download full text (6.1 KiB)

2017-08-03 13:51:06 DEBUG identity-service-relation-changed /usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py:136: UserWarning: Using keystoneclient sessions has been deprecated. Please update your software to use keystoneauth1.
2017-08-03 13:51:06 DEBUG identity-service-relation-changed warnings.warn('Using keystoneclient sessions has been deprecated. '
2017-08-03 13:51:07 DEBUG identity-service-relation-changed Traceback (most recent call last):
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/identity-service-relation-changed", line 353, in <module>
2017-08-03 13:51:07 DEBUG identity-service-relation-changed hooks.execute(sys.argv)
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/core/hookenv.py", line 731, in execute
2017-08-03 13:51:07 DEBUG identity-service-relation-changed self._hooks[hook_name]()
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1931, in wrapped_f
2017-08-03 13:51:07 DEBUG identity-service-relation-changed restart_functions)
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/core/host.py", line 659, in restart_on_change_helper
2017-08-03 13:51:07 DEBUG identity-service-relation-changed r = lambda_f()
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1930, in <lambda>
2017-08-03 13:51:07 DEBUG identity-service-relation-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/identity-service-relation-changed", line 229, in identity_changed
2017-08-03 13:51:07 DEBUG identity-service-relation-changed identity_joined(relid)
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/identity-service-relation-changed", line 223, in identity_joined
2017-08-03 13:51:07 DEBUG identity-service-relation-changed setup_keystone_certs()
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/utils.py", line 325, in _inner2_defer_if_unavailable
2017-08-03 13:51:07 DEBUG identity-service-relation-changed return f(*args, **kwargs)
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/utils.py", line 459, in setup_keystone_certs
2017-08-03 13:51:07 DEBUG identity-service-relation-changed get_ks_ca_cert(settings['admin_token'], auth_endpoint, certs_path)
2017-08-03 13:51:07 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/utils.py", line 325, in _inner2_defer_if_unavailable
2017-08-03 13:51:07 DEBUG identity-service-relation-changed return f(*args, **kwar...

Read more...

David Ames (thedac)
Changed in charm-ceph-radosgw:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → David Ames (thedac)
milestone: none → 17.08
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ceph-radosgw (master)

Fix proposed to branch: master
Review: https://review.openstack.org/498122

Revision history for this message
Narinder Gupta (narindergupta) wrote :

i can confirm that patch https://review.openstack.org/498122

 fixes my issues.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ceph-radosgw (master)

Reviewed: https://review.openstack.org/498122
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-radosgw/commit/?id=22265186125557a53ae77df8f44d0d93f9cb7db5
Submitter: Jenkins
Branch: master

commit 22265186125557a53ae77df8f44d0d93f9cb7db5
Author: David Ames <email address hidden>
Date: Fri Aug 25 16:30:02 2017 -0700

    Use ApacheSSLContext to enable SSL object storage

    Enable SSL object storage using ApacheSSLContext.

    Change-Id: Id044afc8c07696a5447eb9dc4836470203372090
    Closes-Bug: #1690826
    Closes-Bug: #1708464

Changed in charm-ceph-radosgw:
status: In Progress → Fix Committed
Revision history for this message
Narinder Gupta (narindergupta) wrote :
Download full text (4.9 KiB)

Verficiation of bug failed with keystone v3. here is new trace.
2017-08-30 19:00:02 DEBUG identity-service-relation-changed apache2.service is not active, cannot reload.
2017-08-30 19:00:02 DEBUG worker.uniter.jujuc server.go:179 hook context id "ceph-radosgw/0-identity-service-relation-changed-7890508814909344620"; dir "/var/lib/juju/agents/unit-ceph-radosgw-0/charm"
2017-08-30 19:00:02 DEBUG juju-log identity-service:57: Updating ca cert from keystone
2017-08-30 19:00:02 DEBUG identity-service-relation-changed unable to load certificate
2017-08-30 19:00:02 DEBUG identity-service-relation-changed 140698600187544:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE
2017-08-30 19:00:02 DEBUG identity-service-relation-changed CA_CERT: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}
2017-08-30 19:00:02 DEBUG identity-service-relation-changed Traceback (most recent call last):
2017-08-30 19:00:02 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/identity-service-relation-changed", line 388, in <module>
2017-08-30 19:00:02 DEBUG identity-service-relation-changed hooks.execute(sys.argv)
2017-08-30 19:00:02 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/core/hookenv.py", line 768, in execute
2017-08-30 19:00:02 DEBUG identity-service-relation-changed self._hooks[hook_name]()
2017-08-30 19:00:02 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1887, in wrapped_f
2017-08-30 19:00:02 DEBUG identity-service-relation-changed restart_functions)
2017-08-30 19:00:02 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/core/host.py", line 685, in restart_on_change_helper
2017-08-30 19:00:02 DEBUG identity-service-relation-changed r = lambda_f()
2017-08-30 19:00:02 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1886, in <lambda>
2017-08-30 19:00:02 DEBUG identity-service-relation-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2017-08-30 19:00:02 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/identity-service-relation-changed", line 233, in identity_changed
2017-08-30 19:00:02 DEBUG identity-service-relation-changed configure_https()
2017-08-30 19:00:02 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/identity-service-relation-changed", line 377, in configure_https
2017-08-30 19:00:02 DEBUG identity-service-relation-changed setup_keystone_certs(CONFIGS)
2017-08-30 19:00:02 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/utils.py", line 354, in _inner2_defer_if_unavailable
2017-08-30 19:00:02 DEBUG identity-service-relation-changed return f(*args, **kwargs)
2017-08-30 19:00:02 DEBUG identity-serv...

Read more...

Revision history for this message
Narinder Gupta (narindergupta) wrote :

complete log file can be find at
http://paste.ubuntu.com/25433680/

Changed in charm-ceph-radosgw:
status: Fix Committed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ceph-radosgw (master)

Fix proposed to branch: master
Review: https://review.openstack.org/500205

Revision history for this message
Narinder Gupta (narindergupta) wrote : Re: [Bug 1708464] Re: hook failed: "identity-service-relation-changed" for keystone:identity-service

I can confirm that installation succeeded with this patch in OPNFV.
Currently running rally and tempest test to verify.

Thanks and Regards,
Narinder Gupta (PMP) <email address hidden>
Canonical, Ltd. narindergupta [irc.freenode.net]
+1.281.736.5150 narindergupta2007[skype]

Ubuntu- Linux for human beings | www.ubuntu.com | www.canonical.com

On Wed, Aug 30, 2017 at 2:02 PM, Narinder Gupta <email address hidden>
wrote:

> complete log file can be find at
> http://paste.ubuntu.com/25433680/
>
> ** Changed in: charm-ceph-radosgw
> Status: Fix Committed => In Progress
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1708464
>
> Title:
> hook failed: "identity-service-relation-changed" for keystone
> :identity-service
>
> Status in OpenStack ceph-radosgw charm:
> In Progress
>
> Bug description:
> I deployed the Ocata release with Latest git version of charm and
> founds out that with Keystone v3 it refuses to relate with identity
> service.
>
> i clone latest ceph-radosgw charm and having this issue
> http://paste.ubuntu.com/25233428/ where relation change with keystone
> failed during deployment
>
> juju status at http://paste.ubuntu.com/25233488/
>
> fetching charms locally using http://paste.ubuntu.com/25233493/
>
> bundle file to deploy http://paste.ubuntu.com/25233492/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-ceph-radosgw/+bug/1708464/+subscriptions
>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ceph-radosgw (master)

Reviewed: https://review.openstack.org/500205
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-radosgw/commit/?id=a1444c4b2fe17bd1aef537115325f7c8d2e76fd2
Submitter: Jenkins
Branch: master

commit a1444c4b2fe17bd1aef537115325f7c8d2e76fd2
Author: David Ames <email address hidden>
Date: Fri Sep 1 14:58:45 2017 -0700

    Ensure crgw works with Keystone v3

    The URLs for SSL certificates are different and the v2 client
    functions do not work. This change fixes these issues.

    Add missing config.yaml options.

    Change-Id: Ia4c7b508e70f690493b098c6fb07d24a340bc2a6
    Closes-Bug: #1708464

Changed in charm-ceph-radosgw:
status: In Progress → Fix Committed
James Page (james-page)
Changed in charm-ceph-radosgw:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.