Keystone api sporadically stops answering

[Vladimir Kuklin]

Try to deploy simple bvt case. See that some fair percent of requests to keystone API on the first controller fail sporadically. This leads to the failure of keystone deployment task several times failing the deployment completely. This seems to be purely keystone bug as there are no other actions happenning on the node as requests continue working ok after a while and breaking again afterwards.

Full logs and info can be found here:

https://ci.fuel-infra.org/job/master.fuel-library.pkgs.ubuntu.neutron_vlan_ha/5455/

[Vladimir Kuklin (vkuklin)]

Here is a snippet from the logs

2016-01-28 12:02:02 +0000 /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_endpoint[RegionOne/keystone] (info): Starting to evaluate the resource
2016-01-28 12:02:02 +0000 Puppet (debug): Executing '/usr/bin/openstack endpoint create --format shell keystone --region RegionOne --publicurl https://10.109.3.3:5000/v2.0 --internalurl http://10.109.1.3:5000/v2.0 --adminurl http://10.109.1.3:35357/v2.0'
2016-01-28 12:02:22 +0000 Puppet::Type::Keystone_endpoint::ProviderOpenstack (debug): Non-fatal error: 'Command: 'openstack ["endpoint", "create", "--format", "shell", ["keystone", "--region", "RegionOne", "--publicurl", "https://10.109.3.3:5000/v2.0", "--internalurl", "http://10.109.1.3:5000/v2.0", "--adminurl", "http://10.109.1.3:35357/v2.0"]]' has been running for more then 20 seconds!'. Retrying for 40 more seconds
2016-01-28 12:02:22 +0000 Puppet (err): Command: 'openstack ["endpoint", "create", "--format", "shell", ["keystone", "--region", "RegionOne", "--publicurl", "https://10.109.3.3:5000/v2.0", "--internalurl", "http://10.109.1.3:5000/v2.0", "--adminurl", "http://10.109.1.3:35357/v2.0"]]' has been running for more then 20 seconds!
/etc/puppet/modules/openstacklib/lib/puppet/provider/openstack.rb:49:in `rescue in openstack'
/etc/puppet/modules/openstacklib/lib/puppet/provider/openstack.rb:44:in `openstack'
/etc/puppet/modules/openstacklib/lib/puppet/provider/openstack.rb:87:in `block (2 levels) in request'
/etc/puppet/modules/openstacklib/lib/puppet/provider/openstack.rb:65:in `loop'
/etc/puppet/modules/openstacklib/lib/puppet/provider/openstack.rb:65:in `block in request'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:43:in `withenv'
/etc/puppet/modules/openstacklib/lib/puppet/provider/openstack.rb:62:in `request'
/etc/puppet/modules/openstacklib/lib/puppet/provider/openstack/auth.rb:42:in `request'
/etc/puppet/modules/keystone/lib/puppet/provider/keystone.rb:158:in `request'
/etc/puppet/modules/keystone/lib/puppet/provider/keystone_endpoint/openstack.rb:36:in `create'
/usr/lib/ruby/vendor_ruby/puppet/property/ensure.rb:16:in `block in defaultvalues'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:197:in `call_valuemethod'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:498:in `set'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:581:in `sync'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:in `sync'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `...

[Ivan Berezovskiy (iberezovskiy)]

If it seems to be 'purely keystone bug', I think It's better to assign on Keystone team for researching.

[Ilya Shakhat (shakhat)]

There's no memcached in the list of running process nor in the list of active network connections. Is it running?

[Roman Podoliaka (rpodolyaka)]

We haven't seen this on 8.0 - moving to Incomplete until we confirm it's affected.

[Vladimir Kuklin (vkuklin)]

I figured out the issue with tasks priorities for memcached. Closing the bug as invlaid