Keystone failed to create user for Ceilometer when adding Controller+Ironic node

[Kyrylo Romanenko]

Steps to reproduce:
1. Deploy cluster with following settings

Enable Ironic and Ceilometer
Add nodes:
1 Controller
1 Compute
1 Mongo
1 Ironic

Compute QEMU
Network Neutron with VLAN segmentation
Storage Backends Cinder LVM over iSCSI for volumes

2. Then try to add one more node with combined roles: Ironic+Controller

3. Deploy changes.

4. After some time deployment process fails.

Deployment failed on Controller node with errors in puppet log:

2016-02-22 17:45:06 WARNING (/Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user_role[ceilometer@services]) Skipping because of failed dependencies
2016-02-22 17:45:06 ERR (/Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]/ensure) change from absent to present failed: Command: 'openstack ["user", "create", "--format", "shell", ["ceilometer", "--enable", "--password", "tSxFVj9JLjMX2Ph7rj5u9Z77", "--email", "ceilometer@localhost", "--domain", "Default"]]' has been running for more then 20 seconds!

In keystone_wsgi_admin_error.log:
2016-02-22 14:06:08 ERR [Mon Feb 22 14:06:00.674141 2016] [wsgi:error] [pid 3288:tid 140019868092160] [client 192.168.0.2:46563] Timeout when reading response headers from daemon process 'keystone_admin': /usr/lib/cgi-bin/keystone/admin
2016-02-22 14:05:58 ERR [Mon Feb 22 14:05:53.866172 2016] [wsgi:error] [pid 2726:tid 140019834521344] [client 192.168.0.2:46481] Timeout when reading response headers from daemon process 'keystone_admin': /usr/lib/cgi-bin/keystone/admin

I have got those results twice in a row, attempted to redeploy in one more time.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

[Kyrylo Romanenko (kromanenko)]

Diagnostic snapshot link:
https://drive.google.com/a/mirantis.com/file/d/0B6E70aHvCcRQSTR5aDB6WC1KMjA/view?usp=sharing

[Roman Podoliaka (rpodolyaka)]

Kyrylo, are you sure you have enough memory on the controller node? These timeout issues you mentioned makes me think you started to use swap and things became really slow on the controller node...

[Kyrylo Romanenko (kromanenko)]

There is 3072MB RAM on slave nodes. Should there to be some more RAM?

[Kyrylo Romanenko (kromanenko)]

Looks like just lack of RAM on controller node

root@node-1:~# free
             total used free shared buffers cached
Mem: 3082244 2932312 149932 25980 10936 100636
-/+ buffers/cache: 2820740 261504
Swap: 3145724 2508876 636848