Controller deploy fails "settings is not a hash or array when accessing it with nodes at line 1 on node node-1.domain.tld"

[Sergey Murashov]

ISO: {"build_id": "2014-08-06_16-01-30", "ostf_sha": "be71965998364bf8e6415bd38b75c84b63aab867", "build_number": "2", "auth_required": true, "api": "1.0", "nailgun_sha": "75c834f6b99b495c5a71b9e4609a3ab8b6140dbb", "production": "docker", "fuelmain_sha": "88afef2ef0aec4493a17583443ca93ba44e0f20a", "astute_sha": "b52910642d6de941444901b0f20e95ebbcb2b2e9", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "513ec5cdcdef74c7419d5bae967b9edc7da8dbd7"}

Steps to reproduce:
1. Deploy OS(CentOS, neutron GRE, simple mode, murano, 1 controller, 1 compute)

Actual result:
Openstack deployment fails on controller node with the following error in puppet: "settings is not a hash or array when accessing it with nodes at line 1 on node node-1.domain.tld"

[Bogdan Dobrelya (bogdando)]

Looks like db issue

2014-08-06T14:47:08.938935 node-2 ./remote/node-2.domain.tld/murano-manage.log:2014-08-06T14:47:08.938935+01:00 err: 2014-08-06 13:47:08.938 20236 ERROR muranoapi.cmd.manage [-] murano-manage command failed: (ProgrammingError) (1146, "Table 'murano.tag' doesn't exist") 'SELECT tag.created AS tag_created, tag.updated AS tag_updated, tag.id AS tag_id, tag.name AS tag_name \nFROM tag \nWHERE tag.name = %s \n LIMIT %s' ('MuranoPL', 1)
2014-08-06T14:47:08.950591 node-2 ./remote/node-2.domain.tld/murano-api.log:2014-08-06T14:47:08.950591+01:00 err: 2014-08-06 13:47:08.950 20226 ERROR muranoapi.common.statservice [-] Failed to get statistics object form a database. (ProgrammingError) (1146, "Table 'murano.apistats' doesn't exist") 'SELECT apistats.created AS apistats_created, apistats.updated AS apistats_updated, apistats.id AS apistats_id, apistats.host AS apistats_host, apistats.request_count AS apistats_request_count, apistats.error_count AS apistats_error_count, apistats.average_response_time AS apistats_average_response_time, apistats.requests_per_tenant AS apistats_requests_per_tenant, apistats.requests_per_second AS apistats_requests_per_second, apistats.errors_per_second AS apistats_errors_per_second, apistats.cpu_count AS apistats_cpu_count, apistats.cpu_percent AS apistats_cpu_percent \nFROM apistats \nWHERE apistats.host = %s \n LIMIT %s' ('node-2.domain.tld', 1)

[Bogdan Dobrelya (bogdando)]

2014-08-06T14:45:05.867222 node-2 ./remote/node-2.domain.tld/mysqld.log:2014-08-06T14:45:05.867222+01:00 err: 140806 13:45:05 InnoDB: Database was
not shut down normally!
2014-08-06T14:45:05.868189 node-2 ./remote/node-2.domain.tld/mysqld.log:2014-08-06T14:45:05.868189+01:00 err: InnoDB: Starting crash recovery.
2014-08-06T14:45:05.869721 node-2 ./remote/node-2.domain.tld/mysqld.log:2014-08-06T14:45:05.869721+01:00 err: InnoDB: Reading tablespace information
 from the .ibd files...
2014-08-06T14:45:05.872466 node-2 ./remote/node-2.domain.tld/mysqld.log:2014-08-06T14:45:05.872466+01:00 err: InnoDB: Restoring possible half-writte
n data pages from the doublewrite

[Bogdan Dobrelya (bogdando)]

Normally DB will not be broken inside of deployment process, especially for multinode non HA case. The issue should be related to connectivity issues. Need more reproducers

[Aleksey Kasatkin (alekseyk-ru)]

Seems to be sporadical failure. Didn't reproduce on newer build. Lower the priority.

  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "414"
  build_id: "2014-08-08_11-23-51"
  astute_sha: "b52910642d6de941444901b0f20e95ebbcb2b2e9"
  fuellib_sha: "d699fc178559e98cfd7d53b58478b46553ffe39e"
  ostf_sha: "e33390c275e225d648b36997460dc29b1a3c20ae"
  nailgun_sha: "5bc33457e5a1f108b071ed0ef2a771ea0b610b22"
  fuelmain_sha: "16c54168143061724e635a20a99545a756725b49"

[Aleksey Kasatkin (alekseyk-ru)]

Didn't reproduce within two attempts.

[Alexei Sheplyakov (asheplyakov)]

A similar problem hits me every other day. The symptom is:

root@node-1:~# puppet apply /etc/puppet/manifests/site.pp
Error: ::fuel_settings is not a hash or array when accessing it with deployment_id at /etc/puppet/manifests/site.pp:12 on node node-1.domain.tld
Error: ::fuel_settings is not a hash or array when accessing it with deployment_id at /etc/puppet/manifests/site.pp:12 on node node-1.domain.tld

This might be a different bug, though.

[Sergey Vasilenko (xenolog)]

ISO #114, controller node while deploing.

root@node-1:~# grep 'deployment_id' /etc/puppet/manifests/site.pp
tag("${::fuel_settings['deployment_id']}::${::fuel_settings['environment']}")
root@node-1:~# grep 'deployment_id' /etc/astute.yaml
deployment_id: 1
root@node-1:~# grep 'environment' /etc/astute.yaml
root@node-1:~#

i.e. there is no "environment" parameter in astute.yaml

[Sergey Vasilenko (xenolog)]

But no such error in puppet.log

[soumaya (soumayanayek84)]

Surprisingly i am getting the same error " settings is not a hash or array when accessing it with nodes at line 1 on node xxxx.xxx"
and unable to find the resolution . Is is a bug or something else i am not able to analyze , please suggest .