r4.0-17 newton provision of 16.04.2 storage cluster via r4.0-17 mitaka SM.
+-----------------+---------------------+---------------+-------------------+
| id | status | ip_address | mac_address |
+-----------------+---------------------+---------------+-------------------+
| cmbu-ceph-perf1 | provision_failed | 10.87.140.197 | 00:25:90:AB:9C:88 |
| cmbu-ceph-perf2 | provision_completed | 10.87.140.198 | 00:25:90:35:8A:1F |
| cmbu-ceph-perf3 | provision_completed | 10.87.140.199 | 00:25:90:92:0E:6C |
| cmbu-ceph-perf4 | provision_completed | 10.87.140.200 | 00:25:90:92:0D:F2 |
+-----------------+---------------------+---------------+-------------------+
"2017-06-01 23:49:28,210-INFO-sm_ansible_callback.py:43-append(): TASK [node : Create compute_list - step 2 (when ctrl_data_network is defined)]"
"2017-06-01 23:49:28,253-INFO-sm_ansible_callback.py:43-append(): fatal: [10.87.140.197]: FAILED! => {"failed": true, "msg": "the field 'args' has an invalid value, which appears to include a variable that is undefined. The error was: 'dict object' has no attribute 'ansible_hostname'\n\nThe error appears to have been in '/opt/contrail/server_manager/ansible/playbooks/combo_4_0_17newton/playbooks/roles/node/tasks/main.yml': line 194, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Create compute_list - step 2 (when ctrl_data_network is defined)\n ^ here\n"}"
"2017-06-01 23:49:28,257-DEBUG-server_mgr_status.py:112-put_ansible_status(): Server status Data cmbu-ceph-perf1 provision_failed 2017_06_01__23_49_28"
"2017-06-01 23:49:28,267-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.87.140.197] : ok: 43 changed: 17 unreachable: 0 skipped: 30 failed: 1"
"2017-06-01 23:49:28,270-DEBUG-server_mgr_status.py:112-put_ansible_status(): Server status Data cmbu-ceph-perf2 provision_completed 2017_06_01__23_49_28"
"2017-06-01 23:49:28,282-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.87.140.198] : ok: 0 changed: 0 unreachable: 1 skipped: 0 failed: 0"
"2017-06-01 23:49:28,284-DEBUG-server_mgr_status.py:112-put_ansible_status(): Server status Data cmbu-ceph-perf3 provision_completed 2017_06_01__23_49_28"
"2017-06-01 23:49:28,297-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.87.140.199] : ok: 0 changed: 0 unreachable: 1 skipped: 0 failed: 0"
"2017-06-01 23:49:28,300-DEBUG-server_mgr_status.py:112-put_ansible_status(): Server status Data cmbu-ceph-perf4 provision_completed 2017_06_01__23_49_28"
"2017-06-01 23:49:28,315-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.87.140.200] : ok: 0 changed: 0 unreachable: 1 skipped: 0 failed: 0"
"2017-06-01 23:49:28,321-INFO-sm_ansible_server.py:42-run(): Process Done"
"2017-06-01 23:49:36,695-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.140.200, option => key, ERROR => [Errno None] Unable to connect to port 22 on 10.87.140.200"
"2017-06-01 23:50:48,694-DEBUG-server_mgr_mon_base_plugin.py:706-create_server_dict(): Created server dictionary."
"2017-06-01 23:50:51,681-DEBUG-server_mgr_main.py:1273-validate_smgr_request(): validate_smgr_request"
"2017-06-01 23:50:51,694-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.140.199, option => key, ERROR => [Errno None] Unable to connect to port 22 on 10.87.140.199"
"2017-06-01 23:51:18,624-DEBUG-server_mgr_main.py:1273-validate_smgr_request(): validate_smgr_request"
"2017-06-01 23:52:06,695-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.140.198, option => key, ERROR => [Errno None] Unable to connect to port 22 on 10.87.140.198"
Jun 1 23:49:47 cmbu-ceph-perf1 barbican-worker[19718]: 2017-06-01 23:49:47.200 19718 ERROR oslo.messaging._drivers.impl_rabbit [-] [2d7bd482-c075-4d37-9082-45e2e8201064] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: 53846
Jun 1 23:49:47 cmbu-ceph-perf1 nova-console[26105]: 2017-06-01 23:49:47.617 26105 ERROR oslo.messaging._drivers.impl_rabbit [-] [897bb6dc-b97f-48d2-a4dd-7c36bac31351] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: 54810
"provision_role_sequence": "{'completed': [(u'cmbu-ceph-perf1', 'openstack', '2017_06_01__17_58_57'), (u'cmbu-ceph-perf1', 'post_provision', '2017_06_01__23_48_45')], 'steps': []}",
issue not related to storage. I see communication error to the compute nodes. The management interface is not up. ceph-perf2: ~# ifconfig 90ff:fe35: 8a1f/64 Scope:Link
collisions: 0 txqueuelen:1000
Interrupt: 17 Memory: faee0000- faf00000
root@cmbu-
enp4s0 Link encap:Ethernet HWaddr 00:25:90:35:8a:1f
inet6 addr: fe80::225:
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:680469 errors:0 dropped:0 overruns:0 frame:0
TX packets:16276 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:67705163 (67.7 MB) TX bytes:3074974 (3.0 MB)
lo Link encap:Local Loopback
collisions: 0 txqueuelen:1
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
p6p2 Link encap:Ethernet HWaddr 90:e2:ba:1c:be:69 baff:fe1c: be69/64 Scope:Link
collisions: 0 txqueuelen:1000
inet addr:5.0.0.2 Bcast:5.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::92e2:
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:188 errors:0 dropped:0 overruns:0 frame:0
TX packets:132 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:17015 (17.0 KB) TX bytes:42513 (42.5 KB)
The /etc/network/ interface seems to have the correct config though.
# The primary network interface
auto enp4s0
iface enp4s0 inet static
address 10.87.140.198
netmask 255.255.224.0
gateway 10.87.159.254