Contrail Ocata :: R4.0.2.0 build 61 :: ansible_hostname does not handle "-" in hostname.

[Ritam Gangopadhyay]

lb provisioning failure on a server with id lb-b7s18

Copy cert files onto container fails to copy cert file because "-" is missed out in the filename that is constructed from ansible_hosts

*******************************************************************************
*******************************************************************************

root@smb7s16:/opt/contrail/server_manager/ansible/playbooks/contrail_4_0_2_61_new/playbooks# grep -rn "Copy cert files onto container" -A5
roles/node/tasks/copy_ssl_certs.yml:14:- name: Copy cert files onto container
roles/node/tasks/copy_ssl_certs.yml-15- copy: src={{ item.src}} dest={{ item.dest }}
roles/node/tasks/copy_ssl_certs.yml-16- with_items:
roles/node/tasks/copy_ssl_certs.yml-17- - { src: "{{ ssl_certs_src_dir }}/{{ ansible_hostname|lower }}.pem", dest: "{{ ssl_cert_path }}"}
roles/node/tasks/copy_ssl_certs.yml-18- - { src: "{{ ssl_certs_src_dir }}/{{ ansible_hostname|lower }}-privkey.pem", dest: "{{ ssl_cert_key_path }}" }
roles/node/tasks/copy_ssl_certs.yml-19- when: ssl_certs_supplied
root@smb7s16:/opt/contrail/server_manager/ansible/playbooks/contrail_4_0_2_61_new/playbooks#

*******************************************************************************
*******************************************************************************

root@smb7s16:/opt/contrail/server_manager/ansible/playbooks/contrail_4_0_2_61_new/playbooks# ls -larth /etc/contrail_smgr/puppet/ssl/lbb7s18-privkey.pem
ls: cannot access '/etc/contrail_smgr/puppet/ssl/lbb7s18-privkey.pem': No such file or directory
root@smb7s16:/opt/contrail/server_manager/ansible/playbooks/contrail_4_0_2_61_new/playbooks# ls -larth /etc/contrail_smgr/puppet/ssl/lb-b7s18-privkey.pem
-rw-r--r-- 1 root root 1.7K Oct 5 02:50 /etc/contrail_smgr/puppet/ssl/lb-b7s18-privkey.pem
root@smb7s16:/opt/contrail/server_manager/ansible/playbooks/contrail_4_0_2_61_new/playbooks#

*******************************************************************************
*******************************************************************************

"2017-10-05 07:27:25,427-INFO-sm_ansible_callback.py:43-append(): TASK [node : Copy cert files onto container]"
"2017-10-05 07:27:25,698-INFO-sm_ansible_callback.py:43-append(): skipping: [10.84.29.62]"
"2017-10-05 07:27:25,792-INFO-sm_ansible_callback.py:43-append(): skipping: [10.84.29.66]"
"2017-10-05 07:27:25,938-INFO-sm_ansible_callback.py:43-append(): skipping: [10.84.29.70]"
"2017-10-05 07:27:26,029-INFO-sm_ansible_callback.py:43-append(): skipping: [10.84.29.63]"
"2017-10-05 07:27:26,145-INFO-sm_ansible_callback.py:43-append(): skipping: [10.84.29.71]"
"2017-10-05 07:27:26,223-INFO-sm_ansible_callback.py:43-append(): skipping: [10.84.29.64]"
"2017-10-05 07:27:26,335-INFO-sm_ansible_callback.py:43-append(): skipping: [10.84.29.67]"
"2017-10-05 07:27:26,450-INFO-sm_ansible_callback.py:43-append(): skipping: [10.84.29.72]"
"2017-10-05 07:27:26,554-INFO-sm_ansible_callback.py:43-append(): skipping: [10.84.29.68]"
"2017-10-05 07:27:26,675-INFO-sm_ansible_callback.py:43-append(): skipping: [10.84.29.12]"
"2017-10-05 07:27:26,701-INFO-sm_ansible_callback.py:43-append(): fatal: [10.84.29.73]: FAILED! => (item - None) {"changed": false, "failed": true, "msg": "One or more items failed"}"
"2017-10-05 07:27:26,718-INFO-sm_ansible_callback.py:43-append(): fatal: [10.84.29.73] (item={u'dest': u'/etc/contrailctl/ssl/server-privkey.pem', u'src': u'/etc/contrail_smgr/puppet/ssl/lbb7s18-privkey.pem'}) => {"failed": true, "item": {"dest": "/etc/contrailctl/ssl/server-privkey.pem", "src": "/etc/contrail_smgr/puppet/ssl/lbb7s18-privkey.pem"}, "msg": "Unable to find '/etc/contrail_smgr/puppet/ssl/lbb7s18-privkey.pem' in expected paths."}"
"2017-10-05 07:27:26,722-INFO-sm_ansible_callback.py:43-append(): fatal: [10.84.29.73] (item={u'dest': u'/etc/contrailctl/ssl/server.pem', u'src': u'/etc/contrail_smgr/puppet/ssl/lbb7s18.pem'}) => {"failed": true, "item": {"dest": "/etc/contrailctl/ssl/server.pem", "src": "/etc/contrail_smgr/puppet/ssl/lbb7s18.pem"}, "msg": "Unable to find '/etc/contrail_smgr/puppet/ssl/lbb7s18.pem' in expected paths."}"
"2017-10-05 07:27:26,762-INFO-sm_ansible_callback.py:43-append(): skipping: [10.84.29.13]"
"2017-10-05 07:27:26,808-INFO-sm_ansible_callback.py:43-append(): skipping: [10.84.29.14]"

[Ritam Gangopadhyay (ritam)]

Work around:-

copy the cert and key files with expected name without the "-" in the same path /etc/contrail_smgr/puppet/ssl/

[Ramprakash R (ramprakash)]

I am not able to reproduce the issue.
Can we please have the cluster and server jsons. suspicion is that the server id in the server.json might have been specified without the "-" causing the certificate to be generated without the "-".

[Ritam Gangopadhyay (ritam)]

attached are the cluster and server jsons for your reference the server id in server json that I see is with the "-"

        {
            "base_image_id": "",
            "cluster_id": "b7-prod",
            "contrail": {
                "control_data_interface": "enp0s8"
            },
            "discovered": "false",
            "domain": "juniper.net",
            "email": null,
            "gateway": "10.84.29.254",
            "host_name": "lb-b7s18",
            "id": "lb-b7s18",
            "intf_bond": null,
            "intf_control": null,
            "intf_data": null,
            "ip_address": "10.84.29.73",
            "ipmi_address": "",
            "ipmi_interface": "",
            "ipmi_password": "",
            "ipmi_type": "",
            "ipmi_username": "",
            "last_update": "2017-10-05 11:45:33",
            "mac_address": "08:00:27:A8:70:C2",
            "network": {
                "interfaces": [
                    {
                        "default_gateway": "10.84.29.254",
                        "dhcp": false,
                        "ip_address": "10.84.29.73/24",
                        "mac_address": "08:00:27:a8:70:c2",
                        "name": "enp0s3"
                    },
                    {
                        "ip_address": "192.168.2.251/24",
                        "mac_address": "08:00:27:24:e7:fc",
                        "name": "enp0s8"
                    }
                ],
                "management_interface": "enp0s3",
                "provisioning": "kickstart"
            },
            "package_image_id": "",
            "parameters": {
                "interface_name": "enp0s3"
            },
            "password": "c0ntrail123",
            "provisioned_id": "contrail_4_0_2_61_new",
            "reimaged_id": null,
            "roles": [
                "contrail-lb"
            ],
            "ssh_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqBqkzZbZg5sC9HpJqIdorYWADY3nigSzzcewkKGofSAvQj191mvBMG1l2z9IfeNA/zqPhZMaryW4p3xIOPZb0zHH7oaLzx+ZZARpEbDE/kA++wfGYxMjXzVfrsOqq1H/AVJsLDw2evVT7lt/GNxOB6ELlwEjOdGNM43i4L/p6hJtWv88n4TMjXSdeLk35higQSrlOcj0T6anxxhxWGJuzLRWvA1xl3QLxRITAAmUVa/6tP8NmbfD/yl5gsVw9xIPMVkLg+T/+5aK9Zz72wOqgUcSUvfdkC9kZo66uKH3RNxkh1TE56IF9mmgL48kqg41QKZ4b7ArZ6JzaoKK8k2vZ",
            "static_ip": "N",
            "status": "provision_completed",
            "subnet_mask": "255.255.255.0",
            "tag": {},
            "top_of_rack": {}
        },

[Ramprakash R (ramprakash)]

Not able to reproduce the issue. With "-" in hostname and server-id, it works without errors. I suspect the issue happens when the hostname is different than the server-id configured.

"2017-10-25 06:14:04,860-INFO-sm_ansible_callback.py:43-append(): changed: [192.168.1.26] => (item={u'dest': u'/etc/contrailctl/ssl/ca-cert.pem', u'src': u'/etc/contrail_smgr/puppet/ssl/ca-cert.pem'})"
"2017-10-25 06:14:04,862-INFO-sm_ansible_callback.py:43-append(): changed: [192.168.1.26]"
"2017-10-25 06:14:04,870-INFO-sm_ansible_callback.py:43-append(): TASK [node : Copy cert files onto container]"
"2017-10-25 06:14:04,961-INFO-sm_ansible_callback.py:43-append(): skipping: [192.168.1.23]"
"2017-10-25 06:14:04,971-INFO-sm_ansible_callback.py:43-append(): skipping: [192.168.1.24]"
"2017-10-25 06:14:04,992-INFO-sm_ansible_callback.py:43-append(): skipping: [192.168.1.25]"
"2017-10-25 06:14:05,018-INFO-sm_ansible_callback.py:43-append(): skipping: [192.168.1.27]"
"2017-10-25 06:14:05,607-INFO-sm_ansible_callback.py:43-append(): changed: [192.168.1.26] => (item={u'dest': u'/etc/contrailctl/ssl/server.pem', u'src': u'/etc/contrail_smgr/puppet/ssl/server-6.pem'})"
"2017-10-25 06:14:06,211-INFO-sm_ansible_callback.py:43-append(): changed: [192.168.1.26]"
"2017-10-25 06:14:06,214-INFO-sm_ansible_callback.py:43-append(): changed: [192.168.1.26] => (item={u'dest': u'/etc/contrailctl/ssl/server-privkey.pem', u'src': u'/etc/contrail_smgr/puppet/ssl/server-6-privkey.pem'})"

Closing the bug as "not reproducible". Can be re-opened if seen again.

[Ramprakash R (ramprakash)]

[server-manager-lite] ~ # ls /etc/contrail_smgr/puppet/ssl/
ca-cert.pem ca-cert.srl server-3.pem server-4.csr server-4-privkey.pem server-5.pem server-6.csr server-6-privkey.pem server-7.pem tor/
ca-cert-privkey.pem server-3.csr server-3-privkey.pem server-4.pem server-5.csr server-5-privkey.pem server-6.pem server-7.csr server-7-privkey.pem