Copying over haproxy-internal.pem task fails. The haproxy-internal.pem is not created with `kolla-ansible certificates`.

[r3ap3r-d3v]

1. Copying over haproxy-internal.pem task fails. It appears that the haproxy-internal.pem is not created by the `certificates` playbook?

2. The `kolla-ansible certificates` should generate all required certificate files for running tls with a kolla-ansible deployment.

3. Reproduce by going through kolla-ansible quickstart guide, run `kolla-ansible certificates` after making appropriate configurations in globals.yml, then attempt to finish out the deployment. The failure occurs during the `kolla-ansible deploy` playbook.

Environment:
OS: CentOS 7.7(1908)
Kernel: Linux openstack.local.lan 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Docker version: 19.03.6
Kolla-Ansible Version: 9.0.1 (Train)
Docker Image Install Type: Source
Docker Image Distribution: CentOS
I am using official images.

Below is the output from the "deploy" playbook:

TASK [haproxy : Copying over haproxy-internal.pem] *******************************************************************************************************************************************
task path: /home/osadmin/openstack/share/kolla-ansible/ansible/roles/haproxy/tasks/config.yml:128
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: osadmin
<localhost> EXEC /bin/sh -c 'echo ~osadmin && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/osadmin/.ansible/tmp/ansible-tmp-1582154731.45-75981622065221 `" && echo ansible-tmp-1582154731.45-75981622065221="` echo /home/osadmin/.ansible/tmp/ansible-tmp-1582154731.45-75981622065221 `" ) && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/osadmin/.ansible/tmp/ansible-tmp-1582154731.45-75981622065221/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/home/osadmin/openstack/lib/python2.7/site-packages/ansible/plugins/action/copy.py", line 464, in run
    source = self._find_needle('files', source)
  File "/home/osadmin/openstack/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 1178, in _find_needle
    return self._loader.path_dwim_relative_stack(path_stack, dirname, needle)
  File "/home/osadmin/openstack/lib/python2.7/site-packages/ansible/parsing/dataloader.py", line 319, in path_dwim_relative_stack
    raise AnsibleFileNotFound(file_name=source, paths=[to_native(p) for p in search])
AnsibleFileNotFound: Could not find or access '/etc/kolla/certificates/haproxy-internal.pem' on the Ansible Controller.
If you are using a module and expect the file to exist on the remote, see the remote_src option
failed: [localhost] (item=haproxy-internal.pem) => {
    "ansible_loop_var": "item",
    "changed": false,
    "invocation": {
        "dest": "/etc/kolla/haproxy/haproxy-internal.pem",
        "mode": "0660",
        "module_args": {
            "dest": "/etc/kolla/haproxy/haproxy-internal.pem",
            "mode": "0660",
            "src": "/etc/kolla/certificates/haproxy-internal.pem"
        },
        "src": "/etc/kolla/certificates/haproxy-internal.pem"
    },
    "item": "haproxy-internal.pem",
    "msg": "Could not find or access '/etc/kolla/certificates/haproxy-internal.pem' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"
}

PLAY RECAP ***********************************************************************************************************************************************************************************
localhost : ok=54 changed=0 unreachable=0 failed=1 skipped=12 rescued=0 ignored=0

Below is the output from the "certificates" playbook:

Generate TLS Certificates : ansible-playbook -i /home/osadmin/openstack/share/kolla-ansible/ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla /home/osadmin/openstack/share/kolla-ansible/ansible/certificates.yml
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation.
This feature will be removed in version 2.10. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

PLAY [Apply role certificates] ***************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************
ok: [localhost]

TASK [certificates : include_tasks] **********************************************************************************************************************************************************
included: /home/osadmin/openstack/share/kolla-ansible/ansible/roles/certificates/tasks/generate.yml for localhost

TASK [certificates : Ensuring config directories exist] **************************************************************************************************************************************
ok: [localhost]

TASK [certificates : Creating SSL configuration file] ****************************************************************************************************************************************
ok: [localhost] => (item=openssl-kolla.cnf)

TASK [certificates : Creating Key] ***********************************************************************************************************************************************************
ok: [localhost] => (item=/etc/kolla/certificates/private/haproxy.key)

TASK [certificates : Setting permissions on key] *********************************************************************************************************************************************
ok: [localhost]

TASK [certificates : Creating Server Certificate] ********************************************************************************************************************************************
ok: [localhost] => (item=/etc/kolla/certificates/private/haproxy.crt)

TASK [certificates : Creating CA Certificate File] *******************************************************************************************************************************************
ok: [localhost]

TASK [certificates : Creating Server PEM File] ***********************************************************************************************************************************************
changed: [localhost]

PLAY RECAP ***********************************************************************************************************************************************************************************
localhost : ok=9 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

I also checked the https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/certificates/tasks/generate.yml and it also doesn't reference the "haproxy-internal.pem" anywhere that I could find. If any further information is required, please feel free to let me know.

[Marcin Juszkiewicz (hrw)]

Please try with this patch applied.

[r3ap3r-d3v (r3ap3r-d3v)]

I added the above patch and now I appear to be failing at a different point. Below is what I am getting now. Please forgive me, trying to do this from my phone right now.

TASK [haproxy : Fail if internal haproxy certificate is absent] ****************
task path: /home/osadmin/openstack/share/kolla-ansible/ansible/roles/haproxy/tas
ks/precheck.yml:101
fatal: [localhost -> localhost]: FAILED! => {
    "changed": false,
    "msg": "Internal haproxy certificate file is not found. It is configured via
 'kolla_internal_fqdn_cert'"
}

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
localhost : ok=28 changed=0 unreachable=0 failed=1 s
kipped=12 rescued=0 ignored=0

Command failed ansible-playbook -i all-in-one -e @/etc/kolla/globals.yml -e @/et
c/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=precheck /home/o
sadmin/openstack/share/kolla-ansible/ansible/site.yml --verbose --verbose --ver
bose
(openstack) [osadmin@openstack

[r3ap3r-d3v (r3ap3r-d3v)]

I forgot to mention, the above is failing in precheck, haven't even made it the deploy playbook yet.

[Mark Goddard (mgoddard)]

The certificates role in the train release does not support generating certificates for TLS on the internal API network - only the external. Support has been added on the master branch and will be included in the ussuri release.

In general, this command is not recommended for production as it generates self-signed certificates.

[r3ap3r-d3v (r3ap3r-d3v)]

Understood. Just to clarify, I'm still not in "production" with Kolla yet. For production I will be using Let's Encrypt certificates and not self-signed. So what you are saying is that I need to set the internal certificate option in globals to "no" and try with only external enabled?

[Mark Goddard (mgoddard)]

That's correct. The internal and external networks need to be different for it to work.

[r3ap3r-d3v (r3ap3r-d3v)]

Ok, tls is now working on my external vip. I applied the patch provided by hrw to my ansible/roles/haproxy/tasks/config.yml file. I had to then set 'kolla_enable_tls_internal: "no"' (this is the default, I changed it trying to get tls to work) and then change 'kolla_enable_tls_external: "{{ kolla_enable_tls_internal if kolla_same_external_internal_vip | bool else 'no' }}"' to 'kolla_enable_tls_external: "yes"'. After that, I ran the deploy again and it succeeded without failing.

Note that as of the 6th of March, 2020 the 'kolla-ansible certificates' command for Train doesn't generate the appropriate certificates for internal use hence why we have to make the 'kolla_enable_tls' adjustments shown above. If you are using Let's Encrypt or another CA, you shouldn't experience this issue.

[Mark Goddard (mgoddard)]

I don't think hrw's patch is necessary. The play operates against localhost, so remote == local.

[Radosław Piliszek (yoctozepto)]

Seconding Mark.

[r3ap3r-d3v (r3ap3r-d3v)]

Ok, not sure what else would have changed outside of that? Maybe the issue had been resolved some other way that I haven't seen? I don't like that answer but I'm not having the issue after a clean install of everything so I'm not sure what else to "troubleshoot" at this point? I have a smooth running Stack deployed by Kolla and everything I have is working as intended so far.