add-cloud and bootstrap fails when using an Openstack with a self-signed certificate

[Pen Gale]

I ran into this issue when attempting to add a private Openstack cloud with a self-signed certificate.

juju add-cloud fails when run in interactive mode, claiming that an Openstack isn't running at the given address. A build of juju with more debugging turned on yields the following error (I've changed the ip address and port, to anonymize the error):

--------------------------------------------------------------------

Enter the API endpoint url for the cloud: https://10.12.12.120:5000/v3
request available auth options: failed executing the request https://10.12.12.120:5000/
caused by: Get https://10.12.12.120:5000/: x509: cannot validate certificate for 10.12.12.120 because it doesn't contain any IP SANsIdentityAuthOptions() fetching auth options fetch fail: request available auth options: failed executing the request https://10.12.12.120:5000/
caused by: Get https://10.12.12.120:5000/: x509: cannot validate certificate for 10.12.12.120 because it doesn't contain any IP SANs
19:23:22 DEBUG juju.provider.openstack provider.go:219 provider.Ping() failed with auth options fetching failed
caused by: request available auth options: failed executing the request https://10.12.12.120:5000/
caused by: Get https://10.12.12.120:5000/: x509: cannot validate certificate for 10.12.12.120 because it doesn't contain any IP SANs
Can't validate endpoint: No Openstack server running at https://10.12.12.120:5000/v3

--------------------------------------------------------------------

Googling for the "IP SANs" error suggests that this is a Go error that happens when the CN is not set properly. The server in question returns a subject line for the cert with "CN=10.12.12.120". Something might be getting clobbered before being passed to the Go code that tries to verify the cert.

This error does not occur when adding the cloud with a config.yaml file. The cert has been added to the CA for the machine running juju, and curl and wget commands to https://10.12.12.120:5000/v3 succeed, and return what looks like valid Openstack metadata.

[Heather Lanigan (hmlanigan)]

The novarc file contained a env var that juju doesn't use currently:

OS_CACERT=/home/user/openstack.crt

[Heather Lanigan (hmlanigan)]

during interactive juju add-cloud, juju attempts to find out what identity versions the open stack supports. no open stack credentials are used, or should be needed to do so, equivalent to the 'wget https://10.12.12.120:5000'

@petevg, where you able to bootstrap the open stack cloud once it was added with add-cloud -f yamlfile and creds configured?

[Pen Gale (pengale)]

The workaround hear turns out not to work. We run into the same error when attempting to run "juju bootstrap".

-----------------------------------------------------------------------------------------------
16:54:41 INFO juju.cmd supercommand.go:56 running juju [2.3.8 gc go1.10]
16:54:41 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/4423/bin/juju", "bootstrap", "test_openstack", "--debug"}
16:54:41 DEBUG juju.cmd.juju.commands bootstrap.go:835 authenticating with region "default" and credential "petevg" ()
16:54:41 DEBUG juju.cmd.juju.commands bootstrap.go:963 provider attrs: map[use-openstack-gbp:false policy-target-group: use-floating-ip:false use-default-secgroup:false network: external-network:]
16:54:42 INFO cmd authkeys.go:114 Adding contents of "/home/petevg/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
16:54:42 INFO cmd authkeys.go:114 Adding contents of "/home/petevg/.ssh/id_rsa.pub" to authorized-keys
16:54:42 DEBUG juju.cmd.juju.commands bootstrap.go:1019 preparing controller with config: map[https-proxy: ignore-machine-addresses:false enable-os-upgrade:true ftp-proxy: image-stream:released default-series:xenial container-networking-method: max-action-results-age:336h apt-http-proxy: type:openstack automatically-retry-hooks:true max-status-history-age:336h update-status-hook-interval:5m external-network: logforward-enabled:false container-image-stream:released disable-network-management:false apt-ftp-proxy: logging-config: container-inherit-properties: resource-tags: transmit-vendor-metrics:true authorized-keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO/ayaxgQsEiEEzUkfbRd+wP2+QyMKQyBkzr8/Ceb0WcxFHceOtGX8SeBvkXs2we543Ei6U32nxPautoNJK0xUHDXhwKLlCHGsaoXDLayUpcedxAcxK4PrXDxO4ff0kefRXQCTECc0reMMIdu2yBug4P66yeqzpGf/QJPOsk7Wyw1jb5V/Q+MkHTBjUKaIBL+Q00D1dJgN8kAX2pKnK467Ko7MZ7oBxD/OJZ5dmUVcvnb8uRb8xolCoNOsPo+KdwWZtqpZGO9eIOSUbjL6pzPRlSMSJUCMWivChqhbjOwWLAszWjTx8CxSE0S7W6sPURr3rfvltAE4SYu9lVxGqD9Z juju-client-key
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGnvum+IS4M0AAce7CKpEC46YjmdPKQT8ZbzPST4giqWWpy4ZRxXILaV5sy4Cknm6vq7Ux7ViRCzpmw8KcMswp1WHMjYLS2opY7SyE1ocZOUKqMCb33djv1rmjusKOhEBste4AI9AQGIryBxbc8xhPZ8mU9BAntko3dl5Br7NuU2cqTCgDqXvDZ1fduxoap7dJknCG+ca2jmpLcssx6i3/CgRydHTLrc2c/A0XxEv0xuYuoxNKKM9o9SVJip8vuYV1atkk/EAsHp8QeQNTGl/yDakhUXCkcFymsF9sRAHeaL6MayDcUXnEck8wiq1Ik6CXIKKmtdjcD60b+N823K6eTenI6lzskR4MUzHA6Ww3W9zxFYcVtaesBU1CwPflC+XAfQcg/pdpKDFND8bQMq+NpYZ2reVhs+b1mkmMhvO8vPOSBzB3J6s98Dv65kBFYhEENFT7uk5w8G6+LeOqSiFs/aLVs9fhvGoLBU/SaELtyhE/OCl87g+HlOb5XUI2a6a71Ru9G0jjfsIWYNIeNeHuW1/Ti33fMIUv/cEcZOCeBfeioym8eg4E/acK1b+RAEOzKq1NACnagwmQvQW2338HxQ8Q2Wqy+W9H+Ai115vfUdkm/P9a69mmfGnjCIhXgVj7VneuC/gfxXYCWyiV0inenq8K3aS1jvLLOd+116+0zw== petevg@test-client
 max-status-history-size:5G agent-stream:released apt-no-proxy: apt-mirror: image-metadata-url: apt-https-proxy: uuid:927389b6-b440-4b8c-8b12-a95ab14d8674 policy-target-group: enable-os-refresh-update:true development:false name:controller no-proxy:127.0.0.1,localhost,::1 provisioner-harvest-mode:destroyed use-default-secgroup:false use-openstack-gbp:false max-action-results-size:5G egress-subnets: proxy-ssh:false cloudinit-userdata: ssl-hostname-verification:true http-prox...

[Richard Harding (rharding)]

In speaking with the OpenStack team Juju will look to add a new OS_CACERT cloud config option that can be used to inject that. It'll require changes to the provider and a new release of Juju so it'll be a little bit coming.

For now, we believe that adding the certs to the OS level both on the client and on the bootstrap node will help work around this while we update the OpenStack provider code.

[Heather Lanigan (hmlanigan)]

Adding the ability to create a client with a certificate to goose with pr:
https://github.com/go-goose/goose/pull/63

[Heather Lanigan (hmlanigan)]

https://github.com/juju/juju/pull/8972

Fixes the url validation issue with juju add-cloud.
Allows for a CACert to be added to an OpenStack cloud configuration in the juju clouds.yaml:

clouds:
  openstack:
  type: openstack
  .....
  ca-certificates:
    - |
    -----BEGIN CERTIFICATE-----
    .....
    -----END CERTIFICATE-----

An array of certificates as strings may be specified.

There are two known bugs to be resolved:
1. juju storage functionality still has the x509 auth failure to be resolved against an openstack deployment.
2. juju destroy-controller fails to destroy the controller node due to an x509 auth failure.

[Heather Lanigan (hmlanigan)]

Related to #7

 "juju destroy-controllers fails when using an Openstack with a self-signed certificate" https://bugs.launchpad.net/juju/+bug/1783633

[Heather Lanigan (hmlanigan)]

Related to #7

"openstack storage provider fails when using an Openstack with a self-signed certificate" https://bugs.launchpad.net/juju/+bug/1784030

[Heather Lanigan (hmlanigan)]

adds a cloud certificate to juju add-cloud interactive
https://github.com/juju/juju/pull/9015

[Anastasia (anastasia-macmood)]

Marking as Won't Fix for 2.3 and 2.4 series since we are not planning to make any further releases in these series at this stage.