juju bootstrap with ca-cert-path and ca-private-key-path not using assigned cert/key

[Jeff Hillman]

juju 3.1.6
Focal 20.04.6

Whenrunning a local bootstrap pointing to use a local set of CA certs/keys with a decrypted key, juju doesn't utilize that CA/key to generate a certificate against the pair.

A juju-ca issuer is still utilized.

juju bootstrap debug output:

```
$ juju bootstrap maas maas --bootstrap-series focal --config ca-cert-path=./controller_ca.pem --config ca-private-key-path=./decrypt.pem --debug
02:09:36 INFO juju.cmd supercommand.go:56 running juju [3.1.6 f6a66aa91eec620f5ac04a19d8c06bef03ae6228 gc go1.20.8]
02:09:36 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/24626/bin/juju", "bootstrap", "maas", "maas", "--bootstrap-series", "focal", "--config", "ca-cert-path=./controller_ca.pem", "--config", "ca-private-key-path=./decrypt.pem", "--debug"}
02:09:36 DEBUG juju.cmd.juju.commands bootstrap.go:1392 authenticating with region "" and credential "maas" ()
02:09:36 DEBUG juju.cmd.juju.commands bootstrap.go:1551 provider attrs: map[]
02:09:36 INFO cmd authkeys.go:113 Adding contents of "/home/ubuntu/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
02:09:36 DEBUG juju.cmd.juju.commands bootstrap.go:1621 preparing controller with config: map[agent-metadata-url: agent-stream:released apt-ftp-proxy: apt-http-proxy: apt-https-proxy: apt-mirror: apt-no-proxy: authorized-keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDTsBz0GmLHKL5fEZsJ87axUqxETiAnb6Pig3lG9WEkoxzop6KzMQ3RUSGWhGmfRbsoK/+ShiHvyILS2LD5yuJsKvc+aqP4h6pZ+HuPtwiACzCHxpKqSptwGkW/MilInmmoyhy3Ei4A4Oqe61ixcFxv3Sb7CIxrYz0XujVg1R4uYOqLKKhEZbHHISi8W0oEA91MUKk3NNJ/UjQtiyWYhsl2Rz08CkFJu9jR6PhsS4qz/H1uuUszDAN8fz+ANwHJVQKca0aOjpD0uuIINcdl9RNH3ZTTikC9YuK3xHkW7ivnl2WPLlgiOEIvuny5X/wtVkNaNR76eOf6L0ODZYnYxwtacnjNQo5DYnhhIbW8W+HePc2NbenGAIEQXz/B9qxxWkDJW4BI/A6HY3ML8wvWL//F8sp4+7fIv3KOlQVKQiCncep8R3tAGj1w9kYFZw7vX9+nM33yosW6ByAxduevITl+EvF8cHqQ6aKw/Zy7IvuHT7YIVNhlGYmN0zRsnsg/6UoVsqbCJfxDGML9TTpAWC8KBvGqOQXXp2kE4T1sX5/LpLJYRu2eQfw/ICFHZh3CtgBsZJ/aVuwKJtXJRLHxU7BIJOEaAMpnR0EdNLS4T8kqqjfI7mAHBpZGABRkk0hMgDKfPFSnTzAuSZ38G9mNREbXCDXlwhivYDahu4C+6uTVNQ== juju-client-key
 automatically-retry-hooks:true backup-dir: ca-cert-path:./controller_ca.pem ca-private-key-path:./decrypt.pem charmhub-url:https://api.charmhub.io cloudinit-userdata: container-image-metadata-url: container-image-stream:released container-inherit-properties: container-networking-method: default-base: default-space: development:false disable-network-management:false disable-telemetry:false egress-subnets: enable-os-refresh-update:true enable-os-upgrade:true fan-config: firewall-mode:instance ftp-proxy: http-proxy: https-proxy: ignore-machine-addresses:false image-metadata-url: image-stream:released juju-ftp-proxy: juju-http-proxy: juju-https-proxy: juju-no-proxy:127.0.0.1,localhost,::1 logforward-enabled:false logging-config: logging-output: lxd-snap-channel:5.0/stable max-action-results-age:336h max-action-results-size:5G max-status-history-age:336h max-status-history-size:5G mode: name:controller net-bond-reconfigure-delay:17 no-proxy:127.0.0.1,localhost,::1 num-container-provision-workers:4 num-provision-workers:16 provisioner-harvest-mode:destroyed proxy-ssh:false resource-tags: secret-backend:auto snap-http-proxy: snap-https-proxy: snap-store-assertions: snap-store-proxy: snap-store-proxy-url: ssl-hostname-verification:true test-mode:false transmit-vendor-metrics:true type:maas update-status-hook-interval:5m uuid:3eea9ec8-8c3b-4e3c-8868-97921da77ca1]
02:09:36 DEBUG juju.provider.maas environprovider.go:63 opening model "controller".
02:09:36 WARN juju.environs.config config.go:1908 unknown config field "ca-private-key-path"
02:09:36 WARN juju.environs.config config.go:1908 unknown config field "ca-cert-path"
02:09:36 WARN juju.environs.config config.go:1908 unknown config field "ca-private-key-path"
02:09:36 WARN juju.environs.config config.go:1908 unknown config field "ca-cert-path"
02:09:36 INFO cmd bootstrap.go:939 Creating Juju controller "maas" on maas/default
02:09:36 INFO juju.cmd.juju.commands bootstrap.go:1010 combined bootstrap constraints:
02:09:36 DEBUG juju.environs.bootstrap bootstrap.go:319 model "controller" supports application/machine networks: true
02:09:36 DEBUG juju.environs.bootstrap bootstrap.go:321 network management by juju enabled: true
02:09:36 INFO cmd bootstrap.go:406 Loading image metadata
02:09:36 INFO cmd bootstrap.go:479 Looking for packaged Juju agent version 3.1.6 for amd64
02:09:36 INFO juju.environs.bootstrap tools.go:78 looking for bootstrap agent binaries: version=3.1.6
02:09:36 DEBUG juju.environs.tools tools.go:87 finding agent binaries in stream: "released"
02:09:36 DEBUG juju.environs.tools tools.go:89 reading agent binaries with major.minor version 3.1
02:09:36 DEBUG juju.environs.tools tools.go:98 filtering agent binaries by version: 3.1.6
02:09:36 DEBUG juju.environs.tools tools.go:101 filtering agent binaries by os type: ubuntu
02:09:36 DEBUG juju.environs.tools tools.go:104 filtering agent binaries by architecture: amd64
02:09:36 DEBUG juju.environs.tools urls.go:133 trying datasource "keystone catalog"
02:09:36 DEBUG juju.environs.simplestreams simplestreams.go:417 searching for signed metadata in datasource "default simplestreams"
02:09:36 DEBUG juju.environs.simplestreams simplestreams.go:451 looking for data index using path streams/v1/index2.sjson
02:09:38 DEBUG juju.environs.simplestreams simplestreams.go:747 using default candidate for content id "com.ubuntu.juju:released:agents" are {20210329 mirrors:1.0 content-download streams/v1/cpc-mirrors-agents.sjson []}
02:09:38 DEBUG juju.environs.simplestreams simplestreams.go:463 looking for data index using URL https://streams.canonical.com/juju/tools/streams/v1/index2.sjson
02:09:38 DEBUG juju.environs.simplestreams simplestreams.go:486 read metadata index at "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson"
02:09:38 DEBUG juju.environs.simplestreams simplestreams.go:1018 finding products at path "streams/v1/com.ubuntu.juju-released-agents.sjson"
02:09:38 INFO juju.environs.bootstrap tools.go:80 found 1 packaged agent binaries
02:09:38 INFO cmd bootstrap.go:492 Located Juju agent version 3.1.6-ubuntu-amd64 at https://streams.canonical.com/juju/tools/agent/3.1.6/juju-3.1.6-linux-amd64.tgz
02:09:38 WARN juju.environs.config config.go:1908 unknown config field "ca-private-key-path"
02:09:38 WARN juju.environs.config config.go:1908 unknown config field "ca-cert-path"
02:09:38 WARN juju.environs.config config.go:1908 unknown config field "ca-private-key-path"
02:09:38 WARN juju.environs.config config.go:1908 unknown config field "ca-cert-path"
02:09:38 INFO cmd bootstrap.go:590 Starting new instance for initial controller
02:09:39 INFO cmd bootstrap.go:184 Launching controller instance(s) on maas/default...
02:09:39 DEBUG juju.provider.maas environ.go:685 attempting to acquire node in zone "default"
02:09:39 DEBUG juju.cloudconfig.instancecfg instancecfg.go:945 Setting numa ctl preference to false
02:09:39 DEBUG juju.provider.maas environ.go:769 maas user data; 3720 bytes
02:09:43 DEBUG juju.provider.maas environ.go:795 started instance "eghprm"
02:09:43 INFO cmd bootstrap.go:322 - eghprm (arch=amd64 mem=4G cores=2)
02:09:43 INFO juju.environs.bootstrap bootstrap.go:1010 newest version: 3.1.6
02:09:43 INFO juju.environs.bootstrap bootstrap.go:1025 picked bootstrap agent binary version: 3.1.6
02:09:43 INFO cmd bootstrap.go:633 Installing Juju agent on bootstrap instance
02:12:20 DEBUG juju.cloudconfig.instancecfg instancecfg.go:945 Setting numa ctl preference to false
Waiting for address
02:12:20 DEBUG juju.provider.maas instance.go:88 "juju" has addresses ["local-cloud:192.168.122.13@undefined(id:-1)"]
Attempting to connect to 192.168.122.13:22
02:12:21 DEBUG juju.provider.common bootstrap.go:668 connection attempt for 192.168.122.13 failed: /var/lib/juju/nonce.txt does not exist
02:12:26 DEBUG juju.provider.common bootstrap.go:668 connection attempt for 192.168.122.13 failed: /var/lib/juju/nonce.txt does not exist
02:12:30 DEBUG juju.provider.maas instance.go:88 "juju" has addresses ["local-cloud:192.168.122.13@undefined(id:-1)"]
02:12:32 DEBUG juju.provider.common bootstrap.go:668 connection attempt for 192.168.122.13 failed: /var/lib/juju/nonce.txt does not exist
02:12:37 INFO cmd bootstrap.go:436 Connected to 192.168.122.13
02:12:37 INFO juju.cloudconfig userdatacfg_unix.go:598 Fetching agent: curl -sSf --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/agent/3.1.6/juju-3.1.6-linux-amd64.tgz]>
02:12:37 INFO cmd bootstrap.go:506 Running machine configuration script...
02:15:35 INFO cmd bootstrap.go:744 Bootstrap agent now started
02:15:36 DEBUG juju.provider.maas instance.go:88 "juju" has addresses ["local-cloud:192.168.122.13@undefined(id:-1)"]
02:15:36 INFO juju.juju api.go:354 API endpoints changed from [] to [192.168.122.13:17070]
02:15:36 INFO cmd controller.go:88 Contacting Juju controller at 192.168.122.13 to verify accessibility...
02:15:36 INFO juju.juju api.go:86 connecting to API addresses: [192.168.122.13:17070]
02:15:38 DEBUG juju.api apiclient.go:1171 successfully dialed "wss://192.168.122.13:17070/model/3eea9ec8-8c3b-4e3c-8868-97921da77ca1/api"
02:15:38 INFO juju.api apiclient.go:706 connection established to "wss://192.168.122.13:17070/model/3eea9ec8-8c3b-4e3c-8868-97921da77ca1/api"
02:15:38 DEBUG juju.api monitor.go:35 RPC connection died
02:15:38 INFO cmd controller.go:108
Bootstrap complete, controller "maas" is now available
Controller machines are in the "controller" model
02:15:38 INFO cmd bootstrap.go:658
Now you can run
 juju add-model <model-name>
to create a new model to deploy workloads.
02:15:38 INFO cmd supercommand.go:535 command finished
```

openssl output from newly generated cert:

```$ openssl s_client -showcerts -connect 192.168.122.13:17070
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 O = Juju, CN = juju-ca
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 O = Juju, CN = juju-ca
verify return:1
depth=0 O = Juju, CN = Juju server certificate - controllerip, serialNumber = 3d3bee3c-10b7-4e03-8696-ce3fca64b803
verify return:1
---
Certificate chain
 0 s:O = Juju, CN = Juju server certificate - controllerip, serialNumber = 3d3bee3c-10b7-4e03-8696-ce3fca64b803
   i:O = Juju, CN = juju-ca
-----BEGIN CERTIFICATE-----
MIIEUzCCArugAwIBAgIVAMSE9KN5mL18zFdcD4E4Z8y46TYQMA0GCSqGSIb3DQEB
CwUAMCExDTALBgNVBAoTBEp1anUxEDAOBgNVBAMTB2p1anUtY2EwHhcNMjMxMDEx
MDIxMDM2WhcNMzMxMDExMDIxNTM2WjBvMQ0wCwYDVQQKEwRKdWp1MS8wLQYDVQQD
EyZKdWp1IHNlcnZlciBjZXJ0aWZpY2F0ZSAtIGNvbnRyb2xsZXJpcDEtMCsGA1UE
BRMkM2QzYmVlM2MtMTBiNy00ZTAzLTg2OTYtY2UzZmNhNjRiODAzMIIBojANBgkq
hkiG9w0BAQEFAAOCAY8AMIIBigKCAYEA05qrGU2EOlktpy1hpwY6G94Bs0kBxohN
pabCiBuvlCO77S0fK/N58MvmaPrKrXEs4e0fukBQf+MQnf6j/05J2+oE+EG0nqbv
s8vwAXcj71yeTHC8AxOv7LbR0XySy2mDsiou8o4Kvz6NgyvrU2VTEbPnPGhhx5Mi
iIB+/giCt69am0HbMkF+P+1F21YlVeSVqTiaxU4XmmtwqRKZuaiBWbwoIGPE3yJH
fG5KGhfGamFbKKJj3iMwiyG/BZIu3kW8fakyxkE0++pyBlVaxKjK0VcvWDxCvPri
if0Q+PEW0x9VRTiiWg0ACS7SrDyWk+JodVq4uSd+v11KvpzqzuU7SQIPJebAuSFV
+pcIOndEmv41X5Tdtcvq1NPbN7F9/AM1ltK0ck94guY9lJKQEnKyOdf+uo3DzFKr
uKrASaq9iw1/FmLy7TYfsuKNH+s2YW35gBGXaczB0ftR3ZmpD/zw0EdIH3bQEmne
WG0nkKKFX16FZ0dXqMazt1uUb3oKDJ9xAgMBAAGjNDAyMB8GA1UdIwQYMBaAFEWl
r6Ab45m7IzD/5t57F+5pE2QpMA8GA1UdEQQIMAaHBMCoeg0wDQYJKoZIhvcNAQEL
BQADggGBAJZrTMkBbyRKeDRx4XeB9ouUIxQIudVkUesEHgg3aqPwgStyOLp6nq1e
wkvsotGYKAZhUNkPf+Rt4VuT4JO2691JXFaSfJc1VhdIcxE5BknOhwzxkfLaufSX
7NRngcqMoJWKP8PQ2dv3xqwpb77IMXyJuKzMmD1gKUXPbFeCKHRPh81OdWMcA25C
RsQrVwxelrFksneubu7GOkuK9C/KHWKCmB1mx2OK2bLRWBD+nV19yxKIi15a4cCA
t2eWSHb3BhodlgdRFet5EqyFlJB3SOhXGceNLj/GKZ5jSnxJqKDsyDZ3q9VVkfAQ
pY4SA8f7TaxzlVtb/MYu0KHkrbBM4sSpdn8pAt86iZYpibjUD6SPNjxbRdviY7zG
Y1swDCpdriHLTDkn/9QmuUqNCU9VcoIIkFahwwHcVH5ZlRXO6aNts74RBtPcMMGi
/Xbdmt16LRbiF9g150RmJI1/VA48ZEgbHwA8b3k1+IJaXxSzPPyNiaN0l0SC+9N4
R3RVYPFrvA==
-----END CERTIFICATE-----
 1 s:O = Juju, CN = juju-ca
   i:O = Juju, CN = juju-ca
-----BEGIN CERTIFICATE-----
MIIEEjCCAnqgAwIBAgIUVLlMrIoXMF5lx1/dVMANWRoWNJAwDQYJKoZIhvcNAQEL
BQAwITENMAsGA1UEChMESnVqdTEQMA4GA1UEAxMHanVqdS1jYTAeFw0yMzEwMTEw
MjA0MzZaFw0zMzEwMTEwMjA5MzZaMCExDTALBgNVBAoTBEp1anUxEDAOBgNVBAMT
B2p1anUtY2EwggGiMA0GCSqGSIb3DQEBAQUAA4IBjwAwggGKAoIBgQDK4BU2TsKo
L4M2Inv4liaHN14KkK4SXHJNOSTqtGxEPkprkNRlz0BV3ybY/Qo8xDeAqQYCCt2o
SNZW1KqJKCP+FJ5NU7o3ZRD9Weru69y+71C2LCLlAtK0zwqHLLa6J4IWqmjD9kea
efkFV4pfUQA9xtF7+SY9AdGfPOx8aFeCiKRMgP+HZ9/212eoFormA5rdOXEYA1KE
uq1k4Ibfr3lIiwFmzu3j2XCZ++H+VUtO/kFaOzSX9an88a3/s1V8jNz/KB1E4Jwx
FmLduX2+MiBWWl/Sf3LnyHtGEOQ6mfu+gF9deJN1NLwSOVnECMzUFu93wX2A1XMw
Svv4wI41ADE8FQqWy0ZM5QIzQgm2KVzo9OIfC+JEk+5gKnO+tf6w/kDru+bqnkUK
6njiDMxTQBPvyF03bS5fYlC5JrS2vmgO08Bg48tebo5vl5GeF2/1EdJjU/Lk7XzP
dMplx3NZnLBE1QgxdShTPM+VQfdbXtN1ITI4AgNGDIQVOcto4UNg490CAwEAAaNC
MEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFEWl
r6Ab45m7IzD/5t57F+5pE2QpMA0GCSqGSIb3DQEBCwUAA4IBgQCxV+DBYleAu1oz
VnCG66nc9gZigzz499K9cyk9lor3DDiOF9olC4eYuCscXZQA8b0UXD2oOmle3GlD
7hAXglCFjJWkVHLUhm/ZMgZh8G8KVHvz6WOkkXpke4pF9XOdqfmx09T8Buf0eT32
SHlw9bk/78nmU6PeeZV5leVUiJLW9EVTVd3yZz/RTkeHBnHwAgkiMhJ+wCTk3wHS
2z7nOMWmKV0TyzSANCMyANlK8MaQ0wi2Fuh/YEBoNz96r3tjlDV1/q3n4SiK3G3e
qElTpdrK37GENGo+uKsQMpbgL7FnAhJ5RdZa447BRlaEI3kFD+ziB8om6itxSaK/
AVqIqVylxrjLc98ARoDJ61tC3kAZ4lMvq6MIQHgFCaYMnxN2CQHjHx5z9yXcPuxB
QrN5FS17RMDQ1Vs8NMjfxYnsZGIphhKyukc6ruNtZX0qhdrr3xU+aCRIUNMRTwhP
quQKdcNyEY5rxlczGAjwlFcumjcgVikDY9e++hbP7vx3RJjDfC0=
-----END CERTIFICATE-----
---
Server certificate
subject=O = Juju, CN = Juju server certificate - controllerip, serialNumber = 3d3bee3c-10b7-4e03-8696-ce3fca64b803

issuer=O = Juju, CN = juju-ca

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2863 bytes and written 370 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 3072 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol : TLSv1.3
    Cipher : TLS_AES_128_GCM_SHA256
    Session-ID: B61B5D9C853774D3E0D87FD523F4B8582F4657C7443E521A19925DD746650AC8
    Session-ID-ctx:
    Resumption PSK: 91CB1A72A959F9F8B60733B0DC701602596ABA65235F541569DFCEB007E0CF64
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - 8a 5c 38 4a 68 0a c2 ac-47 03 96 3f c5 73 1b 34 .\8Jh...G..?.s.4
    0010 - 69 3f 35 36 56 53 40 e1-41 a8 d8 b2 42 5a 06 46 i?56VS@.A...BZ.F
    0020 - 74 bf 9b 2e cf 26 3a 7a-63 7c 6a f2 d0 94 b3 90 t....&:zc|j.....
    0030 - cb c5 55 de 01 86 44 77-0b e4 fd 24 cc cc 17 c7 ..U...Dw...$....
    0040 - 64 68 2b fd d3 2d b3 02-ec 28 59 65 48 9f ce 4d dh+..-...(YeH..M
    0050 - c2 2d 9c 03 aa 86 5a ad-34 e0 ee 28 50 2f 0f 87 .-....Z.4..(P/..
    0060 - 48 1a a5 8f 56 db e7 1a-17 1f 23 f5 21 04 38 d3 H...V.....#.!.8.
    0070 - 15 .

    Start Time: 1696990592
    Timeout : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
```

[Jeff Hillman (jhillman)]

Possibly unrelated, but when using just ca-cert and ca-private-key without the -path reads the files but fails to bootstrap:

```
~$ juju bootstrap maas maas --bootstrap-series focal --config ca-cert="$(cat ./ca-cert.pem)" --config ca-private-key="$(cat ca-private-key.pem)" --debug
15:42:33 INFO juju.cmd supercommand.go:56 running juju [3.1.6 f6a66aa91eec620f5ac04a19d8c06bef03ae6228 gc go1.20.8]
15:42:33 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/24626/bin/juju", "bootstrap", "maas", "maas", "--bootstrap-series", "focal", "--config", "ca-cert=-----BEGIN CERTIFICATE-----\nMIIDVzCCAj+gAwIBAgIUUKjZYxXOfxfk1pPA4Lu2PmecXK8wDQYJKoZIhvcNAQEL\nBQAwOzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkZMMQ0wCwYDVQQKDARVU0FGMRAw\nDgYDVQQDDAdhbXBob3JhMB4XDTIzMDkyNTE4NTUxMFoXDTMzMDkyMjE4NTUxMFow\nOzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkZMMQ0wCwYDVQQKDARVU0FGMRAwDgYD\nVQQDDAdhbXBob3JhMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArHZt\n2UID0WzK32qyJH7KFtRk7VLQ/FZSOqwQQmjPCk9ffsmVciMqsD02gAluCQyUwOBS\ndPCv52kz95rQfC2LSh1OQYG2GraiF/QFnAEbYF+77W4xAOmgYe77olOxEyWtRD6+\nK8uIov06EROzmnnELdHEua6eCZRIC/W8OIpj7rzThMs4G8vpQam82Xpk1gJMrJws\nFCDUEF8+2intZ6zqL/q5Q6SZ+h0nZSnhmlCw/UTCH12H1ZjI9+W6ZRQ1ZnOHxkvi\nFBjg/SMGnfmId9jrXhEl/FvuY/nhcfC40vW79Vp3ZLdMRyKz4vflDUE4MJm/DIJy\nZyzzv7fkJ+uCgYcB/wIDAQABo1MwUTAdBgNVHQ4EFgQU7OIMjsZCoj2yUBUi/3Fe\nmJg4ZgowHwYDVR0jBBgwFoAU7OIMjsZCoj2yUBUi/3FemJg4ZgowDwYDVR0TAQH/\nBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAOZNLVxaOO3ihc9Ve4yW/tDA9DT7D\nPGr+z/wZGp1bzXF5aN/lW0hNE4VcgrvYkrVJMGRoi9rosZU+M4lN2lK2VaIycw58\n6lGCwugx/HxmmnKji+fv4+OzFI9xuplEAFee+ZL8weJIL7jD0QCrL86K+RPEBXyw\nZtSwR5ZTfoQOxKFvk/I5IKa4piyBOQJzGW/5aorxVC27gwbOzGOhTnwU32B5nCp+\nzVJdhyCMpEtnGiP9M+C3lZD4yBqAeHqLZQ3SpkNs8yIT6ikyCDJshXzdeUQ+yiWB\nRCMRarEEGVdPJXwiJsW2E5XSFPbo4HGxRY8uy47AVgQz06rONS/LMEzgEw==\n-----END CERTIFICATE-----", "--config", "ca-private-key=-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCsdm3ZQgPRbMrf\narIkfsoW1GTtUtD8VlI6rBBCaM8KT19+yZVyIyqwPTaACW4JDJTA4FJ08K/naTP3\nmtB8LYtKHU5BgbYatqIX9AWcARtgX7vtbjEA6aBh7vuiU7ETJa1EPr4ry4ii/ToR\nE7OaecQt0cS5rp4JlEgL9bw4imPuvNOEyzgby+lBqbzZemTWAkysnCwUINQQXz7a\nKe1nrOov+rlDpJn6HSdlKeGaULD9RMIfXYfVmMj35bplFDVmc4fGS+IUGOD9Iwad\n+Yh32OteESX8W+5j+eFx8LjS9bv1Wndkt0xHIrPi9+UNQTgwmb8MgnJnLPO/t+Qn\n64KBhwH/AgMBAAECggEBAJVNcYvEOsnzBS6lj9NIcGuTNEzAhWvuzwE8NFdJpPOo\nA2Qf3+EP0MUCOUxe3YO4AI95o+jY9W/BA9w3ioX2Z7+h3z2WrbAyq+FQwApuauTj\n/C1wV7wShWZDHlVJXuLUm0uZhckkSwXPc5WVGMeC4vNHxqd5JBfm5vhzpJ7OL/Vc\n1tgvUGYv3wKE38Cf/vtFr4Z5/PgJnpq6QAG1rx8kjvrvnVwB+KTJMUFaUeKqR7+I\nyZkUn900Y62D9zTqTHb9NJFw9mCHi47u13uZU4W2y1KOKxPThx0+NzKFCaD1I34w\ntPqf2a+mQauaq32LIpV/zoBaezSeej4YkRn8pCrW5QECgYEA3oTFp31bRLws2eJO\nNI6sMH7OFGbuXE9yIsxSZlVO/XpIYazN+LGYZSOsPGPwApjosQQeoamqTfUz/dqV\n8nSj57ZzOOkJsJZSrB5S+/IYq8Ngki5NFYWdV9FDEEHOu76VqHCLebC2Jm3K+ZzA\nJeUbVNFwbeOI6Dq5LTZC+nYdVwUCgYEAxmmKpubr7TXEox8GXFae1TrBzdAb/EhA\nAxfZ6AGsaHPcrr9FI1iuiCoMNsTn7QMco6bggEwu36Pw9Qdhz5KdnWWWfu2PNOut\nGAh0YMb3ws+QraQxFwYwQzwzhgf9QKsN7TliKk468wa8MVvBLd7CQmEA//hVuDkb\nPCFjhbNbvDMCgYAmyN+mrOvelCoBJDfbY07R+Rg0aCh9wH81X4WxPodRGSJKnBMe\nmN3mKwyXThgEa+CLEhvAs9DyW53fTl06cGgtOBjnP38n31uzkmmGbpEsRxarBBT9\nfleJefkFeWLuSG7PeCZnZIyrMBHj5hV7xW45H3RIQxOW3r1uCvorq7C7fQKBgQDD\nBw0d35B6O0bM...

[John A Meinel (jameinel)]

We worked out the issue is that bash strips the trailing newlines from "--config ca-cert=$(cat ./foo.pem)"
And then juju just concatenates the two files when parsing it, but you end up with:

```
-----BEGIN CERTIFICATE-----
MIIDVzCCAj+gAwIBAgIUUKjZ...
-----END CERTIFICATE----------BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
```

The workaround is to
a) Patch juju with:
diff --git a/pki/authority.go b/pki/authority.go
index 9d812540d5..b255dc9495 100644
--- a/pki/authority.go
+++ b/pki/authority.go
@@ -236,7 +236,10 @@ func NewDefaultAuthorityPem(pemBlock []byte) (*DefaultAuthority, error) {
 // pem ca and key. Returns error if the supplied cert is not a ca or passing of
 // the pem data fails.
 func NewDefaultAuthorityPemCAKey(caPem, keyPem []byte) (*DefaultAuthority, error) {
- return NewDefaultAuthorityPem(append(caPem, keyPem...))
+ combined := append(caPem, []byte("\n\n")...)
+ combined = append(combined, keyPem...)
+ combined = append(combined, []byte("\n\n")...)
+ return NewDefaultAuthorityPem(combined)
 }

or
b) insert a newline at the *start* of the certificate file (it seems bash strips trailing newlines, but not prefixed ones)

c) create a config.yaml with the fields filled out:
$ cat ./config.yaml
ca-cert: |
  -----BEGIN CERTIFICATE-----
  MIIFqTCCA5GgAwIBAgIUVDF76nUFr9DdDbv8+0CGB3Oxsq8wDQYJKoZIhvcNAQEL
...
  gjJGfWMpEPARB6VLJA==
  -----END CERTIFICATE-----
ca-private-key: |
  -----BEGIN PRIVATE KEY-----
  MIIJQAIBADANBgkqhkiG9w0BAQEFAASCCSowggkmAgEAAoICAQC46EMKXPP4xdys
...
  SvhLa0tEpRcPUvIk6UVTPiWJ290=
  -----END PRIVATE KEY-----

$ juju bootstrap lxd --debug --config ./config.yaml

$ openssl s_client 10.10.30.62:17070
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 C = US, ST = FL, L = Gainesville, O = Internet Widgits Pty Ltd, CN = meinel
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 C = US, ST = FL, L = Gainesville, O = Internet Widgits Pty Ltd, CN = meinel
...

[Jeff Hillman (jhillman)]

The newline goes above the key, not the cert.

On Wed, Oct 11, 2023, 12:56 PM John A Meinel <email address hidden>
wrote:

> We worked out the issue is that bash strips the trailing newlines from
> "--config ca-cert=$(cat ./foo.pem)"
> And then juju just concatenates the two files when parsing it, but you end
> up with:
>
> ```
> -----BEGIN CERTIFICATE-----
> MIIDVzCCAj+gAwIBAgIUUKjZ...
> -----END CERTIFICATE----------BEGIN PRIVATE KEY-----
> ...
> -----END PRIVATE KEY-----
> ```
>
> The workaround is to
> a) Patch juju with:
> diff --git a/pki/authority.go b/pki/authority.go
> index 9d812540d5..b255dc9495 100644
> --- a/pki/authority.go
> +++ b/pki/authority.go
> @@ -236,7 +236,10 @@ func NewDefaultAuthorityPem(pemBlock []byte)
> (*DefaultAuthority, error) {
> // pem ca and key. Returns error if the supplied cert is not a ca or
> passing of
> // the pem data fails.
> func NewDefaultAuthorityPemCAKey(caPem, keyPem []byte)
> (*DefaultAuthority, error) {
> - return NewDefaultAuthorityPem(append(caPem, keyPem...))
> + combined := append(caPem, []byte("\n\n")...)
> + combined = append(combined, keyPem...)
> + combined = append(combined, []byte("\n\n")...)
> + return NewDefaultAuthorityPem(combined)
> }
>
>
> or
> b) insert a newline at the *start* of the certificate file (it seems bash
> strips trailing newlines, but not prefixed ones)
>
> c) create a config.yaml with the fields filled out:
> $ cat ./config.yaml
> ca-cert: |
> -----BEGIN CERTIFICATE-----
> MIIFqTCCA5GgAwIBAgIUVDF76nUFr9DdDbv8+0CGB3Oxsq8wDQYJKoZIhvcNAQEL
> ...
> gjJGfWMpEPARB6VLJA==
> -----END CERTIFICATE-----
> ca-private-key: |
> -----BEGIN PRIVATE KEY-----
> MIIJQAIBADANBgkqhkiG9w0BAQEFAASCCSowggkmAgEAAoICAQC46EMKXPP4xdys
> ...
> SvhLa0tEpRcPUvIk6UVTPiWJ290=
> -----END PRIVATE KEY-----
>
> $ juju bootstrap lxd --debug --config ./config.yaml
>
> $ openssl s_client 10.10.30.62:17070
> CONNECTED(00000003)
> Can't use SSL_get_servername
> depth=1 C = US, ST = FL, L = Gainesville, O = Internet Widgits Pty Ltd, CN
> = meinel
> verify error:num=19:self-signed certificate in certificate chain
> verify return:1
> depth=1 C = US, ST = FL, L = Gainesville, O = Internet Widgits Pty Ltd, CN
> = meinel
> ...
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2038974
>
> Title:
> juju bootstrap with ca-cert-path and ca-private-key-path not using
> assigned cert/key
>
> Status in Canonical Juju:
> New
>
> Bug description:
> juju 3.1.6
> Focal 20.04.6
>
> Whenrunning a local bootstrap pointing to use a local set of CA
> certs/keys with a decrypted key, juju doesn't utilize that CA/key to
> generate a certificate against the pair.
>
> A juju-ca issuer is still utilized.
>
> juju bootstrap debug output:
>
> ```
> $ juju bootstrap maas maas --bootstrap-series focal --config
> ca-cert-path=./controller_ca.pem --config
> ca-private-key-path=./decrypt.pem --debug
> 02:09:36 INFO juju.cmd supercommand.go:56 running juju [3.1.6
> f6a66aa91eec620f5ac04a19d8c06bef03ae6228 gc go1.20.8]
> 02:09:36 DEBUG juju.cmd supercommand.go:57 args:
> []stri...

[John A Meinel (jameinel)]

so Thomas said he would investigate
I think some easy fixes are to ensure that there is a '\n' in the cert and key, or just ensuring that when concatenating them we insert a newline.
We also could make sure the '-path' variants are handled correctly (as that makes it easier to use and doesn't depend on how bash interprets $(<file) content.)