Cannot create a Jammy-based controller

[Peter Matulis]

I am seeing that a Bionic-based or Focal-based Juju client cannot create a Jammy-based controller:

16:01:58 DEBUG juju.provider.maas maas2instance.go:89 "virt-node-01" has addresses ["local-cloud:10.246.114.37@public-space(id:1)"]
16:01:59 DEBUG juju.provider.common bootstrap.go:647 connection attempt for 10.246.114.37 failed: Unable to negotiate with 10.246.114.37 port 22: no matching host key type found. Their offer: rsa-sha2-512,rsa-sha2-256
16:02:04 DEBUG juju.provider.common bootstrap.go:647 connection attempt for 10.246.114.37 failed: Unable to negotiate with 10.246.114.37 port 22: no matching host key type found. Their offer: rsa-sha2-512,rsa-sha2-256

I have tried configuring /etc/ssh_config to no avail.

[Simon Déziel (sdeziel)]

I can confirm the problem when trying to bootstrap the controller on a LXD host:

$ juju bootstrap --debug --bootstrap-series=jammy --config default-series=jammy localhost overlord
...
13:52:11 DEBUG juju.provider.common bootstrap.go:647 connection attempt for 172.17.40.162 failed: Unable to negotiate with 172.17.40.162 port 22: no matching host key type found. Their offer: rsa-sha2-512,rsa-sha2-256

If at that point, one goes into the forming controller and mangles sshd_config:

cat << EOF >> /etc/ssh/sshd_config

HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
HostKeyAlgorithms rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
EOF
ssh-keygen -A
systemctl restart ssh

The bootstrapping will resume but will run into issues with clashing DSS key:

13:56:10 INFO cmd bootstrap.go:415 Connected to 172.17.40.162
13:56:10 INFO juju.cloudconfig userdatacfg_unix.go:575 Fetching agent: curl -sSfw 'agent binaries from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/agent/2.9.29/juju-2.9.29-linux-arm64.tgz]>
13:56:10 INFO cmd bootstrap.go:485 Running machine configuration script...
Cloud-init v. 22.1-14-g2e17a0d6-0ubuntu1~22.04.5 running 'init-local' at Thu, 05 May 2022 13:52:02 +0000. Up 4.93 seconds.
Cloud-init v. 22.1-14-g2e17a0d6-0ubuntu1~22.04.5 running 'init' at Thu, 05 May 2022 13:52:04 +0000. Up 7.70 seconds.
ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: | eth0 | True | 172.17.40.162 | 255.255.255.0 | global | 00:16:3e:c2:31:a7 |
ci-info: | eth0 | True | fe80::216:3eff:fec2:31a7/64 | . | link | 00:16:3e:c2:31:a7 |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: ++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
ci-info: +-------+-------------+-------------+-----------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-------------+-------------+-----------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 172.17.40.1 | 0.0.0.0 | eth0 | UG |
ci-info: | 1 | 172.17.40.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U |
ci-info: | 2 | 172.17.40.1 | 0.0.0.0 | 255.255.255.255 | eth0 | UH |
ci-info: +-------+-------------+-------------+-----------------+-----------+-------+
ci-info: +++++++++++++...

[Peter Matulis (petermatulis)]

I was trying to create a controller on a KVM-based MAAS node.

[Simon Déziel (sdeziel)]

Right, but KVM or LXD (container) shouldn't matter much if the problem is with SSH'ing to the controller.

[Peter Matulis (petermatulis)]

I just wanted to say that the issue is not confined to a LXD container. That's how I read your initial reply.

[Ian Booth (wallyworld)]

FWIW here's the upstream issue

https://github.com/golang/go/issues/49952

We need to try and find a work around until upstream is fixed.

[Dan Ardelean (danardelean)]

I am running into the same issue trying to create a Jammy based controller but from a Jammy based client as well.
Jammy (client) -> Focal (controller) works fine.

[Harry Pidcock (hpidcock)]

https://github.com/juju/juju/pull/14171