I am adding three pre-built maas nodes to a juju model via `juju add-machine --debug ssh:ubuntu@<IP>`.
The first and third machines were added successfully, but the second one has been in pending state for several hours.
Removing the machine and readding it does not help.
No messages related to that machine are ever printed in the debug-log of the model they're being added to.
Adding --debug to the command shows that ssh authentication succeeds, but nothing further happens (for comparison, on the other machines it took about 2 minutes to go from 'Detecting series and characteristics' to completion).
$ juju add-machine --debug ssh:ubuntu@100.100.184.17
08:11:59 INFO juju.cmd supercommand.go:57 running juju [2.6.9 gc go1.10.4]
08:11:59 DEBUG juju.cmd supercommand.go:58 args: []string{"/snap/juju/9102/bin/juju", "add-machine", "--debug", "ssh:ubuntu@100.100.184.17"}
08:11:59 INFO juju.juju api.go:67 connecting to API addresses: [100.100.185.33:17070 100.100.185.104:17070 100.100.185.23:17070]
08:11:59 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://100.100.185.104:17070/model/5ba2a092-c7ec-4fd2-8d90-70511fe9bf00/api"
08:11:59 INFO juju.api apiclient.go:624 connection established to "wss://100.100.185.104:17070/model/5ba2a092-c7ec-4fd2-8d90-70511fe9bf00/api"
08:11:59 INFO juju.cmd.juju.machine add.go:246 load config
08:11:59 INFO juju.juju api.go:67 connecting to API addresses: [100.100.185.104:17070 100.100.185.33:17070 100.100.185.23:17070]
08:11:59 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://100.100.185.104:17070/model/5ba2a092-c7ec-4fd2-8d90-70511fe9bf00/api"
08:11:59 INFO juju.api apiclient.go:624 connection established to "wss://100.100.185.104:17070/model/5ba2a092-c7ec-4fd2-8d90-70511fe9bf00/api"
08:11:59 INFO juju.juju api.go:303 API endpoints changed from [100.100.185.104:17070 100.100.185.23:17070 100.100.185.33:17070] to [100.100.185.104:17070 100.100.185.33:17070 100.100.185.23:17070]
08:11:59 INFO cmd authkeys.go:114 Adding contents of "/home/jujumanage/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
08:11:59 INFO cmd authkeys.go:114 Adding contents of "/home/jujumanage/.ssh/id_rsa.pub" to authorized-keys
08:11:59 INFO juju.environs.manual.sshprovisioner sshprovisioner.go:43 initialising "100.100.184.17", user "ubuntu"
08:11:59 DEBUG juju.utils.ssh ssh.go:305 using OpenSSH ssh client
08:12:00 INFO juju.environs.manual.sshprovisioner sshprovisioner.go:54 ubuntu user is already initialised
08:12:00 INFO juju.environs.manual.sshprovisioner sshprovisioner.go:167 Checking if 100.100.184.17 is already provisioned
08:12:00 DEBUG juju.utils.ssh ssh.go:305 using OpenSSH ssh client
08:12:01 INFO juju.environs.manual.sshprovisioner sshprovisioner.go:102 Detecting series and characteristics on 100.100.184.17
08:12:01 DEBUG juju.utils.ssh ssh.go:305 using OpenSSH ssh client
08:12:02 INFO juju.environs.manual.sshprovisioner sshprovisioner.go:158 series: bionic, characteristics: arch=amd64 cores=40 mem=385606M
I don't see any relevant process in the process table of 100.100.184.17 that would suggest juju is doing anything to the machine.
Neither /var/lib/juju nor /var/log/juju ever get created.
The only difference I can see between this stuck machine and the others is that the one that is having problems is currently the pgsql master and is holding both the maas and the postgres VIPs. I don't however see how that could be relevant since 100.100.184.17 is the private address of that machine, not a VIP.
How can I gather more information about what juju is doing on 100.100.184.17?
See if there is anything pertinent in:
- /var/log/
- /var/log/