[VRF] juju bootstrap io timeout
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Triaged
|
Wishlist
|
Joseph Phillips |
Bug Description
Hello,
When bootstrapping juju onto a VRF enabled machine, when bootstrapping - the juju agent script will try to configure itself on the default VRF. This is a problem because it will fail to connect to some internal services due to 127.0.0.0/8 addresses not being routable
For example take the following setup:
Node 0:
Default VRF ->: 10.10.32.3
MGMT VRF ->: 10.10.33.2
Default VRF routes:
default via 10.10.32.1 dev eth0 proto static
10.10.32.0/24 dev eth0 proto kernel scope link src 10.10.32.2
MGMT VRF Routes:
default via 10.10.33.1 dev eth1 proto static
10.10.33.0/24 dev eth1 proto kernel scope link src 10.10.33.
Cloud:
manual-vrf-test:
type: manual
endpoint: 10.10.33.2
regions:
default: {}
Cloud init script starts and works because the default VRF has a default route so is able to fetch its resources.
I actually had a behaviour here where juju over-wrote my systemd files, breaking my ssh configuration (removing my VRF configuration)
[Configuration]
sshd unit file
[Unit]
Description=OpenBSD Secure Shell server
Documentation=
After=network.
ConditionPathEx
[Service]
EnvironmentFile
ExecStartPre=
ExecStart=
ExecReload=
ExecReload=
KillMode=process
Restart=on-failure
RestartPreventE
Type=notify
RuntimeDirector
RuntimeDirector
[Install]
WantedBy=
Alias=sshd.service
(Overriding didnt work here, as it was not possible to override the ExecStart argument)
VRF Config:
network:
ethernets:
eth0:
- 10.10.32.27/24
routes:
- to: default
via: 10.10.32.1
match:
mtu: 1500
- 10.10.32.2
- vlan32.maas
eth1:
- 10.10.33.2/24
match:
mtu: 1500
- 10.10.32.2
- vlan32.maas
routes:
- to: default
via: 10.10.32.1
vrfs:
mgmt:
table: 14
version: 2
[ss output (Default VRF)]
ss -plnt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:37017 0.0.0.0:*
LISTEN 0 128 [::]:22 [::]:*
LISTEN 0 4096 [::]:37017 [::]:*
[Steps to reproduce]
1. Deploy ubuntu jammy machine
2. Setup VRF
3. Attempt to bootstrap manual controller on this machine
[Error output]
ERROR cannot store binary file: cannot clean up after failed storage operation because: read tcp 127.0.0.
2023-08-14 12:53:05 DEBUG cmd supercommand.go:537 error stack:
read tcp 127.0.0.
github.
github.
github.
github.
github.
github.
2023-08-14 12:53:05 DEBUG juju.cmd.jujud main.go:283 jujud complete, code 0, err <nil>
[Workaround]
Apply VRF after juju has bootstrapped
Thanks,
Peter
summary: |
- [VRF] juju bootstrap timeout - juju + [VRF] juju bootstrap timeout - juju agent starts on wrong VRF |
description: | updated |
summary: |
- [VRF] juju bootstrap timeout - juju agent starts on wrong VRF + [VRF] juju bootstrap io timeout - juju agent starts on wrong VRF |
description: | updated |
tags: | added: vrf |
description: | updated |
description: | updated |
summary: |
- [VRF] juju bootstrap io timeout - juju agent starts on wrong VRF + [VRF] juju bootstrap io timeout |
Interestingly, this would probably also fail the way that we currently have Dqlite working on our "main" branch, earmarked for Juju 4.0.
We will bind Dqlite to a local-cloud address *if there is a unique one available*, otherwise falling back to the loopback IP and rebinding when we enter HA.
We have plans to move some behaviour into the controller charm, whereby we could use a cluster peer relation and bindings to allow this unique selection from the outset.
I doubt we will address this on the current 3.x major, but we will keep it in mind as we progress.