IPv6 not enabled by default

[Kevin Otte]

In ubuntu-20.04-server-cloudimg-amd64-disk-kvm.img provided on cloud-images.ubuntu.com does not have IPv6 enabled by default. This precludes the use of an IPv6 webserver (URL supplied with the SMBIOS option) to feed the NoCloudNet datasource.

I can enable IPv6 using "network-config" on a NoCloud cidata volume, but at that point I might as well just put my user-data in there and not use NoCloudNet. I would prefer to avoid this method.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: cloud-init 20.1-10-g71af48df-0ubuntu5
ProcVersionSignature: Ubuntu 5.4.0-1009.9-kvm 5.4.30
Uname: Linux 5.4.0-1009-kvm x86_64
ApportVersion: 2.20.11-0ubuntu27
Architecture: amd64
CasperMD5CheckResult: skip
CloudName: NoCloud
Date: Thu May 14 15:26:23 2020
PackageArchitecture: all
ProcEnviron:
 TERM=vt220
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install)
logs.tgz: Error: [Errno 2] No such file or directory: '/tmp/cloud-init-logs.tgz'

[Paride Legovini (paride)]

Hi Kevin,

I did some testing and ipv6 seems to work as expected for me. I did the following:

1. `virsh net-edit default` and added:

  <ip family='ipv6' address='2001:db8::1' prefix='64'>
  </ip>

to have IPv6 configured on virbr0. As the prefix is a /64 dnsmasq will automatically send router advertisements on the interface, allowing SLAAC configuration.

2. `virsh net-destroy default`
   `virsh net-start default`
   `ip -c addr show virbr0` <- to check the v6 addr is there

3. Downloaded ubuntu-20.04-server-cloudimg-amd64.img from
   cloud-images.ubuntu.com

4. qemu-img create -f qcow2 -F qcow2 -o backing_file=ubuntu-20.04-server-cloudimg-amd64.img focal-v6test.qcow2

6. Created the following meta-data and user-data files:

$ cat meta-data
instance-id: iid-local01
local-hostname: cloudimg

$ cat user-data
#cloud-config
password: passw0rd
chpasswd: { expire: False }
ssh_pwauth: True

6. Served them over IPv6 with: `python3 -m http.server --bind ::`

7. Deployed the VM with:

virt-install --name focal-v6test --os-variant=ubuntu20.04 --disk path=focal-v6test.qcow2 --import --sysinfo 'system.serial=ds=nocloud-net;s=http://[2001:db8::1]:8000/'

Everything worked as expected: the cloud-init picked up user-data and meta-data via ipv6 and configured the VM accordingly.

If something is not working for you I doubt the cause is IPv6 not being enabled. How are IPv6 addresses assigned on your network?

I'm marking this report as Incomplete for the moment, please set it back to New after commenting back, and we'll look at it again. Thanks!

[Kevin Otte (nivex)]

Executive summary: I expected this test to go much differently. This bug should probably be "cloud-init does not wait for IPv6 config in dual-stack network"

--

One of my initial tests was in my primary dual-stack network, and it also worked similar to your example.

Excerpt from my deploy script:
iid="iid-$(printf "%x" $(date +%s))"

virt-install --name ${vm_name} --memory ${mem_size} \
 --sysinfo system_serial="ds=nocloud-net;h=${vm_name};i=${iid};s=http://gemini.home.nivex.net/cidata/" \
 --controller=scsi,model=virtio-scsi \
 --disk "path=${vdisk},bus=scsi,discard=unmap" \
 --import \
 --os-variant ubuntu18.04 \
 --machine q35 \
 --network network=ovs-net,target=ve0-${vm_name} \
 --graphics none \
 --noautoconsole

cloud-init[321]: Cloud-init v. 20.1-10-g71af48df-0ubuntu5 running 'init' at Tue, 19 May 2020 23:03:21 +0000. Up 6.19 seconds.
cloud-init[321]: ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
cloud-init[321]: ci-info: +--------+-------+----------------------------+---------------+--------+-------------------+
cloud-init[321]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
cloud-init[321]: ci-info: +--------+-------+----------------------------+---------------+--------+-------------------+
cloud-init[321]: ci-info: | enp1s0 | True | 172.31.3.134 | 255.255.255.0 | global | 52:54:00:4d:27:e9 |
cloud-init[321]: ci-info: | enp1s0 | True | fe80::5054:ff:fe4d:27e9/64 | . | link | 52:54:00:4d:27:e9 |
cloud-init[321]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
cloud-init[321]: ci-info: | lo | True | ::1/128 | . | host | . |
cloud-init[321]: ci-info: | sit0 | False | . | . | . | . |
cloud-init[321]: ci-info: | tunl0 | False | . | . | . | . |
cloud-init[321]: ci-info: +--------+-------+----------------------------+---------------+--------+-------------------+
cloud-init[321]: ci-info: +++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
cloud-init[321]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+
cloud-init[321]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
cloud-init[321]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+
cloud-init[321]: ci-info: | 0 | 0.0.0.0 | 172.31.3.1 | 0.0.0.0 | enp1s0 | UG |
cloud-init[321]: ci-info: | 1 | 172.31.3.0 | 0.0.0.0 | 255.255.255.0 | enp1s0 | U |
cloud-init[321]: ci-info: | 2 | 172.31.3.1 | 0.0.0.0 | 255.255.255.255 | enp1s0 | UH |
cloud-init[321]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+
cloud-init[321]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
cloud-init[321]: ci-info: +-------+-------------+---------+-----------+-------+
cloud-init[321]: ci-info: | Route | Destination | Gateway | Interf...

[Ryan Harper (raharper)]

In your dual stack example, can you run cloud-init collect-logs and attach the created tarball?

In particular the cloud-init.log and what, if any network configuration you're supplying in your NoCloud datasource.

> This tells me that someone enabled dhcp4 but not dhcp6 for cloud-init. We need to make sure we have feature parity across the protocols.

If you are not providing any network config to cloud-init, current the default fallback is to DHCP (ipv4) only; Ubuntu itself defaults to accepting IPv6 RAs, but asynchronously from boot.

And for ipv6 config, I suspect you'll want to provide an netplan config with:

network:
  version: 2
  ethernets:
    primary:
       match:
          macaddress: 52:54:00:4d:27:e9
       dhcp4: true
       dhcp6: true
       accept-ra: yes

Which ensures that systemd-networkd-wait-online.service will wait the required time for querying and accepting any RA.

[Kevin Otte (nivex)]

> If you are not providing any network config to cloud-init, current the default fallback is to DHCP (ipv4) only

This is precisely the bug I am reporting.

> And for ipv6 config, I suspect you'll want to provide an netplan config with:

In the year 2020, IPv6 should be a first class citizen and I should not need to provide additional config for it to work right. The issue here is not that I can't find a workaround to make my scenarios work. The issue here is that I should not need any such workarounds on a modern network.

[Ryan Harper (raharper)]

Hi Kevin,

> In the year 2020, IPv6 should be a first class citizen and I should not need
> to provide additional config for it to work right. The issue here is not
> that I can't find a workaround to make my scenarios work. The issue here is
> that I should not need any such workarounds on a modern network.

Thanks. I understand, and in general, agree with the goal. Our challenge is
how to balance default configurations with competing goals. We can enable
lots of default features for maximum compatibility; however there is almost
always a trade-off. The trade off for IPv6 is significant delay during boot
if IPv6 is not present.

Cloud-init blocks during boot to ensure that networking is configured (as
expected) before continuing to boot; later stages of boot can expect that
networking is available for fetching data from the network. In the case of
IPv6, and *blocking* for IPv6 during boot there is a very high cost in
networks where IPv6 is not available. The Ubuntu kernel already accepts
IPV6 Router Advertisements by default, but asynchronously from the systems
network configuration. As mentioned, we can enable blocking on RAs by
default but there is a very large boot time delay for networks without RAs.

As an example, I've two instance differing only in one where I've enabled
blocking for RAs and the boot time is 12 seconds slower due to waiting on RAs.

# default fallback networking (dhcp4 only)
root@g3:~# systemd-analyze
Startup finished in 8.132s (userspace)
graphical.target reached after 7.603s in userspace

# dhcp4 and accept-ra: true
root@g4:~# systemd-analyze
Startup finished in 19.988s (userspace)
graphical.target reached after 19.472s in userspace

# using systemd-analyze critical-chain we can see where the time went
root@g3:~# systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @7.603s
└─multi-user.target @7.602s
  └─rsyslog.service @2.524s +5.074s
    └─basic.target @2.491s
      └─sockets.target @2.491s
        └─snapd.socket @2.489s +2ms
          └─sysinit.target @2.485s
            └─cloud-init.service @1.853s +631ms
              └─systemd-networkd-wait-online.service @1.608s +243ms
                └─systemd-networkd.service @1.326s +279ms
                  └─network-pre.target @1.325s
                    └─cloud-init-local.service @747ms +578ms
                      └─systemd-journald.socket @165ms
                        └─system.slice @162ms
                          └─-.slice @162ms

root@g4:~# systemd-analyze
Startup finished in 19.988s (userspace)
graphical.target reached after 19.472s in userspace
root@g4:~# systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @19.472s
└─multi-user.target @19.472s
  └─rsyslog.service @14.378s +5.092s
    └─basic.target @14.338s
      └─sockets.target @14.338s
        └─snapd.socket @14.333s +4ms
          └─sysinit.target @14.329s
            └─cloud-init.service @13.686s +642ms
             ...

[Kevin Otte (nivex)]

Ryan,

Thank you so much for taking the time to go over the constraints and competing goals. It's always helpful to know what constituencies are involved.

> That said, I do think that cloud-init can make it easier to enable ipv6 where
> it's desired (but without having to write your own network config). We're
> looking into supporting a kernel parameter or cloud-config option to enable
> the default config to block on ipv6 (note, we cannot use user-data to indicate
> this preference as user-data is processed after networking is up as user-data
> can be fetched via network URL).

I think this would be a good compromise to get things moving. In my case of downloading the cloud image and specifying NoCloudNet via the SMBIOS serial number option, another key might be the simplest path. Something like:

n=4 : configure IPv4 only (default now)
n=6 : configure IPv6 only
n=64 : configure both stacks

I might then pass in:
ds=nocloud-net;n=6;h=cloudtest;i=iid-local01;s=http://gemini.home.nivex.net/cidata/

Of course it's easy for me to throw out such a suggestion since I don't know how much is involved in the codebase. Is this something that might be doable in the near future?

[Ryan Harper (raharper)]

> Ryan,
>
> Thank you so much for taking the time to go over the constraints and
> competing goals. It's always helpful to know what constituencies are
> involved.

Sure.

>
> > That said, I do think that cloud-init can make it easier to enable ipv6 where
> > it's desired (but without having to write your own network config). We're
> > looking into supporting a kernel parameter or cloud-config option to enable
> > the default config to block on ipv6 (note, we cannot use user-data to indicate
> > this preference as user-data is processed after networking is up as user-data
> > can be fetched via network URL).
>
> I think this would be a good compromise to get things moving. In my case of
> downloading the cloud image and specifying NoCloudNet via the SMBIOS serial
> number option, another key might be the simplest path. Something like:
>
> n=4 : configure IPv4 only (default now)
> n=6 : configure IPv6 only
> n=64 : configure both stacks
>
> I might then pass in:
> ds=nocloud-net;n=6;h=cloudtest;i=iid-local01;s=http://gemini.home.nivex.net/cidata/

I like this a lot. This example would only apply to NoCloud datasource
shorthand, which is fine, but we should expand this into a general form
where users can control the policy for any datasource.

I suspect we might bike-shed a bit on the long and short form of the config.
We internally refer to 'fallback' network config, and to express what the
current default configuration in long form.

I've started a hackmd document.

https://hackmd.io/hoAMkuQ8S_uj62ot3St9fw

>
> Of course it's easy for me to throw out such a suggestion since I don't know
> how much is involved in the codebase. Is this something that might be doable
> in the near future?

Definitely! Would you be interested in contributing? We can help shepard
the work along the way.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3682

[James Falcon (falcojr)]

I set the Ubuntu aspect of this as "Won't Fix" as this is really an upstream bug and will be tracked on GitHub.