Canonical Juju

cannot bootstrap local machine

Bug #2060706 reported by Maksim Beliaev

This bug report will be marked for expiration in 17 days if no further activity occurs. (find out why)

6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Incomplete
Undecided
Unassigned

Bug Description

Hello,
when I try to bootstrap my machine it gets stuck.

```
13:26:19 DEBUG juju.provider.common bootstrap.go:708 connection attempt for 10.26.202.156 failed: /var/lib/juju/nonce.txt does not exist
```

```
maksim@darmbeliaev:~$ sudo snap install lxd
lxd 5.20-f3dd836 from Canonical✓ installed
maksim@darmbeliaev:~$ lxd init --auto
```

```
DISTRIB_DESCRIPTION="Ubuntu 23.10"
```

once I manually SSH into LXC container and run within container:
```
install -D -m 644 /dev/null '/var/lib/juju/nonce.txt'
echo 'user-admin:bootstrap' > '/var/lib/juju/nonce.txt'
```

then juju picks up the rest and bootstraps fine.
Obviously, that is not an ideal case to ssh everytime and run those commands

Revision history for this message
Maksim Beliaev (beliaev-maksim) wrote :

see also https://matrix.to/#/!xzmWHtGpPfVCXKivIh:ubuntu.com/$faN77ySViUC17tXvV37l2mY4iSX2GlAozPvI3oEcs8c?via=ubuntu.com&via=matrix.org&via=mx.aouss.it

Revision history for this message
Harry Pidcock (hpidcock) wrote :

Thanks for the bug report. Are you able to get us the /var/lib/cloud-init-output.log? The container might still be running apt upgrade.

Changed in juju:
status: New → Incomplete
Revision history for this message
Maksim Beliaev (beliaev-maksim) wrote :

@Harry

root@juju-767b1c-0:~# more /var/log/cloud-init-output.log
Cloud-init v. 23.4.4-0ubuntu0~22.04.1 running 'init-local' at Tue, 09 Apr 2024 11:56:15 +0000. Up 0.59 seconds.
Cloud-init v. 23.4.4-0ubuntu0~22.04.1 running 'init' at Tue, 09 Apr 2024 11:56:16 +0000. Up 1.72 seconds.
ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: | eth0 | True | 10.26.202.118 | 255.255.255.0 | global | 00:16:3e:27:c3:47 |
ci-info: | eth0 | True | fe80::216:3eff:fe27:c347/64 | . | link | 00:16:3e:27:c3:47 |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: ++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
ci-info: +-------+-------------+-------------+-----------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-------------+-------------+-----------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 10.26.202.1 | 0.0.0.0 | eth0 | UG |
ci-info: | 1 | 10.26.202.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U |
ci-info: | 2 | 10.26.202.1 | 0.0.0.0 | 255.255.255.255 | eth0 | UH |
ci-info: +-------+-------------+-------------+-----------------+-----------+-------+
ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | 0 | fe80::/64 | :: | eth0 | U |
ci-info: | 2 | local | :: | eth0 | U |
ci-info: | 3 | multicast | :: | eth0 | U |
ci-info: +-------+-------------+---------+-----------+-------+

Revision history for this message
Maksim Beliaev (beliaev-maksim) wrote :

feels like /var/lib/cloud/instance/user-data.txt is not picked up by cloud init

Maksim Beliaev (beliaev-maksim)
Changed in juju:
status: Incomplete → New
Revision history for this message
Joseph Phillips (manadart) wrote :

This is probably something to do with cloud-init getting stuck.

We need the output of /var/log/cloud-init.log (not cloud-init-output.log) to see what step it is getting up to.

Changed in juju:
status: New → Incomplete
Revision history for this message
Maksim Beliaev (beliaev-maksim) wrote :
Download full text (53.2 KiB)

root@juju-47a237-0:~# cat /var/log/cloud-init.log
2024-04-19 10:20:29,006 - util.py[DEBUG]: Cloud-init v. 23.4.4-0ubuntu0~22.04.1 running 'init-local' at Fri, 19 Apr 2024 10:20:28 +0000. Up 0.57 seconds.
2024-04-19 10:20:29,006 - main.py[INFO]: PID [1] started cloud-init.
2024-04-19 10:20:29,006 - main.py[DEBUG]: No kernel command line url found.
2024-04-19 10:20:29,006 - main.py[DEBUG]: Closing stdin.
2024-04-19 10:20:29,009 - util.py[DEBUG]: Writing to /var/log/cloud-init.log - ab: [640] 0 bytes
2024-04-19 10:20:29,009 - util.py[DEBUG]: Changing the ownership of /var/log/cloud-init.log to 104:4
2024-04-19 10:20:29,009 - util.py[DEBUG]: Writing to /var/lib/cloud/data/python-version - wb: [644] 4 bytes
2024-04-19 10:20:29,010 - util.py[DEBUG]: Attempting to remove /var/lib/cloud/instance/boot-finished
2024-04-19 10:20:29,010 - handlers.py[DEBUG]: start: init-local/check-cache: attempting to read from cache [check]
2024-04-19 10:20:29,010 - util.py[DEBUG]: Reading from /var/lib/cloud/instance/obj.pkl (quiet=False)
2024-04-19 10:20:29,010 - stages.py[DEBUG]: no cache found
2024-04-19 10:20:29,010 - handlers.py[DEBUG]: finish: init-local/check-cache: SUCCESS: no cache found
2024-04-19 10:20:29,010 - util.py[DEBUG]: Attempting to remove /var/lib/cloud/instance
2024-04-19 10:20:29,012 - stages.py[DEBUG]: Using distro class <class 'cloudinit.distros.ubuntu.Distro'>
2024-04-19 10:20:29,012 - __init__.py[DEBUG]: Looking for data source in: ['LXD', 'None'], via packages ['', 'cloudinit.sources'] that matches dependencies ['FILESYSTEM']
2024-04-19 10:20:29,014 - __init__.py[DEBUG]: Searching for local data source in: ['DataSourceLXD']
2024-04-19 10:20:29,014 - handlers.py[DEBUG]: start: init-local/search-LXD: searching for local data from DataSourceLXD
2024-04-19 10:20:29,014 - __init__.py[DEBUG]: Seeing if we can get any data from <class 'cloudinit.sources.DataSourceLXD.DataSourceLXD'>
2024-04-19 10:20:29,014 - __init__.py[DEBUG]: Update datasource metadata and network config due to events: boot-new-instance
2024-04-19 10:20:29,015 - __init__.py[DEBUG]: Detected platform: DataSourceLXD. Checking for active instance data
2024-04-19 10:20:29,018 - DataSourceLXD.py[DEBUG]: [GET] [HTTP:200] http://lxd/1.0/meta-data
2024-04-19 10:20:29,020 - DataSourceLXD.py[DEBUG]: [GET] [HTTP:200] http://lxd/1.0/config
2024-04-19 10:20:29,022 - DataSourceLXD.py[DEBUG]: [GET] [HTTP:200] http://lxd/1.0/config/user.juju-controller-uuid
2024-04-19 10:20:29,024 - DataSourceLXD.py[DEBUG]: [GET] [HTTP:200] http://lxd/1.0/config/user.juju-is-controller
2024-04-19 10:20:29,025 - DataSourceLXD.py[DEBUG]: [GET] [HTTP:200] http://lxd/1.0/config/user.juju-model-uuid
2024-04-19 10:20:29,027 - DataSourceLXD.py[DEBUG]: [GET] [HTTP:200] http://lxd/1.0/config/user.user-data
2024-04-19 10:20:29,045 - DataSourceLXD.py[DEBUG]: [GET] [HTTP:200] http://lxd/1.0/devices
2024-04-19 10:20:29,045 - util.py[DEBUG]: Crawl of metadata service took 0.030 seconds
2024-04-19 10:20:29,045 - util.py[DEBUG]: Attempting to load yaml from string of length 94 with allowed root types (<class 'dict'>,)
2024-04-19 10:20:29,054 - util.py[DEBUG]: Reading from /etc/os-release (quiet=False)
2024-04-19 10:20:29,055 - ...

Changed in juju:
status: Incomplete → New
Revision history for this message
Maksim Beliaev (beliaev-maksim) wrote :

I removed both LXD and JUJU, installed both

now it is working. I strongly believe that was an issue in LXD, since juju I got the same revision

Revision history for this message
Joseph Phillips (manadart) wrote :

Marked incomplete to start the subsidence count-down. Resurrect if required.

Changed in juju:
status: New → Incomplete
Revision history for this message
Robert Gildein (rgildein) wrote :
Download full text (3.8 KiB)

I'm hitting the same issue, and I'm running these versions on ARM.
```bash
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy
$ juju --version
3.5.0-genericlinux-arm64
$ lxd --version
5.21.1 LTS
```

cloud-init-output.log
```bash
cat /var/log/cloud-init-output.log
Cloud-init v. 24.1.3-0ubuntu1~22.04.1 running 'init-local' at Fri, 10 May 2024 16:24:05 +0000. Up 0.59 seconds.
Cloud-init v. 24.1.3-0ubuntu1~22.04.1 running 'init' at Fri, 10 May 2024 16:24:06 +0000. Up 1.76 seconds.
ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: | eth0 | True | 10.147.5.15 | 255.255.255.0 | global | 00:16:3e:82:9b:c8 |
ci-info: | eth0 | True | fe80::216:3eff:fe82:9bc8/64 | . | link | 00:16:3e:82:9b:c8 |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: +++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
ci-info: +-------+-------------+------------+-----------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-------------+------------+-----------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 10.147.5.1 | 0.0.0.0 | eth0 | UG |
ci-info: | 1 | 10.147.5.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U |
ci-info: | 2 | 10.147.5.1 | 0.0.0.0 | 255.255.255.255 | eth0 | UH |
ci-info: +-------+-------------+------------+-----------------+-----------+-------+
ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | 0 | fe80::/64 | :: | eth0 | U |
ci-info: | 2 | local | :: | eth0 | U |
ci-info: | 3 | multicast | :: | eth0 | U |
ci-info: +-------+-------------+---------+-----------+-------+
```

user-data.txt
```bash
$ cat /var/lib/cloud/instance/user-data.txt
#cloud-config
output:
  all: '| tee -a /var/log/cloud-init-output.log'
runcmd:
- set -xe
- install -D -m 644 /dev/null '/var/lib/juju/nonce.txt'
- echo 'user-admin:bootstrap' > '/var/lib/juju/nonce.txt'
ssh_keys:
  ecdsa_private: |
    -----BEGIN PRIVATE KEY-----
    ***
    -----END PRIVATE KEY-----
  ecdsa_public: |
    ***
  ed25519_private: |
    -----BEGIN PRIVATE KEY-----
    ***
    -----END PRIVATE KEY-----
  ed25519_public: |
    ***
  rsa_private: |
    ...

Read more...

Revision history for this message
Robert Gildein (rgildein) wrote :

I'm still hitting this issue and I have environment to reproduce it.

Changed in juju:
status: Incomplete → New
Revision history for this message
Ian Booth (wallyworld) wrote :

I wonder what's different in your environment. I'm running lxd 5.21 and tried with juju 3.5.1 and there was no issue bootstrapping.

Does your /var/lib/cloud/data/status.json contain anything interesting?

Revision history for this message
Joseph Phillips (manadart) wrote :

We need the output of /var/log/cloud-init.log (not cloud-init-output.log) to see what step it is getting up to.

Changed in juju:
status: New → Incomplete
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.