Comment 16 for bug 1807615

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote : Re: gce: network broken, no apt update/install with LXD container

Besides "no default gateway" I had this in cloudinit-userdata:

juju model-config cloudinit-userdata
write_files:
- content: |
    [Service]
    Environment=SNAPPY_STORE_NO_CDN=1
  owner: "root:root"
  path: /etc/systemd/system/snapd.service.d/proxy-nocdn.conf
  permissions: '0644'
packages:
  - squashfuse
snap:
  commands:
    "00": systemctl restart snapd

https://paste.ubuntu.com/p/sV9gqCYhVC/
================================================================

I removed this cloudinit-userdata model-config and retried without it (what's below is not present in archives uploaded above) - doesn't look like this model-config affects anything, however, I can see by crtime of 95-juju-proxy-settings in the container that apt proxy settings land in a container after apt update is done (22:11:48 for update vs 22:11:53 for crtime of the proxy settings file:

https://paste.ubuntu.com/p/w4qFpR6KJ6/

Cloud-init v. 18.4-0ubuntu1~18.04.1 running 'modules:config' at Wed, 13 Feb 2019 22:11:45 +0000. Up 27.00 seconds.
Ign:1 http://security.ubuntu.com/ubuntu bionic-security InRelease
Err:2 http://security.ubuntu.com/ubuntu bionic-security Release
  Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8c01::19). - connect (101: Network is unreachable) Cannot initiate the connection to security
.ubuntu.com:80 (91.189.92.201). - connect (101: Network is unreachable)

# ...

2019-02-13 22:11:48,632 - util.py[WARNING]: Failed to install packages: ['curl', 'cpu-checker', 'bridge-utils', 'cloud-utils', 'tmux', 'ubuntu-fan']
2019-02-13 22:11:48,633 - cc_package_update_upgrade_install.py[WARNING]: 2 failed with exceptions, re-raising the last one
2019-02-13 22:11:48,634 - util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed

debugfs -R 'stat /var/lib/lxd/storage-pools/default/containers/juju-6d9aac-3-lxd-2/rootfs/etc/apt/apt.conf.d/95-juju-proxy-settings' /dev/sda1 2>/dev/null | grep -P 'crtime|mtime'
 mtime: 0x5c64b32f:04de7d60 -- Thu Feb 14 00:15:43 2019
crtime: 0x5c649629:813b3c54 -- Wed Feb 13 22:11:53 2019

============================================

Although I can't see why that happens because bootcmd commands (including the one to write proxy apt settings) run before cc_package_update_upgrade_install

2019-02-13 22:11:21,350 - stages.py[DEBUG]: Running module bootcmd (<module 'cloudinit.config.cc_bootcmd' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_bootcmd.py'>) with frequency always
# ...
2019-02-13 22:11:21,679 - handlers.py[DEBUG]: finish: init-network/config-bootcmd: SUCCESS: config-bootcmd ran successfully

2019-02-13 22:11:46,314 - stages.py[DEBUG]: Running module runcmd (<module 'cloudinit.config.cc_runcmd' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_runcmd.py'>) with frequency once-per-instance
2019-02-13 22:11:46,314 - handlers.py[DEBUG]: start: modules-config/config-runcmd: running config-runcmd with frequency once-per-instance

# ...
2019-02-13 22:11:47,029 - stages.py[DEBUG]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) with frequency once-per-instance

/var/lib/cloud/instance/user-data.txt

#cloud-config
apt_mirror: ""
bootcmd:
# ...
- |-
  printf '%s\n' 'Acquire::http::Proxy "http://10.232.0.1:3128";
  Acquire::https::Proxy "http://10.232.0.1:3128";
  Acquire::http::Proxy::"127.0.0.1" "DIRECT";
  Acquire::https::Proxy::"127.0.0.1" "DIRECT";
  Acquire::ftp::Proxy::"127.0.0.1" "DIRECT";
  Acquire::http::Proxy::"localhost" "DIRECT";
  Acquire::https::Proxy::"localhost" "DIRECT";
  Acquire::ftp::Proxy::"localhost" "DIRECT";
  Acquire::http::Proxy::"::1" "DIRECT";
  Acquire::https::Proxy::"::1" "DIRECT";
  Acquire::ftp::Proxy::"::1" "DIRECT";' > /etc/apt/apt.conf.d/95-juju-proxy-settings