16.04 image commissioning/installation failure due to config-ntp failure

[Nick]

I'm using MAAS 2.1.1 hosted on 16.04. When I use 16.04 for Commissioning or Deployment, it always fails due to config-ntp failing. However, when I use 14.04 for Commissioning and Deployment, both succeed. I'd like to be able to use 16.04 for Deployment.

I am aware of bug #1628337, and my ephemeral image is using the newly fixed version of cloud-init and I still have a problem with the ntp state causing my 16.04 commissioning and deployment actions to fail.

During cloud-init, it appears that the ntp stage tries to call "apt-get update" twice. The first time succeeds, but the second time fails. I am not sure what is causing the failure.

The rsyslog contents from the maas server during the attempted commissioning are in attachment maas_rsyslog_messages.txt
- 10.4.3.3 is the maas server
- apt.example.com is the apt-cache I have (example.com replaces my own domain)
- Starting at line 354 the ntp stage begins. Lines 521-528 is where it fails and dumps a python traceback.

The host's mass UI event logs from both commissioning and deployment attempts are in latest_machine_events.txt.

The packages on the 16.04 ephemeral image are listed in attachment packages_on_16.04_node.txt.

I've attached the cloud-init.log and the associated cloud-init-output.log.

[Nick (mrtizmo)]

Adding the rest of the attachments...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steven McCullie (stevyn)]

been trying to get round this all afternoon, and think the error is misleading, even uninstalling the cloud-init package give the same error. disabling the "Enable the use of an APT and HTTP/HTTPS proxy" setting got my nodes to a ready state

[Nick (mrtizmo)]

Disabling the "Enable the use of an APT and HTTP/HTTPS proxy" setting did not change the outcome on my system. It still failed at the ntp state.

Interestingly, I did the Commissioning with the checkbox on to "Allow SSH access and prevent machine from turning off". When I log onto the host, the host appears to be fully functional, even though MAAS thinks it has failed.

Using the cloud-init command, I ran the ntp state of cloud-init:

sudo cloud-init -d single --name ntp --frequency once

This completed successfully, so the ntp state itself works. It must be an assumed pre-requisite that is missing or broken at the point ntp is called in the standard config stage. I'll look into that.

[Nick (mrtizmo)]

I have resolved this problem.

The base issue was caused because I am using a local apt mirror and I did not have the deb-src packages available on my mirror. When the ntp state runs, it does an apt-get update that processes the entire list of deb and deb-src package sources and if it cannot access one of them, apt-get will print error messages and return code 100.

The end of the apt-get update showed this, which is the stderr output from apt-get:

E: Failed to fetch http://apt.example.com/ubuntu/dists/xenial-backports/main/source/Sources 404 Not Found
E: Failed to fetch http://apt.example.com/ubuntu/dists/xenial/main/source/Sources 404 Not Found
E: Failed to fetch http://apt.example.com/ubuntu/dists/xenial-updates/main/source/Sources 404 Not Found
E: Failed to fetch http://apt.example.com/ubuntu/dists/xenial-security/main/source/Sources 404 Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.

And following that in the cloud-init.log was the python traceback:

Nov 18 22:08:44 rig9 [CLOUDINIT] util.py[DEBUG]: Running module ntp (<module 'cloudinit.config.cc_ntp' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ntp.py'>) failed#012Traceback (most recent call last):#012 File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 785, in _run_modules#012 freq=freq)#012 File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 70, in run#012 return self._runners.run(name, functor, args, freq, clear_on_fail)#012 File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 199, in run#012 results = functor(*args)#012 File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ntp.py", line 58, in handle#012 check_exe="ntpd")#012 File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ntp.py", line 69, in install_ntp#012 install_func(packages)#012 File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 87, in install_packages#012 self.update_package_sources()#012 File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 189, in update_package_sources#012 ["update"], freq=PER_INSTANCE)#012 File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 199, in run#012 results = functor(*args)#012 File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 185, in package_command#012 args=(cmd,), kwargs={'env': e, 'capture': False})#012 File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2163, in log_time#012 ret = func(*args, **kwargs)#012 File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1832, in subp#012 cmd=args)#012cloudinit.util.ProcessExecutionError: Unexpected error while running command.#012Command: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update']#012Exit code: 100#012Reason: -#012Stdout: ''#012Stderr: ''

Finally, the reason I was able to successfully run the ntp state manually after cloud-init had finished was because the ntp state itself works, and I was not sending a failure message to MAAS like the cloud-init process does. Also, I didn't care that apt-get ...

[James Falcon (falcojr)]

Ubuntu 16.04 is no longer supported and the last message says the problem was resolved. Because of this, I am marking this issue as Invalid.