16.04 image commissioning/installation failure due to config-ntp failure

Bug #1643116 reported by Nick
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
cloud-init (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

I'm using MAAS 2.1.1 hosted on 16.04. When I use 16.04 for Commissioning or Deployment, it always fails due to config-ntp failing. However, when I use 14.04 for Commissioning and Deployment, both succeed. I'd like to be able to use 16.04 for Deployment.

I am aware of bug #1628337, and my ephemeral image is using the newly fixed version of cloud-init and I still have a problem with the ntp state causing my 16.04 commissioning and deployment actions to fail.

During cloud-init, it appears that the ntp stage tries to call "apt-get update" twice. The first time succeeds, but the second time fails. I am not sure what is causing the failure.

The rsyslog contents from the maas server during the attempted commissioning are in attachment maas_rsyslog_messages.txt
- 10.4.3.3 is the maas server
- apt.example.com is the apt-cache I have (example.com replaces my own domain)
- Starting at line 354 the ntp stage begins. Lines 521-528 is where it fails and dumps a python traceback.

The host's mass UI event logs from both commissioning and deployment attempts are in latest_machine_events.txt.

The packages on the 16.04 ephemeral image are listed in attachment packages_on_16.04_node.txt.

I've attached the cloud-init.log and the associated cloud-init-output.log.

Revision history for this message
Nick (mrtizmo) wrote :
Revision history for this message
Nick (mrtizmo) wrote :

Adding the rest of the attachments...

Revision history for this message
Nick (mrtizmo) wrote :
Revision history for this message
Nick (mrtizmo) wrote :
Revision history for this message
Nick (mrtizmo) wrote :
Revision history for this message
Nick (mrtizmo) wrote :
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in cloud-init (Ubuntu):
status: New → Confirmed
Revision history for this message
Steven McCullie (stevyn) wrote :

been trying to get round this all afternoon, and think the error is misleading, even uninstalling the cloud-init package give the same error. disabling the "Enable the use of an APT and HTTP/HTTPS proxy" setting got my nodes to a ready state

Revision history for this message
Nick (mrtizmo) wrote :

Disabling the "Enable the use of an APT and HTTP/HTTPS proxy" setting did not change the outcome on my system. It still failed at the ntp state.

Interestingly, I did the Commissioning with the checkbox on to "Allow SSH access and prevent machine from turning off". When I log onto the host, the host appears to be fully functional, even though MAAS thinks it has failed.

Using the cloud-init command, I ran the ntp state of cloud-init:

sudo cloud-init -d single --name ntp --frequency once

This completed successfully, so the ntp state itself works. It must be an assumed pre-requisite that is missing or broken at the point ntp is called in the standard config stage. I'll look into that.

Revision history for this message
Nick (mrtizmo) wrote :
Download full text (3.2 KiB)

I have resolved this problem.

The base issue was caused because I am using a local apt mirror and I did not have the deb-src packages available on my mirror. When the ntp state runs, it does an apt-get update that processes the entire list of deb and deb-src package sources and if it cannot access one of them, apt-get will print error messages and return code 100.

The end of the apt-get update showed this, which is the stderr output from apt-get:

E: Failed to fetch http://apt.example.com/ubuntu/dists/xenial-backports/main/source/Sources 404 Not Found
E: Failed to fetch http://apt.example.com/ubuntu/dists/xenial/main/source/Sources 404 Not Found
E: Failed to fetch http://apt.example.com/ubuntu/dists/xenial-updates/main/source/Sources 404 Not Found
E: Failed to fetch http://apt.example.com/ubuntu/dists/xenial-security/main/source/Sources 404 Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.

And following that in the cloud-init.log was the python traceback:

Nov 18 22:08:44 rig9 [CLOUDINIT] util.py[DEBUG]: Running module ntp (<module 'cloudinit.config.cc_ntp' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ntp.py'>) failed#012Traceback (most recent call last):#012 File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 785, in _run_modules#012 freq=freq)#012 File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 70, in run#012 return self._runners.run(name, functor, args, freq, clear_on_fail)#012 File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 199, in run#012 results = functor(*args)#012 File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ntp.py", line 58, in handle#012 check_exe="ntpd")#012 File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ntp.py", line 69, in install_ntp#012 install_func(packages)#012 File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 87, in install_packages#012 self.update_package_sources()#012 File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 189, in update_package_sources#012 ["update"], freq=PER_INSTANCE)#012 File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 199, in run#012 results = functor(*args)#012 File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 185, in package_command#012 args=(cmd,), kwargs={'env': e, 'capture': False})#012 File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2163, in log_time#012 ret = func(*args, **kwargs)#012 File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1832, in subp#012 cmd=args)#012cloudinit.util.ProcessExecutionError: Unexpected error while running command.#012Command: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update']#012Exit code: 100#012Reason: -#012Stdout: ''#012Stderr: ''

Finally, the reason I was able to successfully run the ntp state manually after cloud-init had finished was because the ntp state itself works, and I was not sending a failure message to MAAS like the cloud-init process does. Also, I didn't care that apt-get ...

Read more...

Revision history for this message
James Falcon (falcojr) wrote :

Ubuntu 16.04 is no longer supported and the last message says the problem was resolved. Because of this, I am marking this issue as Invalid.

Changed in cloud-init (Ubuntu):
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.