karmic netboot UEC install fails at registering nodes

[Nick Moffitt]

I have just installed two systems using karmic over PXEboot. Both were given the anna/choose_modules=eucalyptus-udeb boot option.

The Node Controller box was given the preseed config line:

 d-i eucalyptus/install-mode string Node

The Cluster Controller box was given:

 d-i eucalyptus/install-mode string Cluster

As well as appropriate settings for eucalyptus/cluster-name, eucalyptus/publicips, and eucalyptus/private-interface.

After boot, I followed the instructions from https://help.ubuntu.com/community/UEC/NodeInstallation to use ssh-copy-id to get the eucalyptus pubkey from the CC to the NC. I then tried the instructions from https://help.ubuntu.com/community/UEC/CDInstall to autodiscover nodes with:

 sudo euca_conf --no-rsync --discover-nodes

My NC was found, and I accepted the prompt to proceed to add it. The scp failed, because root does not have the eucalyptus user's key. I then tried:

 sudo -u eucalyptus euca_conf --discover-nodes

And this succeeded in copying keys around, but failed when it tried to edit /etc/eucalyptus/eucalyptus.conf because that is owned by root.

This seems to be a fairly grave problem in either the instructions or the euca_conf program. At the moment, though, I'd appreciate a helpful workaround. How am I to get my NC added?

[Neil Soman (neilsoman)]

I've linked it against the Karmic package. Upstream eucalyptus does not have the "--discover-nodes" option and seems to be a UEC issue.

[Thierry Carrez (ttx)]

I agree there seems to be a mismatch in the instructions here. Assigning Dustin since he authored most of that doc, he should be able to point to an error in your procedure or his instructions.

[Nick Moffitt (nick-moffitt)]

To clarify, I never managed to get the --no-rsync versions working.

Ultimately I managed to hack around this by symlinking the eucalyptus user's id_rsa into root's .ssh/ dir and re-running "sudo euca_conf --discover-nodes".

[Nick Moffitt (nick-moffitt)]

Recently I've stopped clobbering the node-preseed.conf quite as much, and even with the CC's late_command the whole setup still has this problem.

[Nick Moffitt (nick-moffitt)]

And to add insult to injury, it would appear that I have no NC lines in euca-describe-availability-zones verbose after all that. It seems that autodiscovery only *appeared* to complete.

[Nick Moffitt (nick-moffitt)]

After debugging with Dustin in IRC some this evening, it looks like this is the result of some failure of the eucalyptus-udeb within my netboot environment (or of the ordinary eucalyptus packages) to make the cloud controller. If I purge and reinstall everything works as you'd expect (but without the clever bootstrapping to make new NCs). I'm busily stripping down my preseed configuration to try and make this work, but it looks like there's little left to remove.

[Thierry Carrez (ttx)]

Updated title to reflect latest findings. I also changed "autodiscovery" -> "registration of nodes" since --discover-nodes seems to discover nodes alright, it's the node register operation that --discover-nodes triggers that ends up buggy. I suspect you could reproduce it by manually running "--register-nodes IPADDRESS".

[Nick Moffitt (nick-moffitt)]

I pulled everything out that I thought could be interfering (mostly by not specifying a preseed/late_command *anywhere*) but still no luck. I'll poke around in the maintainer scripts to see if i can spot any obvious reason why this stuff would fail based on the install environment like this.

[Nick Moffitt (nick-moffitt)]

I've attached a snippet of /var/log/installer/syslog from a system just after PXE installation attempted to make a CC of it via the eucalyptus-udeb. You'll notice that it's Setting up version 1.6~bzr931-0ubuntu7 of eucalyptus-cc. It then runs ssh-keygen via an su to the eucalyptus user, and promptly complains about there being no upstart to talk to.

The reason I take interest in this is that it appears that part of my problem is the failure of code in this postinst to run. For example, I don't even have an /etc/eucalyptus/eucalyptus-cc.conf until after my purge/reinstall of eucalyptus-cc. I also don't seem to have the VNET_MODE or VNET_PUBLICIPS settings, for some reason.

So it would seem that either the test for a fresh install is failing somehow, or the debconf query for eucalyptus/cluster-name is failing along with some more subtle problem.

[Nick Moffitt (nick-moffitt)]

Ignoring for the moment that the subnet is all wrong for the IP range I have chosen, here is the diff of /etc/eucalyptus before and after I purged/reinstalled eucalyptus-cc

[Nick Moffitt (nick-moffitt)]

I just added a purge/install of eucalyptus-cc (which works when done after first boot) into my late_command, and it does this:

Nov 18 15:39:12 in-target: The following NEW packages will be installed:
Nov 18 15:39:12 in-target: eucalyptus-cc
Nov 18 15:39:12 in-target: 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Nov 18 15:39:12 in-target: Need to get 506kB of archives.
Nov 18 15:39:12 in-target: After this operation, 2404kB of additional disk space will be used.
Nov 18 15:39:12 in-target: Get:1 http://archive.ubuntu.com karmic/main eucalyptus-cc 1.6~bzr931-0ubuntu7 [506kB]
Nov 18 15:39:13 in-target: debconf: unable to initialize frontend: Passthrough
Nov 18 15:39:13 in-target: debconf: (Failed to open fd 3: Bad file descriptor at (eval 22) line 3)
Nov 18 15:39:13 in-target: debconf: falling back to frontend: Noninteractive
Nov 18 15:39:13 in-target: Preconfiguring packages ...
Nov 18 15:39:13 in-target: Fetched 506kB in 0s (8705kB/s)
Nov 18 15:39:13 in-target: Selecting previously deselected package eucalyptus-cc.
Nov 18 15:39:13 in-target: (Reading database ...
Nov 18 15:39:13 in-target: 50999 files and directories currently installed.)
Nov 18 15:39:13 in-target: Unpacking eucalyptus-cc (from .../eucalyptus-cc_1.6~bzr931-0ubuntu7_amd64.deb) ...
Nov 18 15:39:13 in-target: Setting up eucalyptus-cc (1.6~bzr931-0ubuntu7) ...
Nov 18 15:39:13 in-target: debconf: unable to initialize frontend: Passthrough
Nov 18 15:39:13 in-target: debconf: (Failed to open fd 3: Bad file descriptor at (eval 22) line 3)
Nov 18 15:39:13 in-target: debconf: falling back to frontend: Noninteractive
Nov 18 15:39:13 in-target: sed: warning: failed to get security context of //etc/eucalyptus/eucalyptus.conf: No data available
Nov 18 15:39:13 in-target: sed: warning: failed to get security context of //etc/eucalyptus/eucalyptus.conf: No data available
Nov 18 15:39:13 in-target: sed: warning: failed to get security context of //etc/eucalyptus/eucalyptus.conf: No data available
Nov 18 15:39:13 in-target: sed: warning: failed to get security context of //etc/eucalyptus/eucalyptus.conf: No data available
Nov 18 15:39:13 in-target: sed: warning: failed to get security context of //etc/eucalyptus/eucalyptus.conf: No data available
Nov 18 15:39:13 in-target: start: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
Nov 18 15:39:14 in-target:
Nov 18 15:39:14 in-target: modified:
Nov 18 15:39:14 in-target: eucalyptus/eucalyptus.conf
Nov 18 15:39:14 in-target: eucalyptus/eucalyptus.conf.bak

What is that security context message? Google simply turns up a lot of very confused people.

[Nick Moffitt (nick-moffitt)]

(also, my system comes up in much the same state as if I hadn't done the reinstall in my late_command script)

[Nick Moffitt (nick-moffitt)]

On a whim, I decided to query debconf with debconf-show eucalyptus-cc before and after the purge/reinstall:

Nov 18 17:04:03 in-target: eucalyptus/mode: MANAGED-NOVLAN
Nov 18 17:04:03 in-target: * eucalyptus/publicips:
Nov 18 17:04:03 in-target: eucalyptus/dns: 172.24.1.1
Nov 18 17:04:03 in-target: * eucalyptus/cluster-name:
Nov 18 17:04:03 in-target: eucalyptus/netmask: 255.255.0.0
Nov 18 17:04:03 in-target: eucalyptus/addrspernet: 32
Nov 18 17:04:03 in-target: eucalyptus/subnet: 172.19.0.0

and after the purge/reinstall:

Nov 18 17:04:08 in-target: eucalyptus/mode: MANAGED-NOVLAN
Nov 18 17:04:08 in-target: eucalyptus/publicips:
Nov 18 17:04:08 in-target: eucalyptus/dns: 172.24.1.1
Nov 18 17:04:08 in-target: eucalyptus/cluster-name:
Nov 18 17:04:08 in-target: eucalyptus/netmask: 255.255.0.0
Nov 18 17:04:08 in-target: eucalyptus/addrspernet: 32
Nov 18 17:04:08 in-target: eucalyptus/subnet: 172.19.0.0

So what happened to my preseeded info?

# grep publicips /var/log/installer/syslog
Nov 18 16:57:58 debconf: --> SET eucalyptus/publicips 172.24.56.0-172.24.63.255
Nov 18 16:57:58 debconf: <-- 10 eucalyptus/publicips doesn't exist
Nov 18 16:57:58 debconf: --> REGISTER debian-installer/dummy eucalyptus/publicips
Nov 18 16:57:58 debconf: --> SET eucalyptus/publicips 172.24.56.0-172.24.63.255
Nov 18 16:57:58 debconf: --> SUBST eucalyptus/publicips ID eucalyptus/publicips
Nov 18 16:57:58 debconf: Adding [ID] -> [eucalyptus/publicips]
Nov 18 16:57:58 debconf: --> FSET eucalyptus/publicips seen true
Nov 18 17:00:27 debconf-copydb: Adding [ID] -> [eucalyptus/publicips]
Nov 18 17:00:28 debconf-copydb: Adding [ID] -> [eucalyptus/publicips]
Nov 18 17:04:03 in-target: * eucalyptus/publicips:
Nov 18 17:04:08 in-target: eucalyptus/publicips:

[Nick Moffitt (nick-moffitt)]

Once I narrowed it down to debconf, Colin Watson immediately clued me to the fact that I most likely had the wrong owner for my eucalyptus-cc settings. In fact, I had just cut&wasted the d-i from the eucalyptus-udeb settings and not even thought twice about it. Changing the owner to "eucalyptus-cc" fixed the problem and the configuration files are as they should be.

I'm still testing the full installation, though.

[Nick Moffitt (nick-moffitt)]

It works, so you can resolve/reject this bug as user error.

[Dustin Kirkland  (kirkland)]

Cool, thanks for the followup, Nick!