Race condition when configuring IP ranges

[Jeff Lane ]

There appears to be a race when we set the IP ranges. IN some cases, for whatever reason, the rack controller is delayed in becoming available and this causes maniacs-setup to quietly exit with no real indication what the failure is, or that one even happened.

This was first reported by Samantha, and I have been able to recreate it in VMs in cloudspace in different network scenarios...

The most recent was with a network set up with a pure class B subnet, so no manual settings necessary as would be with something smaller than a class C (/25 - /31).

Here is the output of the script detailing what the user will see:

ubuntu@ubuntu-2gb-lon1-01:~$ sudo ifconfig
eth0 Link encap:Ethernet HWaddr f2:5a:67:b4:69:82
          inet addr:46.101.2.103 Bcast:46.101.63.255 Mask:255.255.192.0
          inet6 addr: fe80::f05a:67ff:feb4:6982/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:60219 errors:0 dropped:0 overruns:0 frame:0
          TX packets:43577 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:548954731 (548.9 MB) TX bytes:3445426 (3.4 MB)

eth1 Link encap:Ethernet HWaddr 52:b7:1b:26:49:0f
          inet addr:10.131.0.1 Bcast:10.131.255.255 Mask:255.255.0.0
          inet6 addr: fe80::50b7:1bff:fe26:490f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:4 errors:0 dropped:0 overruns:0 frame:0
          TX packets:58 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:168 (168.0 B) TX bytes:8744 (8.7 KB)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:191916 errors:0 dropped:0 overruns:0 frame:0
          TX packets:191916 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:1394007954 (1.3 GB) TX bytes:1394007954 (1.3 GB)

ubuntu@ubuntu-2gb-lon1-01:~$ sudo vim /etc/maas-cert-server/config
ubuntu@ubuntu-2gb-lon1-01:~$ sudo maniacs-setup

***************************************************************************
* Identified networks:
* INTERNAL: 10.131.0.1 on eth1
* EXTERNAL: 46.101.2.103 on eth0
*
* Is this correct (Y/n)? y

***************************************************************************

* Reconfiguring maas-region-controller
apache2.service is not a native service, redirecting to systemd-sysv-install
Executing /lib/systemd/systemd-sysv-install is-enabled apache2
* Reconfiguring maas-rack-controller

***************************************************************************
* Verifying that MAAS is running....
MAAS is running

***************************************************************************
* A MAAS administrative account with a name of ubuntu is being
* created.
*
* Please enter a password for this account:
* Please re-enter the password for verification:
* Setting up the ubuntu MAAS account using the supplied password

You are now logged in to the MAAS server at
http://10.131.0.1/MAAS/api/2.0/ with the profile name 'admin'.

For help with the available commands, try:

  maas admin --help

***************************************************************************
* Setting up SSH keys for ubuntu

***************************************************************************
* NAT enables this computer to connect the nodes it controls to the Internet
* for direct downloads of package updates and to submit certification results
* to C3.
*
* You can configure this computer to automatically start NAT. If you do so, you
* can disable it temporarily by using the 'flushnat.sh' script or permanently
* by removing the reference to /usr/sbin/startnat.sh from /etc/rc.local.
*
* Do you want to set up this computer to automatically enable NAT (Y/n)? y

***************************************************************************
* Mirroring an archive site is necessary if you'll be doing testing while
* disconnected from the Internet, and is desirable if your test site has
* poor Internet connectivity. Performing the mirroring operation takes
* time and disk space, though -- about 150 GiB per release mirrored.
* To defer this task, respond 'N' to the following question.
*
* Do you want to mirror an archive site for local use (y/N)?
*
* You can set up a local mirror later by running this script with the
* --mirror-archives (or -m) option.
*

***************************************************************************
* An Ubuntu cloud image is required for virtualization tests. Having such
* an image on your MAAS server can be convenient, but downloading it can
* take a while (each image is about 250MiB). This process will import cloud
* images for whatever releases and architectures you specify.
*
* To defer this task, respond 'N' to the following question.
*
* Do you want to set up a local cloud image mirror for the virtualization
* tests (Y/n)? n
*
* You can download virtualization images later by running this script with
* the --download-virtualization-images (or -d) option.
*

***************************************************************************
* Setting up MAAS IP address ranges:
* Low IP address = 10.131.2.0
* High IP address = 10.131.3.254
* Initializing rack controller
ubuntu@ubuntu-2gb-lon1-01:~$

I've now recreated this three times out of maybe 8 attempts, and it doesn't seem tied to any particular network configuration.

I've attached the MAAS logs to this bug, there are some tracebacks in rackd.log, which could be important here since that's the point where the script fails.

This is happening here:
if [ "$MAAS2" = "Y" ] ; then
        echo "* Setting up MAAS IP address ranges:"
        echo "* Low IP address = $IP_RANGE_LOW"
        echo "* High IP address = $STATIC_IP_RANGE_HIGH"

        local RANGES=`maas admin ipranges read | grep end_ip`
        if [ -z "$RANGES" ] ; then
            echo "* Initializing rack controller"
            maas admin ipranges create type=dynamic start_ip=$IP_RANGE_LOW end_ip=$STATIC_IP_RANGE_HIGH > /dev/null
            INTERNAL_FABRIC=`maas admin ipranges read | grep fabric | head -n 1 | tr -s " " | cut -d "\"" -f 4`
            PRIMARY_RACK=`maas admin rack-controllers read | grep hostname | head -n 1 | tr -s " " | cut -d "\"" -f 4`
            maas admin vlan update $INTERNAL_FABRIC untagged dhcp_on=True primary_rack=$PRIMARY_RACK > /dev/null
            local SUBNET_ID=`maas admin ipranges read | jshon -a -e subnet -e id`
            maas admin subnet update $SUBNET_ID gateway_ip=$INTERNAL_IP
        else
            echo "* Rack controller DHCP configuration already exists; leaving it alone!"
        fi

I think this is actually fixed in the latest version of maniacs-setup, but that'll need some more testing.

[Jeff Lane  (bladernr)]

Marking invalid, this was resolved by a MAAS update at some point.