fuel menu doesn't start after install from USB stick

[Jason]

I did a bare metal USB install on a server.

All went fine.

After the first reboot, there is no fuel menu that comes up and the server doesn't seem to fully boot. the screen gets stuck after rhsmecertd start.

I go into another tty and try to load fuel menu manually.

I get this error.

[root@fuel ~]# fuelmenu
Traceback (most recent call last):
  File "/usr/bin/fuelmenu", line 9, in <module>
    load_entry_point('fuelmenu==6.1.0', 'console_scripts', 'fuelmenu')()
  File "/usr/lib/python2.6/site-packages/fuelmenu/fuelmenu.py", line 471, in main
    setup()
  File "/usr/lib/python2.6/site-packages/fuelmenu/fuelmenu.py", line 337, in setup
    FuelSetup()
  File "/usr/lib/python2.6/site-packages/fuelmenu/fuelmenu.py", line 90, in __init__
    self.main()
  File "/usr/lib/python2.6/site-packages/fuelmenu/fuelmenu.py", line 192, in main
    self.children, self.choices = loader.load_modules(module_dir=moduledir)
  File "/usr/lib/python2.6/site-packages/fuelmenu/fuelmenu.py", line 66, in load_modules
    modobj = clsobj(self.parent)
  File "/usr/lib/python2.6/site-packages/fuelmenu/modules/cobblerconf.py", line 85, in __init__
    self.oldsettings = self.load()
  File "/usr/lib/python2.6/site-packages/fuelmenu/modules/cobblerconf.py", line 291, in load
    self.defaults[setting]["value"] = oldsettings[part1][part2]
KeyError: 'dhcp_gateway'
[root@fuel ~]#

Another install attempt:

During initial install, I have to change the hd:LABEL="mirantis_fuel":/ks.cfg to hd:sda1:/ks.cfg

Thats if my usb mounts to sda. setting the ks line to the repo label will also work...

Howver, AFTER the install and the system reboots, it stops after "starting rhsmcertd start". Either the setup isnt actually running the "post installation scripts" or docker isnt starting?

I have also tried a nightly from the fuel site.

I can get a DVD install to work perfectly. I have noticed that on the dvd install, the post install scripts took about 3-5 min to complete. For usb, it takes 2 seconds.

[Jason (5-jason-h)]

I have found that the /etc/fuel/astute.yaml is not putting the following under ADMIN NETWORK:

dhcp_gateway: "10.20.0.1"

The gateway is declared on boot in the kernel string.

[Jason (5-jason-h)]

 During initial install, I have to change the hd:LABEL="mirantis_fuel":/ks.cfg to hd:sda1:/ks.cfg

Thats if my usb mounts to sda. setting the ks line to the repo label will also work...

Howver, AFTER the install and the system reboots, it stops after "starting rhsmcertd start". Either the setup isnt actually running the "post installation scripts" or docker isnt starting?

I have also tried a nightly from the fuel site.

I can get a DVD install to work perfectly. I have noticed that on the dvd install, the post install scripts took about 3-5 min to complete. For usb, it takes 2 seconds.

[Matthew Mosesohn (raytrac3r)]

Jason,

Can you repeat the installation with the USB disk and get me the following items?
1 - /var/log/puppet/bootstrap_admin_node.log
2 - /var/log/fuelmenu.log
3 - output of `ip a`
4 - output of `ip ro`

If you can provide me with remote access to your host, I could do some troubleshooting with you. I'm "mattymo" on IRC in #fuel-dev on freenode.

[Jason (5-jason-h)]

I can in about 8 hours.

I will say that fuelmenu.log doesn't get created unless I run it manually. Same with bootstrap_admin_node.log except it doesn't get created at all. I can't even run bootstrap_admin_node. It doesn't exist.

[Jason (5-jason-h)]

[root@fuel ~]# cat /var/log/puppet/bootstrap_admin_node.log
cat: /var/log/puppet/bootstrap_admin_node.log: No such file or directory

[root@fuel ~]# cat /var/log/fuelmenu.log
cat: /var/log/fuelmenu.log: No such file or directory

[root@fuel ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:25:90:2a:f4:ea brd ff:ff:ff:ff:ff:ff
    inet 10.20.0.2/24 brd 10.20.0.255 scope global eth0
    inet6 fe80::225:90ff:fe2a:f4ea/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:25:90:2a:f4:eb brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.12/24 brd 10.0.1.255 scope global eth1
    inet6 fe80::225:90ff:fe2a:f4eb/64 scope link
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:25:90:38:ae:c7 brd ff:ff:ff:ff:ff:ff
5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 92:d7:8c:5f:cc:57 brd ff:ff:ff:ff:ff:ff
    inet 172.17.42.1/16 scope global docker0
    inet6 fe80::90d7:8cff:fe5f:cc57/64 scope link
       valid_lft forever preferred_ft forever

[root@fuel ~]# ip ro
10.20.0.0/24 dev eth0 proto kernel scope link src 10.20.0.2
10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.12
169.254.0.0/16 dev eth0 scope link metric 1002
169.254.0.0/16 dev eth1 scope link metric 1003
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.42.1
default via 10.0.1.1 dev eth1

I added the eth1 config in there so I could have network to my pc.

All I have done is ran the installer and rebooted. The boot screen never fully completes.

e1000 0000:01:02.0: eth2: (PCI:33MHz:32-bit) 00:25:90:38:ae:c7
e1000 0000:01:02.0: eth2: Intel(R) PRO/1000 Network Connection
iTCO_vendor_support: vendor-support=0
iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
iTCO_wdt: Found a ICH10R TCO device (Version=2, TCOBASE=0x0860)
iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts:
EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts:
EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts:
EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts:

It stops there.

to get the info you requested, i had to open another console and login.

[Jason (5-jason-h)]

BTW, this is mirantis fuel 6.1 iso download to USB.

[Stanislaw Bogatkin (sbogatkin)]

I can't reproduce this on 8.0 ISO when trying to boot from USB. Will try more on 6.1 ISO.

[Stanislaw Bogatkin (sbogatkin)]

Sorry for mistake, I mean 7.0 ISO when talked about 8.0

[Stanislaw Bogatkin (sbogatkin)]

The same for 6.1 - Fuel menu appeared and worked.

[Jason (5-jason-h)]

You are doing it on a hardware server? It load fine on virtual box. I have tried the USB install on 3 different servers now and had the same issue on all of them.

[Jason (5-jason-h)]

How do you create your USB ?

What software are you using?

[Jason (5-jason-h)]

If I run the install on these servers from DVD, I am successful.

[Stanislaw Bogatkin (sbogatkin)]

Nope, I used virtual machines for that based on KVM.
USB was created by usual dd command (diskdupe).

[Stanislaw Bogatkin (sbogatkin)]

Jason, would you be so kind to provide access to your fuel master node (via ssh, for example)? It would be much easier to catch a root cause of a problem on hardware where it happens.

[Jason (5-jason-h)]

yes, you can go to 72.201.141.72

You will see how it seems that post install scripts don't really get run. Thats what it looks like to me.

[Jason (5-jason-h)]

I figured something out!

If the ISO is created with windows, it fails for me.

Now it was successful for the first time via USB using dd on linux.

So, something is happening when images are creating. I do know that the label of the USB is always cut off and long. I think this mismatch is breaking it. I think that making image labels less than 8 characters may fix this.

[Stanislaw Bogatkin (sbogatkin)]

Alexey, why you put it in 'Confirmed'? We have a clear documentation how to create usb stick and if you will follow it all is okay.

[Jason (5-jason-h)]

It was definitely a bug and its definitely not ok if you take the MOS 6.1 ISO and create the USB stick on windows. Then install on a REAL server. It works fine on virtual box. I no longer use USB as I now use a virtual CD.