i386: Client drops into BusyBox

[TrialAndError]

precise, 2012-03-10

After building a new chroot on a clean fully updated Ubuntu Precise i386, client drops into BusyBox as described in #924681.

[Alkis Georgopoulos (alkisg)]

Could you remove "quiet splash" from /var/lib/tftpboot/ltsp/i386/pxelinux.cfg/default and then reboot the client and upload a photo of its screen when it drops to busybox, so that we see where the problem is?

Also post the output of the following commands on your server:
ip -oneline -family inet addr show
grep -v ^# /etc/nbd-server/config /etc/nbd-server/conf.d/*
cat /var/lib/tftpboot/ltsp/i386/pxelinux.cfg/default
grep nbd /etc/inetd.conf
sudo netstat -nap | grep :10809

[TrialAndError (trial-and-error)]

It is a default Virtualbox install, but here your are informations anyway:

ltsp@ltsp:~$ ip -oneline -family inet addr show
1: lo inet 127.0.0.1/8 scope host lo
2: eth0 inet 192.168.1.100/24 brd 192.168.1.255 scope global eth0
ltsp@ltsp:~$ grep -v ^# /etc/nbd-server/config /etc/nbd-server/conf.d/*
/etc/nbd-server/config:[generic]
/etc/nbd-server/config: user = nbd
/etc/nbd-server/config: group = nbd
/etc/nbd-server/config: includedir = /etc/nbd-server/conf.d
/etc/nbd-server/config:
ltsp@ltsp:~$ cat /var/lib/tftpboot/ltsp/i386/pxelinux.cfg/default
default ltsp

label ltsp
kernel vmlinuz
append ro initrd=initrd.img root=/dev/nbd0 init=/sbin/init-ltsph
plymouth:force-splash vt.handoff=7 nbdroot=:ltsp_i386

ltsp@ltsp:~$ grep nbd /etc/inetd.conf
ltsp@ltsp:~$ sudo netstat -nap | grep :10809
ltsp@ltsp:~$

Removing quiet splash has no effect any more ... But after hitting esc I
got the Informations you need I think.

[Alkis Georgopoulos (alkisg)]

The problem is that you don't have an /etc/nbd-server/conf.d/ltsp_i386.conf file and so nbd-server doesn't start itself.
That file is generated by ltsp-update-image. So the next information you need to include is any errors that you see after running:

sudo ltsp-update-image
(you might also need: sudo service nbd-server start)

That's supposed to be ran automatically at the end of ltsp-build-client. So if it runs fine for you now, it might mean that ltsp-build-client was interrupted and didn't complete successfully. That in turn might have happened for any number of reasons, even temporary ones like the Ubuntu archive being in an inconsistent state. We'd need the whole ltsp-build-client output to be certain, you probably didn't keep that so if that was the original problem we won't know it until you try ltsp-build-client again.

[TrialAndError (trial-and-error)]

Sorry, but if I only had tried it once I would not have filed a bug.

I am building the chroot on 12.04 i386 constantly from scratch for a
while now: erros are getting lesser but ...

ltsp-built-client complained about an existing ltsp_i386.conf which was
empty.

After deleting it and ltsp-update-image everything is fine now but
perhaps an existing ltsp_i386.conf should be deleted automatically?

Thanks for your help

Am 12.03.2012 16:24, schrieb Alkis Georgopoulos:
> The problem is that you don't have an /etc/nbd-server/conf.d/ltsp_i386.conf file and so nbd-server doesn't start itself.
> That file is generated by ltsp-update-image. So the next information you need to include is any errors that you see after running:
>
> sudo ltsp-update-image
> (you might also need: sudo service nbd-server start)
>
>
> That's supposed to be ran automatically at the end of ltsp-build-client. So if it runs fine for you now, it might mean that ltsp-build-client was interrupted and didn't complete successfully. That in turn might have happened for any number of reasons, even temporary ones like the Ubuntu archive being in an inconsistent state. We'd need the whole ltsp-build-client output to be certain, you probably didn't keep that so if that was the original problem we won't know it until you try ltsp-build-client again.
>

[Alkis Georgopoulos (alkisg)]

> perhaps an existing ltsp_i386.conf should be deleted automatically?

No, because it would mean that the user manually edited the file, so we should respect his changes.

I don't know what created an empty ltsp_i386.conf file in your case. Maybe a bug in some previous Precise LTSP package which has been resolved in the meantime?

Or do you suspect that this happens on every new installation? I haven't heard of others with that problem so far...

[TrialAndError (trial-and-error)]

Am 13.03.2012 12:03, schrieb Alkis Georgopoulos:
>> perhaps an existing ltsp_i386.conf should be deleted automatically?
> No, because it would mean that the user manually edited the file, so we
> should respect his changes.
>
OK, then in this case there should be a notification, that manual
editing is probably necessary to make ltsp work or a standard
ltsp_i386.conf.ltsp could be saved with a notification. Anything that
users could help themselves would be nice.

[Alkis Georgopoulos (alkisg)]

If a user edits a configuration file and deliberately empties its contents, I don't see why nbd-server or LTSP should bother notifying the user that what he did may (or may not, depending on his other configuration files) impact his server functionality.

I think that the question at this point is why that file was empty.
If it was a manual change, there's no point in keeping the bug report open, we might as well close it.

Regards,
Alkis

[TrialAndError (trial-and-error)]

Pardon, I never changed anything?! I did a clean install of a earlier
12.04 and ltsp-server-standalone.

The point is that any program might change a .conf file. So in the case
of an update there is the question what to do with an non-standard .conf
file. This is a good thing and a ltsp install could do the same.

Am 13.03.2012 17:18, schrieb Alkis Georgopoulos:
> If a user edits a configuration file and deliberately empties its
> contents, I don't see why nbd-server or LTSP should bother notifying the
> user that what he did may (or may not, depending on his other
> configuration files) impact his server functionality.
>
> I think that the question at this point is why that file was empty.
> If it was a manual change, there's no point in keeping the bug report open, we might as well close it.
>
> Regards,
> Alkis
>

[niplas (niplas)]

Same problem here, appears to happen when: ltsp-build-client...
I do actually have the complete ltsp-build-client output, unfortunatelly it's in Swedish.
I can paste it somewhere if you like.

As a sidenote, when executed the second time no message about restarting NBD-server shows up (!)
Instead this appears:
"Configuration file /etc/nbd-server/conf.d/ltsp_i386.conf already exists, no action taken."

Trying to restart NBD using:
>ltsp1-pe2600:/$ service nbd-server restart [OR]
>ltsp1-pe2600:/$ sudo /etc/init.d/nbd-server restart

.. renders:
Restarting the Network Block Device server is pretty harsh on clients still using it.
waiting 5 seconds...You have been warned!
Restarting Network Block Device server: Stopping Network Block Device server: nbd-server.

** (process:20241): WARNING **: Could not parse config file: The config file does not specify any exports
** Message: No configured exports; quitting.
 nbd-server.

Seen errors in the build log:

Creating config file /etc/default/grub with new version
grub-probe: error: cannot find a device for / (is /dev mounted?).
grub-probe: error: cannot find a device for /boot (is /dev mounted?).
grub-probe: error: cannot find a device for /boot/grub (is /dev mounted?).
*(logical disks on RAID5 handled by bios, controller wont let kernel see physical disks (

Seen warnings in the log:
#1
Ställer in language-selector-common (0.79) ...
/usr/lib/python2.7/dist-packages/LanguageSelector/LocaleInfo.py:256: UserWarning: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
  warnings.warn(msg.args[0].encode('UTF-8'))

#2
Ställer in colord (0.1.16-2) ...
adduser: Warning: The home directory `/var/lib/colord' does not belong to the user you are currently creating.

#3
Ställer in acpi-support (0.140) ...
invoke-rc.d: policy-rc.d denied execution of restart.
update-rc.d: warning: /etc/init.d/acpi-support missing LSB information
update-rc.d: see <http://wiki.debian.org/LSBInitScripts>

#4
Ställer in binfmt-support (2.0.8) ...
update-binfmts: warning: unable to close /proc/sys/fs/binfmt_misc/register: Invalid argument
update-binfmts: warning: unable to enable binary format python2.7
update-binfmts: warning: /usr/share/binfmts/cli: no executable /usr/bin/cli found, but continuing anyway as you request
update-binfmts: warning: found manually created entry for cli in /proc/sys/fs/binfmt_misc; leaving it alone
invoke-rc.d: policy-rc.d denied execution of start.

#5
Ställer in epoptes-client (0.5.3-1) ...
update-rc.d: warning: epoptes-client stop runlevel arguments (0 1 6) do not match LSB Default-Stop values (none)
invoke-rc.d: policy-rc.d denied execution of start.

#6
update-binfmts: warning: found manually created entry for cli in /proc/sys/fs/binfmt_misc; leaving it alone

minor:
Ställer in software-center (5.2) ... WARNING:softwarecenter.db.update:Failed to setup dbus (ignoring)

Else all good...

Sys info:
Vanilla installation except for:
* using XFS filesystem.
* DHCP-server on other machine (firewall)
* Clients set to use NBD-swap
* some tweaks for FAT-client (won't affect if turned on or of)

Original installation i386 10.04 -> 10.10 -> 11.04 -> 11.10 kernels altered to i386...

[niplas (niplas)]

Previous LTSP specific NBD conf differs from new generic default generated (?) on new build..

New:
        [generic]
 user = nbd
 group = nbd
 includedir = /etc/nbd-server/conf.d

Old:
        [generic]
        user = nobody
        group = nogroup
        oldstyle = false

        [ltsp_i386]
        exportname = /opt/ltsp/images/i386.img
        readonly = true

[Anthony (coolio986)]

I can confirm TrialandError's findings. I also am running virtualbox 4.1.14 with extension pack.

I have the same results. After removinng the quiet splash I get the results that Trial posted above.

Specs:
Virtualbox 4.1.14 w/ extension pack
AMD fx 6 core
16gb ram
WIndows 7 Ultimate host OS
Ubuntu 12.04 i386 guest with 512MB ram, 2 processor

Bridged network using proxydhcp 1 nic with dd-wrt router

client boots to busybox shell

[Alkis Georgopoulos (alkisg)]

If I understood correctly so far,
trial-and-error tried a beta 12.04 installation, and the problem was an initial empty /etc/nbd-server/conf.d/ltsp_i386.conf, which we don't know how it was generated,
and niplas did an upgrade and had a wrong /etc/nbd-server/conf.d/ltsp_i386.conf from his older setup which wasn't upgraded (did 11.10 use name-based nbd exports?).

I don't think anyone has reported this problem on a clean 12.04 installation after release, right?
Anthony was yours a clean installation or an upgrade?

[Anthony (coolio986)]

My install was clean. It was with i386, with updates installed while Ubuntu was installing.

The nbd-server, ltsp_i386.Conf file is not empty, it was created when I did ltsp-build-client.

[Alkis Georgopoulos (alkisg)]

> I can confirm TrialandError's findings. I also am running virtualbox 4.1.14 with extension pack.
> The nbd-server, ltsp_i386.Conf file is not empty, it was created when I did ltsp-build-client.

Anthony, trial-and-error's problem was an empty configuration file.
I'm not sure what your problem is.
Did you restart nbd-server after running ltsp-update-image?
Was you problem fixed, and how? Can you boot your clients now?

[Anthony (coolio986)]

Yes I restarted nbd-server after ltsp-update-image. However even though nbd-server was restarted, the daemon is not listening on port 2000.

My problem is that upon booting the clients, the clients drop to a busy box shell.

If I understand correctly, nbd-server doesn't use port 2000 anymore. Even if I add port =2000 to ltsp_i386.Conf, I get the same busy box shell prompt on boot up.

[Anthony (coolio986)]

The bottom line here is that I have built a ltsp-server using the proxydhcp info from here:

https://help.ubuntu.com/community/UbuntuLTSP/ProxyDHCP

I realize that it is not meant for 12.04, but most of the information works.

I do not need to put ipappend 3 in like it suggests. However if I do, it doesn't matter.

I also do not need "DEFAULT vmlinuz ro initrd=initrd.img quiet splash" like it suggests since it is already in the default file

Upon starting a ltsp-client, pxe boot works perfectly. it loads vmlinuz and initrd

after some time on bootup it sends me into the busybox shell.

No errors no warnings. I have checked syslog and there is nothing to elude to a problem occuring.

The only one error that I do have is tftp : client does not accept options, but i have that on my 11.04 ltsp server and those clients boot perfectly.

I want to find out the reason that the clients are dropping to busybox, but at this point Im not sure how. I can post my logs, and config files. Im not currently at the computer so I will post later

[Alkis Georgopoulos (alkisg)]

OK so your problem is unrelated to this bug report; and I also think it's not an LTSP bug but a misconfiguration.
Best way to get help in troubleshooting it would be to join the #ltsp irc.freenode.net channel, or to ask in the ltsp-discuss mailing list.

IPAPPEND 3 is required in proxyDHCP mode otherwise your clients try to connect to the router to get the NBD image.

So to sum up,
trial-and-error had an empty configuration file while in beta; I think we should ignore that one until someone else reports it for 12.04 final,
and niplas had a old configuration file which should have been upgraded by ltsp-server.postinst but for some reason it wasn't, and we should keep this bug open until the problem is located.

[niplas (niplas)]

> niplas did an upgrade and had a wrong /etc/nbd-server/conf.d/ltsp_i386.conf from his older setup

Just to clarify:
I've tampered with some parts of the ltsp-setup but never touched the nbd-servers configuration up until now.
This might give a hint towards the problem...

My quickfix was to append the contents of /etc/nbd-server/conf.d/ltsp_i386.conf to /etc/nbd-server/config

includedir broken ?

(Just tested to remove my added exports in the default config and placed several .conf-files in the includedir (/etc/nbd-server/conf.d. Nbd-server exits due to no exports.. Seems like the includedir argument indeed is broken or I have file/dir permissions problems)

[The Fan Club (thefanclub)]

Have the same issue. My thin clients were working perfectly yesterday. Now the client drops to Busybox after the latest round of updates.

I have double checked all my configuration files and they seem fine. Also read, and re-read all the posts here and many others.

It seems that nbd-server is not starting on system boot, or not starting correctly.

Test 1:
- Reboot system
- Boot thin client
- Thin client drops to Busybox

Test 2:
- Reboot system
- Restart nbd-server manually with : sudo /etc/init.d/nbd-server restart
- Boot thin client as usual without any problem

[Anthony (coolio986)]

What I found that worked was installing: ltsp-server-standalone

Then purge ltsp-server-standalone, to install ltsp-server

I only then modified the /var/lib/tftpboot/ltsp/i386/pxelinux.cfg/default file

Only add IPAPPEND 3, nothing else. Add it to the end of the default file

DO NOT CHANGE the port numbers (as listed in the proxydhcp page). The port numbers for the default file and the nbd-server config file should be removed if they exist.

Using the proxyDHCP page from the ubuntu ltsp website doesn't work for 12.04. The dnsmasq ltsp.conf file from the ubuntu proxydhcp page still works for the i386 clients.(just remember to change your dhcp-range for the range on your network.

It may also help users to remove quiet splash and plymouth=force to get a regular client console display. You can also just hit escape during the splash screen to see the console (which effectively does the same as removing quiet splash, but for only that one time bootup)

As far as the exports goes, leave it as is. There is nothing wrong with it. you can't grep nbd-server for port 2000, since nbd-server doesn't use port 2000 anymore. If you use port 2000 it will give you file permission problems, (something like Socket Error: connection refused., or Error: Not enough cliserv_magic.)

niplas:
Have you tried skipping the exports option and using oldstyle=true? Then putting all you configuration data into the config file?
If my memory serves, you can put the exports data into the config file and remove the export option. so that nbd-server doesn't say that exports don't exist.

Hope this helps everyone.

[Alkis Georgopoulos (alkisg)]

> Using the proxyDHCP page from the ubuntu ltsp website doesn't work for 12.04.

Anthony, please edit the wiki page and write your findings there, as it's there that will help people, while here it just messes up the bug report.
In the next LTSP version an ltsp-server-dnsmasq package will be available which will make the wiki page unnecessary.

christiaan_, if nbd-server only starts sometimes on boot for you, you should file a separate bug in the nbd-server package, and also mention whatever errors/warnings you see in the logs.

[The Fan Club (thefanclub)]

Hi Alkis

The nbd-server does not start sometimes - it does not start at boot at all. If I try to boot a thin client after server reboot - the thin client drops to BusyBox. If I login as a user and restart the nbd-server manually, the thin clients can login without a problem.

I also did a fresh install of Ubuntu 12.04 on new hardware - and a clean install of LTSP-server-standalone. Everything worked as expected. But after some updates of the client image, the ndb-server stopped starting at boot time ?

The only error on the thin client during boot is "Error: Socket failed: Connection refused", but this is because the nbd-server is not running at this point. If I then start the nbd-server manually clients can login as normal

Question - how and where is nbd-server started during startup ? What could prevent nbd-server from starting ?

I have spent many days trying to resolve the issue. Any help will be appreciated.

[Alkis Georgopoulos (alkisg)]

> Question - how and where is nbd-server started during startup ? What could prevent nbd-server from starting ?

nbd-server ships an initscript in /etc/init.d/nbd-server
If `sudo service nbd-server restart` didn't work, it would be a configuration problem, and since LTSP is responsible for the nbd-server configuration, the problem would be related to LTSP.

But it does work, so it's not an LTSP issue; it might be an nbd-server problem or an upstart problem.
So as I mentioned, you should file a separate bug in the nbd-server package and attach your nbd-server configuration files (in /etc/nbd-server/*) and any related logs you can find in /var/log/syslog.

[The Fan Club (thefanclub)]

> Alkis - Thank you for the reply and assistance. The mystery for me was that the nbd-server was actually not loading during startup at all.

Finally solved - and it had nothing to do with an LTSP config problem in the end. It turned out to be my own custom startup script that called the Network manager command line (nmcli) during the same startup run level rc2. The script caused no conflicts previously but for some reason about a week ago it became the source of the "drop to busybox message", as nbd-server was not loaded at all during startup.

LTSP works really smoothly now in 12.04. Thank you to all.

[Launchpad Janitor (janitor)]

[Expired for ltsp (Ubuntu) because there has been no activity for 60 days.]

[ppmuetze (ppm)]

This little issues is driving me crazy, I have search the internet to no prevail. Everything appeared fine until I added an admin user in chroot and did a ltsp-upgrade-image then this error started.
I moved /opt/ltsp/i386 to /opt/ltsp/i386-ORIG and i did a sudo ltsp-build-client --arch i386 after that finished, still dropped to busybox shell, then I did sudo service nbd-server restart, tried booting the thin client and the same thing happened, then I tried
(cat <<EOF
ipappend 3
EOF
) | sudo tee -a /var/lib/tftpboot/ltsp/i386/pxelinux.cfg/default
and that did the trick.

I really want someone to explain to me what the chroot is for? If it is only supplying the boot image why would we ever chroot in to that environment and do an apt-get update, then rebuild the image? I am just going to note in my blog about my install to never (at the moment) go into your chroot environment. I swear there was a lot more documentation out there on setting up the LTSP-server, I know at one point I was able to have the pxe-boot menu automatically select the arch during boot. Maybe this is just my little rant for finding it impossible to "fix" this problem for the past 6-1/2 hours. I am going to attempt to break things again and see if just adding that code above does the trick.

Thanks for giving me a jumping point guys!!

[ppmuetze (ppm)]

After playing around with it a bit it really appears that after every new "build" or image update you need to re-run
(cat <<EOF
ipappend 3
EOF
) | sudo tee -a /var/lib/tftpboot/ltsp/{arch}/pxelinux.cfg/default
on every build in your LTSP.