does not read /etc/event.d if inotify unavailable

[James D. Freels]

Binary package hint: upstart

I discovered this bug when upgrading from Dapper to Edgy. The upgraded system contained upstart and dependent packages ubuntu-minimal, upstart-compat-sysv, startup-tasks, system-services, upstart-logd. After upgrading, I rebooted and the system hang at the message "Freeing unused kernel memory: 204K freed". After much searching on the web for the cause of the problem, I discovered that a corrupt /sbin/init could be the cause, and if I issued "init=/bin/sh" at the kernel prompt at boot time, I could verify this; which I did.

My system at home was similarly upgraded from Dapper to Edgy. However, for some reasson, it was installed with sysvinit and NOT upstart and dependent packages. Since my home system did not need upstart and dependent packages to work properly, I realized I could delete them from my work machine and then install sysvinit instead. So, I booted up under the Dapper live CD (don't have an Edby CD yet) and performed this task after copying the /sbin/init file and the sysvinit*.deb file from my home machine.

My system now boots properly without upstart. The size of the /sbin/init file in upstart is different from sysvinit, so they ARE different. Why it does not work in upstart for my system is a puzzle I do not care to solve further, since I must now use my system for working.

[Scott James Remnant (Canonical) (canonical-scott)]

Please reinstall ubuntu-minimal (and thus upstart, etc.) and boot your system removing "quiet" and "splash" from the kernel command-line, and adding "--debug"

(you can do this from the GRUB menu)

Take a photo of the screen at the point it hangs

[Scott James Remnant (Canonical) (canonical-scott)]

Note that the entire point of upstart is to replace /sbin/init, that's the upstart binary!

[James D. Freels (freelsjd)]

Scott,

I tried this as you suggested. This basically just undoes what I had to
do to fix the problem. So, when I undid things (added back the packages
I deleted and then deleted sysvinit), then it caused the system to
behave the same way. The --debug switch gave me no information
when /sbin/init runs and it hung the system again. So, I put the system
back the way it was (replacing upstart et. al., with sysvinit) and
rebooted so I can send you this message. I also checked the log files
in /var/log and nothing there from the --debug output either.

For your information, I also run a custom kernel at v2.6.18.1.
The .config for that kernel is attached.

On Mon, 2006-10-30 at 13:10 +0000, Scott James Remnant wrote:
> Please reinstall ubuntu-minimal (and thus upstart, etc.) and boot your
> system removing "quiet" and "splash" from the kernel command-line, and
> adding "--debug"
>
> (you can do this from the GRUB menu)
>
> Take a photo of the screen at the point it hangs
>
> ** Changed in: upstart (Ubuntu)
> Assignee: (unassigned) => Scott James Remnant
> Status: Unconfirmed => Needs Info
>
> ** Summary changed:
>
> - /sbin/init of upstart is different from /sbin/init of sysvinit and causes hang on boot
> + upstart causes hang on boot
>
--
James D. Freels, Ph.D.
Oak Ridge National Laboratory
<email address hidden>
http://www.comsol.com/stories/hfir/

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.18.1
# Sat Oct 14 16:26:19 2006
#
CONFIG_X86_32=y
CONFIG_GENERIC_TIME=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_SEMAPHORE_SLEEPERS=y
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_DMI=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32

#
# General setup
#
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
# CONFIG_POSIX_MQUEUE is not set
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
# CONFIG_TASKSTATS is not set
# CONFIG_AUDIT is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
# CONFIG_CPUSETS is not set
# CONFIG_RELAY is not set
CONFIG_INITRAMFS_SOURCE=""
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_SLAB=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_RT_MUTEXES=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
# CONFIG_SLOB is not set

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y

#
# Block layer
#
# CONFIG_LBD is not set
# CONFIG_BLK_DEV_IO_TRACE is not set
# CONFIG_LSF is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_AS is not set
# CONFIG_IOSC...

[Scott James Remnant (Canonical) (canonical-scott)]

Yes, I'm afraid that it's important that your system is reverted to the non-working state for us to find out what's wrong. So far, you are the only person with this problem out of the millions of people who are now running edgy; so we need to figure out what's special about your system and fix things.

Does upstart literally not print anything? You don't see a line beginning "init:" ? I would really appreciate a photo of the screen at that point, there may be something really important on the screen at that point that you've missed, or the lack of something important may be what tells me what's wrong.

There won't be anything in /var/log, at that point the system hasn't booted so the filesystem isn't writable. We need to debug this the old fashioned way I'm afraid.

One thing I noticed in your kernel config:

# CONFIG_INOTIFY is not set

Could you try building a kernel with inotify support?

I have a feeling that the failure to configure an inotify watch may cause configuration files to not get parsed!

[James D. Freels (freelsjd)]

OK. I will try this again and quote everything I see on the screen. I
will try to find a digital camera for the screen image too ! And add
the inotify switch to the kernel configuration. Then let you know.
Hopefully later today.

On Wed, 2006-11-01 at 12:58 +0000, Scott James Remnant wrote:
> Yes, I'm afraid that it's important that your system is reverted to the
> non-working state for us to find out what's wrong. So far, you are the
> only person with this problem out of the millions of people who are now
> running edgy; so we need to figure out what's special about your system
> and fix things.
>
> Does upstart literally not print anything? You don't see a line
> beginning "init:" ? I would really appreciate a photo of the screen at
> that point, there may be something really important on the screen at
> that point that you've missed, or the lack of something important may be
> what tells me what's wrong.
>
> There won't be anything in /var/log, at that point the system hasn't
> booted so the filesystem isn't writable. We need to debug this the old
> fashioned way I'm afraid.
>
> One thing I noticed in your kernel config:
>
> # CONFIG_INOTIFY is not set
>
> Could you try building a kernel with inotify support?
>
> I have a feeling that the failure to configure an inotify watch may
> cause configuration files to not get parsed!
>
> ** Summary changed:
>
> - upstart causes hang on boot
> + upstart causes hang on boot (possible inotify)
>
--
James D. Freels, Ph.D.
Oak Ridge National Laboratory
<email address hidden>
http://www.comsol.com/stories/hfir/

[James D. Freels (freelsjd)]

Dear Scott et al.:

Adding the CONFIG_INOTIFY flag to my kernel fixed this problem.
Wonderful. upstart and dependent packages are now install on my system.

This leads to another question. Perhaps I am the only one of millions
of Unbuntu/Edgy users who likes to build my own kernel, but this seems
incredible to me. I like a "lean" kernel with only what is necessary
and only the proprietary drivers loaded as drivers (nvidia, vmware). I
have been maintaining my linux systems for years this way long before
Ubuntu was even thought of. I would like to continue to do this. I
have two Ubuntu desktops and 3 Debian/stable servers I maintain in this
fashion.

Exclusive of the hardware drivers required, is there a list of kernel
flag options that Ubuntu expects to be active that I can reference to
make sure I am using all that Ubuntu has to offer ? (such as this one
INOTIFY ?)

Thanks for you help ! It is fixed and this bug can be removed.

On Wed, 2006-11-01 at 12:58 +0000, Scott James Remnant wrote:
> Yes, I'm afraid that it's important that your system is reverted to the
> non-working state for us to find out what's wrong. So far, you are the
> only person with this problem out of the millions of people who are now
> running edgy; so we need to figure out what's special about your system
> and fix things.
>
> Does upstart literally not print anything? You don't see a line
> beginning "init:" ? I would really appreciate a photo of the screen at
> that point, there may be something really important on the screen at
> that point that you've missed, or the lack of something important may be
> what tells me what's wrong.
>
> There won't be anything in /var/log, at that point the system hasn't
> booted so the filesystem isn't writable. We need to debug this the old
> fashioned way I'm afraid.
>
> One thing I noticed in your kernel config:
>
> # CONFIG_INOTIFY is not set
>
> Could you try building a kernel with inotify support?
>
> I have a feeling that the failure to configure an inotify watch may
> cause configuration files to not get parsed!
>
> ** Summary changed:
>
> - upstart causes hang on boot
> + upstart causes hang on boot (possible inotify)
>
--
James D. Freels, Ph.D.
Oak Ridge National Laboratory
<email address hidden>
http://www.comsol.com/stories/hfir/

[Scott James Remnant (Canonical) (canonical-scott)]

Upstream bug

[Marcelo Barros (marcelobarros)]

Hi James, you are not alone anymore :-D.

I had the same problem, but in different situation (not doing an upgrade). I used init with option "-v" and it reported ( I don't know if it helps, Scott):

"init: Reading configuration from /etc/event.d"

But, it hangs after this message and scripts in /etc/event.d are not executed. Using other kernel, with CONFIG_INOTIFY enabled, the startup continues as expected.

My case:
I am using a kernel compiled by myself and I was trying to modify a livecd (phantomsystem, at sourceforge) based on systemv startup. My idea is to use upstart instead old init.

Just for helping other people: CONFIG_INOTIFY can be set at submenu "File Systems", option "Inotify file change notification support".

[Andy Smith (grifferz)]

Just a note to say that I was also bitten by this issue while trying to set up an Edgy Xen domU. Since my dom0 is not Ubuntu, and all the other (working) domUs use the same non-Ubuntu kernel, I did not immediately suspect the kernel.

In fact I spent several hours googling around, setting --debug and so on, before giving up and deciding to come and file a bug against upstart, at which point I found this one and then of course it became obvious.

Some error message about lack of inotify support would probably enable people to work this out a lot quicker, and this probably should be mentioned in the docs also.

Cheers,
Andy

[Scott James Remnant (Canonical) (canonical-scott)]

Committed in rev 383

[Scott James Remnant (Canonical) (canonical-scott)]

Release in 0.3.2

[Scott James Remnant (Canonical) (canonical-scott)]

This has been fixed since 0.3.5 was uploaded into Ubuntu