upstart doesn't start cron automatically on boot in lucid for server on amd64

[skar]

Binary package hint: upstart

1)
Description: Ubuntu 10.04 LTS
Release: 10.04
2)
upstart:
  Installed: 0.6.5-6
  Candidate: 0.6.5-6
  Version table:
 *** 0.6.5-6 0
        500 http://in.archive.ubuntu.com/ubuntu/ lucid/main Packages
        100 /var/lib/dpkg/status
3) Cron should start on system boot and should respawn if it dies accidentally.
4) On system boot, cron isn't starting by default. Also, somehow cron dies once every 6-7 hours and then it's not respawned by upstart.

[Jim Carroll (thecarrolls)]

See this thread: http://ubuntuforums.org/showthread.php?t=1479423

This bug affects a number of people. Based on having been able to get it working I have a guess as to what the problem is. First, here is what I did to get it working.

Cron wasn't starting through upstart but it would start by hand. I went through this process to correct it:

1) Boot/login
2) sudo service cron stop
   This would result in th warning message, "unknown instance," since cron wasn't running.
3) Reboot
   At this point upstart seemed to start cron, however, if I rebooted again, it wouldn't be running. So, prior to rebooting I ...
4) sudo service cron stop
  Again. This (I'm guessing) gave it a clean shutdown because it's been working ever since.

I'm guessing that, if state information about running services is kept somewhere by upstart, that data can be corrupted. I also think the way to corrupt it is to cut the power (or hard reboot) in the middle of the boot process - though now that it's working I can't seem to get it back into the failure mode - so this is a guess. I'm guessing this because I repeatedly hard reset the machine while I was getting it set up (it's a home theater PC running an unstable version of XBMC as the main session).

Hope this helps.
Jim

[Jim Carroll (thecarrolls)]

Sorry, step 5 in the numbered list above should have been "Reboot"

[Jim Carroll (thecarrolls)]

One more note (Launchpad should allow edits). This problem is only upstart getting cron running on boot. I haven't had a problem with it continuing to run after that.

[David Scotson (david-scotson)]

I appear to have this problem too, with cron not starting up at boot since upgrade from Karmic to Lucid. (Well to be precise I noticed this a couple of weeks after the upgrade after only sporadic use of the machine, I don't know if it stopped working directly because of the upgrade or stopped later due to something else happening.)

I've tried the workaround that Jim Carroll posted above with no success.

One thing I wanted to note was that like Jim I too am seeing this on an HTPC box running an XBMC session (i.e. it boots directly into the XBMC app, not a standard Gnome desktop). I wouldn't think that's a particularly common setup but it could just be coincidence as you would think that something like cron would run regardless of what was happening at higher levels like those since it would be used on headless servers, embedded devices etc.

Another common factor that might be affecting this is a number of hard-resets and a series of power failures I experienced, which may have left something in an awkward state.

[skar (skar-karthikeyan)]

David Scotson wrote:
> Another common factor that might be affecting this is a number of hard-
> resets and a series of power failures I experienced, which may have left
> something in an awkward state.
>
I'm running the setup on a pair of IBM 3650s and the problem isn't with
just cron. Even my own custom upstart jobs don't start. So I'd venture
to guess that upstart is the culprit here :)

And the servers are shut down properly and have never had any abrupt
poweroffs or reboots yet. So no way to blame the h/w, or power too.

cheers,
skar.

--
--
The life so short, the craft so long to learn.

[JamesH (jnahughes)]

I've got it too. Not tried the fix above as cannot turn machine (bog standard Acer desktop with AMD processor) off at the moment.

[rem (patterson)]

I have the problem, too. I have not found a workaround. I am running 10.04 on an AMD64 processor. It seems I'll have to drop back a version.

[Nick Moffitt (nick-moffitt)]

I saw this on a fresh Lucid laptop install at some point after applying security updates two weeks ago. Cron had not been running since 10 Sept 2010, and running "sudo stop cron" caused it to reappear on next boot.

This bug caused data backup jobs not to run, and could have been disastrous.

[Nick Moffitt (nick-moffitt)]

Furthermore, this was on a 32-bit desktop (laptop, really) install, so it may be worth amending the title of this bug to make it more generic than "amd64 server".

[Bernd Schlapsi (bernd-sch)]

This bug was reported on 2010-06-10 for an Ubuntu LTS release and since today there is no fix released?
For me this means the latest LTS release is completely useless or is there a fix available?

[lcn_mustard (lcn-mustard)]

I have similar problem with 32bit kubuntu lucid: cron doesn't running after reboot

sudo service cron status
cron stop/waiting

chkconfig:

acpi-support off
acpid off
alsa-mixer-save off
anacron off
apparmor on
apport off
atd off
avahi-daemon off
binfmt-support on
bluetooth off
bootlogd off
brltty on
casper 0
console-setup off
cron 012345S
cryptdisks 0
cryptdisks-early 0
cryptdisks-enable off
cryptdisks-udev off
cups on
dbus off
dmesg off
dns-clean on
ecryptfs-utils-restore off
ecryptfs-utils-save off
failsafe-x off
fancontrol on
grub-common on
hddtemp on
hostname off
hwclock off
hwclock-save off
irqbalance off
joystick on
kdm off
kerneloops on
killprocs on
lm-sensors on
module-init-tools off
network-interface off
network-interface-security off
network-manager off
networking 0
ondemand on
pcmciautils on
plymouth off
plymouth-log off
plymouth-splash off
plymouth-stop off
polipo on
pppd-dns on
privoxy off
procps off
rc.local on
rcS off
rsync off
rsyslog off
saned off
screen-cleanup on
sendsigs 0
stop-bootlogd off
stop-bootlogd-single off
timidity off
tor off
ubiquity off
udev off
udev-finish off
udevmonitor off
udevtrigger off
ufw off
umountfs 0
umountnfs.sh 0
umountroot 0
unattended-upgrades 0
urandom 0S
winbind on
wpa-ifupdown 0
x11-common on

[Jorge L. Diaz Flores (jldflores)]

we solve the problem uninstalling console-tools package, it was breaking upstart

[Clint Byrum (clint-fewbar)]

Taking a look at this bug, and I'm wondering if there are any log entries that could be helpful in determining why cron isn't running.

Could somebody who has experienced this please look through their log files, especially /var/log/daemon.log and /var/log/syslog, for anything from crond.

Also, Jorge, can you elaborate on why you think console-tools was breaking upstart?

Marking Incomplete pending response from affected users and/or Jorge.

[Joshua Kugler (jkugler)]

Looking in daemon.log from our last reboot, I see this:

Jan 27 02:00:00 dev1 init: cron main process (3627) killed by TERM signal

And then during bootup, I see:

Jan 27 02:02:43 dev1 init: cron main process (1007) killed by TERM signal

I don't have any syslog data right now because for some reason Ubuntu, it appears Ubuntu does daily rotations of syslog, and only keeps 7 days. Really!? On a server system?

When I have syslog entries to report, I'll post them as well.

[Jorge L. Diaz Flores (jldflores)]

We have a mix environment of Debian and Ubuntu and we use puppet for configuration management. One of our recipes for Debian include console-tools as a dependency. We reused that recipe in some of ours Ubuntu Lucid 64 bit servers until we discovered that a group of services did not start at boot time including Cron. Looking at /var/log/boot.log we found rare characters just after the message of console-setup. We just assumed it was breaking upstart. We didn't have to much time to research further in the problem so we change console-tools for kbd in the Ubuntu recipes and Voila! problem solved.

Unfortunately we still don't have to much time, but I think you can easily reproduce the bug
just install console-tools package.

[Rob Rushworth (rob-rushworth)]

I think I may also have this problem.

Ubuntu 10.04 Lucid LTS 32bit.

Cron will now not start after boot-up, and as above, my box has had several recent hard starts. I don't have console-tools though - I already have KBD.

Yesterday, I managed to use the workaround boot / sudo cron stop / boot . etc, then "sudo cron start" and cron worked for a while - I opened gedit with it as a test, but at some point yesterday it shut itself down again. Now, the re-boot workaround won't work anymore, and I can't start cron at all.

I'm relatively new to Ubuntu - and maybe I'm wrong, but I can't find any info anywhere else that relates to this problem quite as well.

If anyone would like to see reports/logs/etc - tell me where to find them or how to generate them and I'll send them right along.

Rob

[Clint Byrum (clint-fewbar)]

Ok I'm marking it Confirmed because so many people have found that they are affected.

Now we still need some help reproducing it. I'm wondering if we need to change cron.conf to log crond's stdout/stderr.

If somebody who is affected could change /etc/init/cron.conf to remove the 'expect fork' and change the exec line to:

exec cron -f 2>&1 | logger -t cronlogger

Then boot and look at /var/log/daemon.log for things tagged as 'cronlogger', that might offer some insight.

Thanks!

[Rob Rushworth (rob-rushworth)]

Evening, Clint.

I edited my cron.conf file as above to:

# cron - regular background program processing daemon
#
# cron is a standard UNIX program that runs user-specified programs at
# periodic scheduled times

description "regular background program processing daemon"

start on runlevel [2345]
stop on runlevel [!2345]

respawn

exec cron -f 2>&1 | logger -t cronlogger

and had gedit search the /var/log/daemon.log, but there were no mentions of chronlogger at all.

Complete log from boot after changing cron.conf attached, hth.

Rob

[Clint Byrum (clint-fewbar)]

Excerpts from Rob Rushworth's message of Fri Apr 29 16:03:20 UTC 2011:
> Evening, Clint.
>
> I edited my cron.conf file as above to:
>
> # cron - regular background program processing daemon
> #
> # cron is a standard UNIX program that runs user-specified programs at
> # periodic scheduled times
>
> description "regular background program processing daemon"
>
> start on runlevel [2345]
> stop on runlevel [!2345]
>
>
> respawn
>
> exec cron -f 2>&1 | logger -t cronlogger
>
>
> and had gedit search the /var/log/daemon.log, but there were no mentions of chronlogger at all.

Rob thanks for trying this out!

I'm not sure that daemon.log is where you'll see the errors. Did you
look at /var/log/syslog? Also did cron fail to start at boot? I would
only expect this to help in times where it fails.

[Rob Rushworth (rob-rushworth)]

Happy to help.

I checked the whole of syslog back to January and see no mention of "cronlogger" there either. Same for the whole daemon log.

Cron didn't run at boot after booting with the edited cron.conf file.
I removed the edit last night and booted again this morning. Cron not running after boot this morning. Attached are the syslog entries for the cron.conf edit boot and the following original, unedited cron.conf boot.

Cheers

Rob

[Rob Rushworth (rob-rushworth)]

and attachment

[RoyK (roysk)]

We're seeing this on a number of servers. Starting cron manually after bootup is obviously a solution, albeit obviously not really a good one. Are anyone working on this at all???

[Clint Byrum (clint-fewbar)]

RoyK, its been difficult to work on this as its not clear how to reproduce yet. I'm wondering if this has to do with alternative user/group sources like ldap and such. runlevel 2 happens before the network is up, so calls to getent() for users that don't exist might cause a failure. This would affect systems using winbind, NIS, etc. etc.

Can somebody who is affected try changing /etc/init/cron.conf's start on line to be

start on net-device-up IFACE!=lo

to confirm that?

Also it may help to just move the start on line to after all sysvinit jobs..

start on stopped rc RUNLEVEL=[2345]